Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 37 минут 45 секунд назад

Connectome-Specific Harmonic Waves

5 декабря, 2019 - 03:23
Published on December 5, 2019 12:23 AM UTC

Selen Atasoy's new theory of brain function called Connection-Specific Harmonic Waves (CSHW) could be a giant step forward in our understanding of consciousness. To put this in perspective I will first review what we know so far about how minds work.

Artificial Neural Networks

Much of the most recent progress in machine learning (ML) is in the realm of image recognition. That's because Artificial Neural Networks (ANNs) are unusually good at image processing. Much of the recent progress in machine learning (ML) has been applying artificial neural networks to new problem domains.

This has been made possible by a single invention, Graphical Processing Units (GPUs). A GPU is a computer chip that generates the images in graphically-intensive videogames by performing matrix algebra. Training an ANN is matrix algebra too so the fastest way to train an ANN is with a GPU. This has allowed ANN hardware to outpace than Moore's Law, if just for a moment.

The simplest kind of neural network is a feed forward neural network (FFNN). An FFNN has layers of nodes with connections between them. You train a neural network with the backpropagation algorithm. The important things to take away from the backpropagation algorithm is it's recursive. Therefore neural networks possess a fractal architecture. This is important and we'll get back to it later.

Besides FFNNs, the other kind of neural network is called a recurrent neural network (RNN). This is a neural network with cyclic feedback loops, which creates hysteresis (short-term memory). RNNs have seen success in natural language processing but are not used in the flashy new advancements such as self-driving cars. That's because RNNs are more complicated than FFNNs. We understand RNNs less well than FFNNs. RNNs are simple enough to use on very short time scales but we don't know how to scale them up to long time scales.

Due to our reliance on FFNNs over RNNs, we don't know how to get an ANN to handle time-series data. I know this because I run a startup that uses ML to process time series data. Self-driving cars use FFNNs. FFNNs' inability to process time series data was a contributing factor to the Uber self-crashing car.

We'll get back to RNNs later too.

Neuroscience

Your brain is a neural network. There are two CSHW-related differences between your brain's neurons and ANN neurons.

  • ANN neurons produce floating point output. Biological neurons can't control the amplitude (voltage) of their action potential. It's binary all-or-nothing. Instead, the frequency of a neuron's action potentials increases with the neuron's net stimulus. Neurons in ANNs do not modulate the frequency of their output.
  • The human brain exhibits variety of different large-scale harmonic neural oscillations (brainwaves) corresponding to different mental states like wakefulness, sleep phases and REM sleep. ANNs do not exhibit these.

We know how individual neurons in the human brain work. We know what different regions of the brain do because we observe how human behavior changes when different brain regions are damaged. But we don't know how neurons work together to create these brain regions. CSHW suggests an answer to this question.

At the same time, global workspace theory (GWT) offers an observable definition of consciousness[1]. Basically it's the idea that consciousness is the thing your whole mind is thinking about. Under the hood, one of your brain's parts networks broadcasts itself to the rest of your brain and this becomes the thing you're thinking about. GWT is well-supported by both psychological experiments and contemplative tradition but we don't know how the brain does it. Our ANNs have no global workspace.

To summarize.

  1. We know what neurons are.
  2. We can simulate artificial neurons to accomplish simple tasks at a speed competitive with human beings. This is the best method of writing software to do things like image recognition and playing go.
  3. We know what each part of the human brain does.
  4. We don't know to build a machine to accomplish these complicated tasks using artificial neurons.

In short, our ANNs are fast, scalable and parallelizable. Our ANNs can solve problems where conceptual complexity is very small. However, our ANNs have trouble handling conceptual complexity and time-series data. We don't know how to make the inductive step of putting our simple neural networks together into a larger intelligence. They can pattern-match but they can't strategize. And they lack consciousness. This might not be a coincidence.

If brains possess a fractal architecture then we're missing the hierarchical inductive step.

CSHW

In 2016, Selen Atasoy published a paper in Nature titled "Human brain networks function in connectome-specific harmonic waves". Here's the most important sentence.

[E]igendecomposition of the Laplace operator...can predict the collective dynamics of human cortical activity at the macroscopic scale.

―Atasoy, S., Donnelly, I. & Pearson, J. Human brain networks function in connectome-specific harmonic waves. Nat Commun 7, 10340 (2016) doi:10.1038/ncomms10340

"[E]igendecomposition of the Laplace operator" means finding the harmonics of the connectome. Unstated in this sentence is the possibility that eigendecomposition of the Laplace operator can predict collective dynamics on arbitrary scales. In case that doesn't make sense, here's a crash course on acoustics.

Every sound wave can be broken down into the superposition of resonant frequencies or harmonics[2]. This forms a basis for sound waves oscillating through the geometry. This isn't limited to sound waves. Any kind of wave can be broken down this way. Different harmonics have different frequencies. Higher-frequency harmonics oscillate faster and propagate shorter distances. Lower frequency harmonics oscillate slower and propagate farther distances.

This kind of resonance happens whenever waves bounds through a solid structure, such as sound waves through a violin or x-rays through a crystal. Selen Atasoy and her lab have confirmed resonance of brainwaves bounding through the connectome. The neurons in a single functional region of the brain (a region we've observed to do something important) resonate together. She calls this a "state network".

So what?

When you press middle C on a piano it's not just middle C that vibrates. The other C strings will vibrate too, especially those closest to middle C. That's because from left to right the strings for each octave are half as long as the previous. Integer multiples like this produce resonance.

Every note on the piano has a particular resonance with each other keys. Some pairs of notes are highly resonant with each other. Other pairs have low resonance[3]. It depends on the ratio of one frequency to another.

CSHW is a simple, elegant way to coordinate many different sub-networks into a human brain. When different networks are out of phase with each other the inputs of one turn into static for the other, which is mathematically equivalent to tuning out a radio.

This could explain what meditation does.

Meditation

Conventionally scientific information on meditation is hard-to-come-by because:

  1. Psychology as a science is less than a hundred years old. We've barely recovered from the behaviorist overreaction to Freud.
  2. Governments suppress research on psychedelics. We lost scientific research into meditation amidst the collateral damage.
  3. Government funding for psychological research revolves around curing diseases. Meditation is for healthy people to get better.
  4. MRIs are expensive.
  5. Long-term meditation takes serious dedication every day for decades. Figuring out what does and doesn't works takes centuries.
  6. The only intellectual traditions to record this knowledge in useful form exist outside the Western intellectual tradition.

Eastern monks and yogis have been experimenting with meditation, comparing their results and iterating this technology in an unbroken dharma for several millennia. Our understanding of meditation is like naturalism in the time of Darwin except this time religion has all the data.

Enlightenment

Two and a half thousand years ago lived an Indian prince named Siddhartha. He mastered the already ancient Hindu yogic techniques. He meditated for several years. Then one day, while meditating under a tree, he saw the truth of reality. We don't know exactly what this means. But we do know he got there via meditation, it freed Siddhartha of dukkha (suffering) and it was permanent. This state is called enlightenment.

Then Siddhartha established a monastic order to pass down his discoveries and improve upon them. This organization evolved into the world religion of Buddhism. Some sects have moved on from meditation. Others have been improving upon Siddhartha's techniques up to the present day. But all of them share the objective of recreating the enlightenment state of mind Siddhartha achieved so long ago.

You're probably wondering how we can verify this. The Dalai Lama likes science so he helped convince some asceticism yogis to come down from the mountains and fly across the world and submit to brain scans while meditating and while at rest[4]. Some important discoveries stand out.

  1. The yogis had reduced galavanic skin response in anticipation of physical pain.
  2. The yogis' default mode network did not activate during wakeful rest.
  3. The yogis were in a constant state of gamma wave activity[5].

Discovery (1) is suggestive of reduced dukkha. Discovery (2) is relevant to cybernetics, which we'll get to later. Discovery (3) might be an objective metric we could use to measure enlightenment states.

I've replicated these results myself by achieving meditative states where my default mode network stops making noise. This happens around the 30 minute mark, after access meditation and muscle spasms. Time spend in these states reduces my dukkha. I wouldn't be surprised if I've also increased my gamma wave activity for minutes at a time. I can't afford an fMRI to verify any of this objectively, but my experience is typical[6] for meditators on a path to enlightenment.

If gamma waves are related to enlightenment then we can finally ground enlightenment in the physical universe.

Under CSHW, gamma waves are when everything in your brain is literally in sync with everything else. If CSHW and GWT are both true and enlightenment equals continuous gamma waves then together they would explain the meaning of the weird subjective descriptions of enlightenment people give like "my mind is bigger"[7]. Your consciousness really is bigger instead of being fractured like a split brain patient.

Cybernetics

For a mind to interact intelligently with its environment the mind has to include a simplified model of its environment. Even a thermos does this when it decides whether to keep your drink hot or cold.

Your consciousness lives on the connectome and never interacts with reality directly. Instead, your consciousness interacts with the simplified model of reality created by your mind. Your Self and your Other are both mental constructs.

If CSHW is true and enlightenment equals gamma waves then in an enlightenment state the Self and the Other would be plugged into one another. Under normal circumstances your Self and bits of your mind's representation of the external world may be out of resonance. This is easy to understand if you've been closed off parts of yourself in reaction to abuse.

In this way, CSHW may go a long way towards explaining anattā and its cessation. See, everyone who attains enlightenment does so through one of the three marks of existence. The three marks are aniccā (impermanence), dukkha (unsatisfactoriness or suffering), and anattā (non-self). To understand one of them is to understand all of them. In other words, they're three different ways of getting at the same Truth.

The Truth is that everything you experience is a construction of your mind. But "seeing the Truth" doesn't mean understanding this intellectually. "Seeing the Truth" means getting your various state networks into resonance. You can do this by sitting still (or walking calmly or doing simple chores) and clearing your mind. If your mute your sensory inputs and default mode network long enough then eventually all your state networks will sync up like a roomfull of pendulum clocks.

These traditions present evidence CSHW is an important part of how the brain solves its challenge of coordinating neural network subsystems.

A popular secular meditation manual Mastering the Core Teachings of the Buddha: An Unusually Hardcore Dharma Book by Daniel M. Ingram "Dharma Dan" approaches enlightenment via vipassana meditation. Vipassana is the technique of paying close attention to what's happening in your mind. Dharma Dan emphasizes paying attention to individual high frequency brainwaves. According to Therevada theory, if you use vipassana to look at it your conscious experience at a high enough time resolution your conscious experience breaks down into discrete frames or oscillations. This process also generates insight with leads to enlightenment.

From the perspective of CSHW what's going on in high frequency vipassana is you're directing your global workspace to a single high frequency oscillation instead of jumping around from one signal source to another. Since every network is always working towards anticipating its own inputs this naturally leads to increased resonance as each state network syncs its internal clock with the target of vipassana attention.

The fMRI data corroborating this (the Tibetan yogis from earlier) is based on lovingkindness meditation, not vipassana, but the principle still applies. All contemplative traditions focus consciousness on a single object[8] for a long time. No matter what the object is, eventually this should lead to increased resonance, which explains how contemplative techniques as different as kasina fire meditation can produce such similar outcomes.

CSHW establishes a mathematical foundation for why anger and hatred are universally uprooted across the various contemplative traditions. Anger and hatred do not exist in isolation. They are felt toward your mind's conceptualization of something that isn't you. The distinction between yourself and the other is premised off of idea that you are separate from it. But your mental model of the Other is literally part of your brain. If your brain is in total resonance then you can't feel separation from the Other. Enlightened individuals don't feel anger or hatred because that would be an a priori contradiction. Similarly, anger and hatred are obstacles to enlightenment.

Fractals

Our ANNs can scale to arbitrarily large input/output dimensionality because they're fractal structures in two directions: input/output dimensionality and number of hidden layers. You can cut an ANN in half along either of these directions and get two smaller neural networks.

This is a special case of a general principle. An information processing system can scale to arbitrarily complex problems if and only if the system is structured fractally. Systems without a fractal structure will eventually encounter a computational cliff.

We've hit this cliff with our ANNs. Our ANNs scale well in the aformentioned directions for which they possess fractal geometry. But their hierarchical structure is non-fractal, so they don't work on hierarchical conceptual problems. Our brains are better than our ANNs when it comes to strategic reasoning.

CSHW suggests a framework for how to coordinate ANNs hierarchically. Low frequency waves propagate farther than high frequency waves. So whenever you go up an order of magnitude in physical scale the wavelength increases of the relevant brainwaves increase. CSHW works the same on every scale of observation...all the way down to individual neurons. Remember how biological neurons send a pulse of action potentials instead of modulating voltage? That could be the base case to our fractal induction.

The individual components should be easy to build out of RNNs. They could be scaled with CSHW inductively to larger physical dimensions and time dimensions. This could automatically solve the other problem of how to build RNNs that work for large time scales thereby putting us a giant step forward towards building an artificial general intelligence.

  1. I'm using "consciousness" to refer to the global workspace in GWT. I mean to imply nothing metaphysical with the term. ↩︎

  2. In real-world applications this results in reconstruction errors, especially for square waves. This is addressed in Atasoy's paper. ↩︎

  3. Piano strings differing by exactly the golden ratio have minimal resonance. ↩︎

  4. You can find this research and more in the book Altered Traits by Daniel Goleman, a fascinating book on what science knows on the long-term effects of serious meditation. It was published in 2017 so it contains up-to-date information. However, many of the studies are unreplicated. Considering the historical obstacles to this research, we're lucky to have anything at all. ↩︎

  5. It frustrates me that scientists haven't conducted the same experiments on Zen masters who live in cities and have a formal system for certifying who has become enlightened. I want to know if fMRI scans correlate with dharma transmission. ↩︎

  6. For a first-person account of what it's like to follow this path to its conclusion I recommend Hardcore Zen: Punk Rock, Monster Movies, & the Truth About Reality by Brad Warner and The Science of Enlightenment: How Meditation Works by Shinzen Young. ↩︎

  7. This description comes from a young woman who stumbled into stream entry (a secular name for enlightenment) outside of any meditative tradition. If the woman comes from a Christian tradition (as this woman did) the experience can be confusing. It is believed a small number of random people stumble into enlightenment unpredictably. ↩︎

  8. Except nondual traditions like Zen. They abandon the meditation target and shoot straight towards enlightenment. ↩︎



Discuss

Kansas City Dojo meetup 11-19-19

5 декабря, 2019 - 03:13
Published on December 5, 2019 12:13 AM UTC

I.) ROUND TABLE

  • I began by sharing an instance today where I fell for Cognitive Fusion, but successfully noticed it myself. After our catering order of subs last week, I had developed the routine “pick up food on the way to the Dojo”. This week, I changed the food to pre-packaged fruit and nuts, but I still had the algorithm “pick up food on the way to the dojo”, even though I could have purchased the food days ahead of time, saving me a significant amount of trouble in predicting my already hectic commute to the Dojo meeting space. I now plan to follow through on this in the future. I bring it up mostly as a status update on my own self awareness. I directly credit the Dojo for this particular awareness, since they have pointed out my Cognitive Fusions in the past.
  • I continued by bringing up the situation with my friends, which has escalated to a point where one of them had attempted suicide. I feel awful about the situation, and am trying to be there to support them as their friend. However, I wish I could do more, and I have this anxiety that if only I were smarter and more knowledgeable, there would be something more I could do. Secondarily, I have anxiety that I enjoy other people’s company more than they enjoy mine. I asked the others if they have any advice on how to deal with these feelings.
    • W pointed out that being a friend doesn’t usually involve “fixing” the other person. Just being present, and instilling a feeling of not being alone. Being non-judgmental. As to my secondary concern, he said “If you are enjoying something more than another person, then good for you!” Which was a helpful reframing of my feelings.
    • Life Engineer took a different approach to the things I had said, and wondered out loud if I felt like I was being used by certain friends. He stressed the importance of setting boundaries, and the equality of people in a relationship. He has had friendships in the past that fell apart as soon as he stopped being the initiator, and considers that a sign of an unhealthy relationship. He also emphasized the importance of Self Care. I said I will have to examine my friendships individually, but that I will take this concern seriously and endeavor to establish boundaries for my own mental health.
    • W brought up the difference between “being” vs “doing” relationships. A “being” relationship is one wherein the people simply “exist” together; their relationship does not revolve around a particular scheduled activity or set of activities. A “doing” relationship does, in fact, revolve around a particular activity or set of activities, and it does not interact much (if at all) with the rest of the person’s life. Another way of looking at it: “You engage in a doing relationship because of the things the other person does. You engage in a being relationship because of who they are.” He thinks this is a useful distinction to keep in mind, as if my relationship with my friends is one kind of relationship, it could result in awkwardness or drama if I try to change the relationship into the other kind of relationship. Of course, these two types of relationships are a spectrum, not two opposite poles.

II.) META DISCUSSION

  • I proposed establishing a norm whereby we are intentional about noticing and congratulating our members when they admit they were wrong, or recount a time when they changed their mind. It goes back to “a genuine desire to change”. W does this internally, but is on board with doing this for others.
    I took it a step further and proposed that we could even make it a norm at the Dojo that we share a time when we were wrong, during our turn to speak, in order to foster such a practice in each other, and in anyone that might not be familiar with this concept at all.
    Life Engineer recounts how in one of his lines of work being wrong is something you seek out; you actively try to break things, in pursuit of refining and strengthening your work.
    • We dug deeper into the example of the “2, 4, 8” sequence game, and how one goes about teaching the mindset of falsification. I argued that the first step is to be comfortable being wrong; to understand that it is okay to be wrong, and in fact a necessary step towards being right. I used the example of Past Me from a couple years ago, who conflated “being wrong” with “being immoral”.
      • W brought up how scientific thinking is all about questions. Religious thinking is about answers; it says that you reach an answer, then you are simply done.
      • Life Engineer shifts this a bit, and sees that post-enlightenment religion teaches that the most important thing is to believe the right thing. And if you believe the wrong thing, bad stuff happens. But there is pushback against that these days in modern religion. But it’s not even really just religion; Life Engineer points out how much the human brain loves stories. I pointed out that we also like certainty, because certainty is comfortable, which exacerbates our need for a story.
      • W brings up an Ezra Kline interview, when he was talking to a climate scientist. Ezra asked “So do we need to tell different stories about climate change?” And the scientist says “No, we really just need to stop telling stories at all. Stories have a beginning, a middle, and an end. Real life doesn’t work like that.”
      • Life Engineer quotes Einstein at this point, saying “The problems of today can’t be solved by the same minds that created them.” The takeaway that Life Engineer gets from this is that it’s an old mind that wants these stories. We need to upgrade our minds. We are indeed wired for stories… but this is unlikely to be a good thing.
      • Life Engineer digs deeper into the reasons why there is such discomfort around being wrong; he thinks there’s more to it than the obvious reasons (we’re taught that it is bad in school, social judgement, etc). He thinks there is potentially… “biological”(?) underpinnings to this.
      • W says that when he becomes uncomfortable about being wrong is when he has done something as a result of that belief, and regrets those actions. I reply that I personally struggle with the obvious: social judgement. Which is one reason I wanted to establish our norm of rewarding changing one’s mind.


Discuss

What are some non-sampling-based deep learning methods?

5 декабря, 2019 - 03:09
Published on December 5, 2019 12:09 AM UTC

.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

Conventionally in machine learning, if you want to learn to minimize some loss or maximize some expected return, you do so by sampling a bunch of losses/rewards and training on those. Since the model only ever sees the loss or reward function through the lens of those specific samples, this basic approach introduces a proxy alignment problem.

For example, suppose you train an RL agent to maximize its future discounted return according to some reward function r. Furthermore, suppose there exists some other reward function r′ such that r and r′ give equivalent samples on the training distribution, but diverge elsewhere. If you just train your agent via evaluating r on a bunch of samples, however, then even if your model is in some sense trying to do the right thing, it has no possible way of knowing whether r or r′ is the right generalization.

In many cases, however, we know exactly what r is—we have explicit code for it and everything (or at least some sort of natural language description of it)—but we still only make use of r via sampling/evaluation. Of course, in many RL settings, you actually do only know how to evaluate r, not inspect it in any other way. However, I think a system that only works in settings where you have more access to the reward function than that can still do quite a lot—even if you explicitly know an environment’s reward function, it can still be quite difficult to figure out the optimal policy (think Go, for example) such that having an ML system which can figure it out for you is quite powerful.

So, here’s my question: at least for environments in which you have an explicit reward function, what are some ways of making use of that information in training a deep learning model other than evaluating that reward function on a bunch of samples? I’m also interested in ways of doing this in non-RL settings, though I still mostly only want to focus on deep learning approaches—there are certainly ways of doing this in more classical machine learning, but I’m less interested in those.

Some possibilities that I’ve considered so far:

  • Put a differentiable copy of the reward function inside the network during training such that the network is able to arbitrarily query the reward function however it wants (credit to Nevan Witchers for this idea). For a smooth reward function you could also give your model the ability to explicitly query gradients as well.
  • Express your reward function as a differentiable function with tunable parameters, put a bunch of copies in your network, and then train without freezing those tunable parameters (or maybe freeze for the first n steps then unfreeze). This specific implementation seems pretty janky, but the basic idea here is to find a way to bias the network towards learning an algorithm that includes an objective that’s similar to the actual reward function.
  • Using transparency/interpretability tools, figure out how the model is internally representing the reward function and then enforce that it do so in a way that maps correctly onto the actual reward function.
  • Use a language model to make sense of a natural language description of your reward function in a way that allows it to act as an RL agent. For example, you could fine-tune a language model on the task of mapping natural-language descriptions of reward functions into optimal actions under that reward.
  • Same as the language model idea, but instead of using natural language, use some sort of mathematical/logical/programming language instead. For example, you might be able to do something like this if you had a powerful deep-learning-based theorem prover.

I’m sure there are other possibilities that I haven’t thought about yet, however—possibly including papers on this in the literature that I’m not familiar with. Any ideas?



Discuss

Karate Kid and Realistic Expectations for Disagreement Resolution

5 декабря, 2019 - 02:25
Published on December 4, 2019 11:25 PM UTC

There’s an essay that periodically feels deeply relevant to a situation:

Someday I want to write a self-help book titled “F*k The Karate Kid: Why Life is So Much Harder Than We Think”.

Look at any movie with a training montage: The main character is very bad at something, then there is a sequence in the middle of the film set to upbeat music that shows him practicing. When it's done, he's an expert.

It seems so obvious that it actually feels insulting to point it out. But it's not obvious. Every adult I know--or at least the ones who are depressed--continually suffers from something like sticker shock (that is, when you go shopping for something for the first time and are shocked to find it costs way, way more than you thought). Only it's with effort. It's Effort Shock.

We have a vague idea in our head of the "price" of certain accomplishments, how difficult it should be to get a degree, or succeed at a job, or stay in shape, or raise a kid, or build a house. And that vague idea is almost always catastrophically wrong.

Accomplishing worthwhile things isn't just a little harder than people think; it's 10 or 20 times harder. Like losing weight. You make yourself miserable for six months and find yourself down a whopping four pounds. Let yourself go at a single all-you-can-eat buffet and you've gained it all back.

So, people bail on diets. Not just because they're harder than they expected, but because they're so much harder it seems unfair, almost criminally unjust. You can't shake the bitter thought that, "This amount of effort should result in me looking like a panty model."

It applies to everything. [The world] is full of frustrated, broken, baffled people because so many of us think, "If I work this hard, this many hours a week, I should have (a great job, a nice house, a nice car, etc). I don't have that thing, therefore something has corrupted the system and kept me from getting what I deserve."

Last time I brought this up it was in the context of realistic expectations for self improvement

This time it’s in the context of productive disagreement.

Intuitively, it feels like when you see someone being wrong, and you have a simple explanation for why they’re wrong, it should take you, like, 5 minutes of saying “Hey, you’re wrong, here’s why.”

Instead, Bob and Alice people might debate and doublecrux for 20 hours, making serious effort to understand each other’s viewpoint… and the end result is a conversation that still feels like moving through molasses, with both Alice and Bob feeling like the other is missing the point.

And if 20 hours seems long, try years. 

AFAICT the Yudkowsky/Hanson Foom Debate didn’t really resolve. But, the general debate over “should we expect a sudden leap in AI abilities that leaves us with a single victor, or a multipolar scenario?" has actually progressed over time. Paul Christiano's Arguments About Fast Takeoff seemed most influential of reframing the debate in a way that helped some people stop talking past each other, and focus on the actual different strategic approaches that the different models would predict.

Holden Karnofsky initially had some skepticism about some of MIRI's (then SIAI's) approach to AI Alignment. Those views changed over the course of years

On the LessWrong team, we have a lot of disagreements about how to make various UI tradeoffs, which we still haven't resolved. But after a year or so of periodic chatting about I think we at least have better models of each other's reasoning, and in some cases we've found third-solutions that resolved the issue.

When you have deep frame disagreements, I think "years" is actually just a fairly common timeframe for processing a debate. I don't think this is a necessary fact about the universe, but it seems to be the status quo. 

Why?

The reasons a disagreement might take years to resolve vary, but a few include:

i. Complex Beliefs, or Frame Differences, that take time to communicate. 

Where the blocker is just "dedicating enough time to actually explaining things." Maybe the total process only takes 30 hours but you have to actually do the 30 hours, and people rarely dedicate more than 4 at a time, and then don't prioritize finishing it that highly. 

ii. Complex Beliefs, or Frame Differences, that take time to absorb

Sometimes it only takes an hour to explain a concept explicitly, but it takes awhile for that concept to propagate through your implicit beliefs. (Maybe someone explains a pattern in social dynamics, and you nod along and say "okay, I could see that happening sometimes", but then over the next year you start to see it happening, and you don't "really" believe in it until you've seen it a few times.)

Sometimes it's an even vaguer thing like "I dunno man I just needed to relax and not think about this for awhile for it to subconsciously sink in somehow"

iii. Idea Innoculation + Inferential Distance

Sometimes the first few people explaining a thing to you suck at it, and give you an impression that anyone advocating the thing is an idiot, and causes you to subsequently dismiss people who pattern match to those bad arguments. Then it takes someone who puts a lot of effort into an explanation that counteracts that initial bad taste.

iv. Hitting the right explanation / circumstances

Sometimes it just takes a specific combination of "the right explanation" and "being in the right circumstances to hear that explanation" to get a magical click, and unfortunately you'll need to try several times before the right one lands. (And, like reason #1 above, this doesn't necessarily take that much time, but nonetheless takes years of intermittent attempts before it works)

v. Social pressure might take time to shift

Sometimes it just has nothing to do with good arguments and rational updates – it turns out you're a monkey who's window-of-possible beliefs depends a lot on what other monkeys around you are willing to talk about. In this case it takes years for enough people around you to change their mind first.

Hopefully you can take actions to improve your social resilience, so you don't have to wait for that, but I bet it's a frequent cause.

Optimism and Pessimism

You can look at this glass half-empty or half-full. 

Certainly, if you're expecting to convince people of your viewpoint within a matter of hours, you may sometimes have to come to terms with that not always happening. If your plans depend on it happening, you may need to re-plan. (Not always: I've also seen major disagreements get resolved in hours, and sometimes even 5 minutes. But, "years" might be an outcome you need to plan around. If it is taking years it may not be worthwhile unless you're actually building a product together.)

On the plus side... I've now gotten to see several deep disagreements actually progress. I'm not sure I've seen a years-long disagreement resolve completely, but have definitely seen people change their minds in important ways. So I now have existence proof that this is even possible to address.

Many of the reasons listed above seem addressable. I think we can do better. 



Discuss

Recent Progress in the Theory of Neural Networks

5 декабря, 2019 - 02:11
Published on December 4, 2019 11:11 PM UTC

It's common wisdom that neural networks are basically "matrix multiplications that nobody understands" , impenetrable to theoretical analysis, which have achieved great results largely through trial-and-error. While this may have been true in the past, recently there has been significant progress towards developing a theoretical understanding of neural networks. Most notably, we have obtained an arguably complete understanding of network initialization and training dynamics in a certain infinite-width limit. There has also been some progress towards understanding their generalization behavior. In this post I will review some of this recent progress and discuss the potential relevance to AI alignment.

Infinite Width Nets: Initialization

The most exciting recent developments in the theory of neural networks have focused the infinite-width limit. We consider neural networks where the number of neurons in all hidden layers are increased to infinity. Typically we consider networks with a Gaussian-initialized weights, and scale the variance at initialization as .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} 1√H, where H is the number of hidden units in the preceding layer(this is needed to avoid inputs blowing up, and is also the initialization scheme usually used in real networks). In this limit, we have obtained an essentially complete understanding of both behavior at initialization and training dynamics[1]. (Those with limited interest/knowledge of math may wish to "Significance and Limitations" below).

We've actually had a pretty good understanding of the behavior of infinite-width neural networks at initialization for a while, since the work of Radford Neal(1994). He proved that in this limit, fully-connected neural networks with Gaussian-distributed weights and biases limit to what are known as Gaussian processes. Gaussian processes can be thought of the generalization of Gaussian distributions from finite-dimensional spaces to spaces of functions. Neal's paper provides a very clear derivation of this behavior, but I'll explain it briefly here.

A neural network with m real-valued inputs and 1 real valued outputs defines a function from Rm to R. Thus, a distribution over the weights and biases of such a neural network -- such as the standard Gaussian initialization -- implicitly defines a distribution over functions on Rm. Neal's paper shows that, for fully-connected neural networks, this distribution limits to a Gaussian process.

What is a Gaussian process? It's a distribution over functionsf with the property that, for any finite collection of points .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} X1,...,XN, the valuesf(X1),...,f(XN) have a joint distribution which is a multivariate Gaussian. Any Gaussian process is uniquely defined by its mean and covariance functions, μ(x) and C(x,x′). For points X1,...,XN, the distribution of f(X1),...,f(XN) will have mean μ(X1),...,μ(XN) with covariance matrix Cij=C(Xi,Xj).

The argument that fully-connected neural networks limit to Gaussian processes in the infinite-width limit is pretty simple. Consider a three-layer neural network, with an activation function σ in the second layer and a single linear output unit. This network can be defined by the equation y=∑Vkσ(∑WkjXj) . At initialization, V and W are filled with independent Gaussians, with variance of V scaled as the inverse square-root of the number of hidden-units.

Each hidden unit hk will has a value for each of the inputs Xi , hk(Xi)=σ(∑WkjXji). Since W is random, for each k, hk(→X) is an independent random vector(where we write →X for X1,...,XN). All of these random vectors follow the same distribution, and the output →y=f(→X) of the network is simply the sum of these identical distributions multiplied by the univariate Gaussians Vk. By the multidimensional central limit theorem, this sum will tend to a multidimensional Gaussian.

Image: a sample from a GP defined by a neural net. From McKay(1995).

Extending this argument to multiple hidden layers is also pretty easy. By induction, the pre-activations of each unit in hidden layer J have identical Gaussian process distributions, which induces identical(non-Gaussian,generically) joint distributions on the activations. The pre-activations of layer J+1 are the sum of these activations multiplied by univariate Gaussians, hence the central limit theorem can be applied again to show that these pre-activations have a joint Gaussian distribution for any set of inputs, hence they have a Gaussian process distribution. This inductive process can be used to compute the mean μ(→X) and covariance C(→X,→X) of the output to an arbitrary depth for a given set of inputs →X. For many activation functions including ReLUs, this computation can be done exactly, giving an explicit expression for the distribution over outputs at initialization.

More recently, this behavior was proved to extend to CNNs, and then pretty much all classes of neural network architecture currently used. In convolutional neural nets, the infinite 'width' limit is taken with respect to the number of filters.

Infinite Width: Training

Okay, so we can understand how neural nets behave at initialization in this limit. But we don't care that much about initialization -- what really matters is what function it represents after the training process is over. The training process is over a complex, non-linear space, and seems much less tractable to the kind of analysis used at initialization. Surprisingly, however, there is a similar simplification that occurs when we pass to the infinite-width limit. For those of you who know some machine learning, it turns out that, in this limit, neural networks behave as a kind of kernel machine, using the so-called neural tangent kernel(NTK).

In this case, the derivation of the infinite-width behavior is more complex, so I'll just explain what that behavior is. The key is to consider the effect of the training process on the values f(X1),...,f(XN) at the points to be classified, rather than the weights of the network. Consider two inputs X1,X2. Imagine taking a step of gradient descent on the network weights &#x3B8; to adjust the network output f(X1). What will the effect of this be on f(X2)? To first order, a given change in the weights &#x394;&#x3B8; will change f(X2) by <∇&#x3B8;f(X2),&#x394;&#x3B8;>. Taking a gradient step in the direction of f(X1) will cause a change in the weights of ∇&#x3B8;f(X1). Therefore, taking a step of gradient descent at f(X1) will have the effect of changing f(X2) by <∇&#x3B8;f(X1),∇&#x3B8;f(X2)> We can construct an N×N matrix K&#x3B8; with entries K&#x3B8;,i,j=<∇&#x3B8;f(Xi),∇&#x3B8;f(Xj)> Taking a step of full-batch gradient descent in the direction →&#x394; (indexed along →X) will, to first order, effect a change in the outputs of K&#x3B8;→&#x394;.

Of course, this doesn't really simplify things much, as the matrix K&#x3B8; is itself dependent on the weights, which vary both randomly at initialization and during training. The insight of the NTK paper is that in the infinite-width limit, this dependence disappears. For infinite-width networks:

i) at initialization, K&#x3B8; becomes a deterministic matrix K∞

ii) during training, K∞ doesn't change. (The weights still change during training, but their change is small enough that K∞ is unaffected)

Therefore, training on a set of N inputs can be perfectly simulated by just calculating K∞ for those inputs, then using K∞ to iterate the training(in practice, the end result of training is instead calculated directly, which can be done by inverting K∞) An inductive formula for calculating K∞ is given in the NTK paper.

Another way of thinking about the NTK is that it is essentially equivalent to taking the first-order Taylor expansion of a neural network about its initial parameters. In this regime, the response of the output to changes in the parameters is linear(though the output is not linear in the network input!) Then the above papers prove that, in the infinite-width limit, the training trajectory stays close to that of its Taylor expansion.

Image: The network's training trajectory stays close to that of its linearization ¯h . From Chizat&Bach(2018).

The NTK was originally defined for simply-connected models, but was later extended to convolutional nets, and now pretty much all network architectures. (As a historical note, many of the ideas behind the NTK were discovered before the paper coining the term NTK, check out this paper for instance)

For those of you wanting to attain a deeper understanding, the original NTK paper is a pretty clear read, as is this blog post.

Significance and Limitations of Infinite-Width Limit

So what's the upshot of all this? Does studying the infinite-width limit tell us anything about the success of finite neural networks? I'd argue that it does. Several of the papers above include comparisons between the output of finite-width networks and the analytically-computed predictions of the associated Gaussian processes and neural tangent kernel. Agreement was often pretty close:

Image: convergence of NTK for one-dimensional input space. From the NTK paper.

Moreover, the performance of the NTK-based methods on learning tasks was impressive. This paper used the kernel associated with deep CNN to classify CIFAR-10 images, achieving 77% accuracy, a new record for kernel-based methods. This is only 6% lower than the performance of the original network. The kernel-like behavior of neural networks may not account for all of their good performance, but it seems to explain at least some of it.

Ultimately, the point of relating neural networks to kernel methods is that kernel methods are much simpler. Kernel methods are a sort of generalization of linear models, in which inputs are projected into a higher-dimensional space where they can be linearly separated. Kernels are tractable to mathematical analysis. It's possible to prove that kernel methods will always converge to a global minimum (on the training points) under gradient descent, and thus prove that neural networks will always converge to a minimum when they have enough hidden units. Another mathematical tool for analyzing kernels is their eigen-decomposition: see for instance this paper which finds that the NTK is diagonalized in the Fourier basis on the binary cube. They then use the eigenvalue associated to various functions as a measure of complexity, finding that it correlates well with the generalization performance of the neural network when learning that function.

Despite this, there are limitations to kernel-based analysis. A given NTK will usually underperform its associated neural network, and as far as I know nobody has even tried to apply NTK methods to problems such as ImageNet. (mostly due to computational costs, as using the NTK for regression scales like N3 in number of data points). There are theoretical works that suggest that there exist problems solvable by neural networks which no kernel-based method can solve. See also this paper on the limits of the 'lazy regime', their term for training regimes in which classifiers are approximately linear in their parameters(which includes the infinite-width limit).


Generalization Theory

The works above on the infinite-width limit explain, to some extent, the success of SGD at optimizing neural nets, because of the approximately linear nature of their parameter-space. A remaining piece of the puzzle is generalization, explaining why the global minimum found on the training set will tend to work well on new data points.

Traditionally, statistical learning theory has focused on classes of models where there number of potential functions learnable by that class is small. However, neural networks are usually capable of fitting arbitrary functions of their dataset, so many tools used to prove that models have low generalization error have failed: the bounds they give are vacuous, meaning that they can't certify that the model will perform better than random guessing. This issue was popularized in a 2017 paper by Zhang et al.

Despite this, recently some non-vacuous generalization bounds have been proven. Thus far, the only non-vacuous bounds for 'real' datasets such as MNIST have used PAC-Bayes methods. These methods replace an individual neural net with a learned distribution over network parameters, and introduce a fixed prior over the parameters. The generalization error is bounded by (the square root of) the KL divergence between the prior distribution and the learned distribution. Intuitively: a low KL divergence means the learned distribution has a short description length w.r.t. the prior, and there are only so many such distributions(sort of), so one of them matching the training inputs would be unlikely unless it truly captured part of the underlying function. PAC-Bayes bounds cannot guarantee high performance off the training set, but they can provably bound the error with high probability, assuming that the training data has been fairly sampled from the underlying distribution.

The first work to use PAC-Bayes bounds for modern neural networks was written by Dziugaite&Roy. They were able to prove non-vacuous bounds on a binarized version of MNIST -- not as trivial as it sounds, given that the classifying networks had hundreds of thousand of parameters. Taking as their prior a Gaussian distribution centered at initialization, the authors represented the learned network with another Gaussian distribution whose parameters they optimized with SGD to minimize the PAC-Bayes bound on total error. This work was inspired by the notion of flat minima, which is the idea that gradient descent is biased toward wide minima in parameter space, where perturbing the parameters does not affect the loss much. From a minimum description length principle, flat minima can be described using fewer bits because of their width, which should imply that solutions found by SGD have good generalization performance. The 'nonvacuous bounds' paper used a formalism inspired by this notion to derive provable generalization bounds.

Image: flat vs. sharp minima. From Hochreiter&Schmidhuber(1996).

A later paper made the connection between PAC-Bayes bounds and compression more explicit. They used techniques for compressing the parameters of a neural network to store networks solving full MNIST and ImageNet using far fewer bits than their original size. Using a PAC-Bayes prior over code-words, they were able to provably verify at least 54% accuracy on MNIST and at least 3.5% accuracy on ImageNet(non-trivial given the huge number of classes in ImageNet). More recently still[2], an approach using random projections proved a bound of 85% accuracy for MNIST. Random projections constrain the network parameters to lie in a random low-dimensional subspace. That training is still possible under such a regime indicates that the exact direction chosen by the network is not too important: it's still likely possible to find a good minimum no matter which way it goes, so long as it has enough wiggle room. This further decreases the description length of the model and hence provides a way of obtaining generalization bounds.

Deriving bounds on the generalization error might seem pointless when it's easy to do this by just holding out a validation set. I think the main value is in providing a test of purported theories: your 'explanation' for why neural networks generalize ought to be able to produce non-trivial bounds on their generalization error.


Relevance for Alignment

At this stage, theoretical research on neural networks is not yet directly useful for alignment. Its goal is more conceptual clarity than producing tools that would be useful for practitioners, or even theoretical insights that are directly relevant to alignment-type issues.

In the long run, though, I believe that this sort of research could be crucial for creating aligned AI. It seems plausible that neural networks will be used to build AGI, or be a major component of AGI. If that happens, deeply understanding the implicit bias and optimization properties of these networks will be extremely important for a variety of purposes from choosing the class of models to enabling ongoing monitoring of what they have learned. This sort of theoretical understanding will likely be essentially in the implementation of alignment schemes such as IDA, and could enable more powerful versions of existing transparency and robustness methods.

But even if you think, as MIRI does(?), that neural networks are ultimately too insecure to build aligned AI, I believe trying to understand neural networks is still a worthwhile goal. Neural networks are one of the only techniques we have with anything approaching the generality needed for AGI. If alignment researchers want to build a 'secure version' of neural networks, then it seems necessary to first understand what factors contribute to their strong performance. Then it may be possible to isolate those factors in a more secure and transparent class of models. In contrast, attempting to derive such a class of models from pure thought, or experiments isolated from the mainstream of ML, seems much more difficult. Almost no AI techniques people think of work very well, so the existence of one that does seem to work well on a variety of realistic problems is a powerful and hard-won clue.

The upshot of this for how people interested in alignment should spend their time and money isn't as clear. This seems like an area that academia and industry is already pretty interested in and successful at studying. At the same time, I think there is still a huge amount of work to do and lots of stuff we don't understand, so I could imagine marginal researchers being useful. At the very least, I think it would be a good thing if alignment researchers were aware that advances in the theory of neural networks are happening, and kept tabs on new developments.

Footnotes

[1]: Technical note: Taking the limit of all layers to infinity is ambiguous; do you take the first layer to infinity, then the second, etc., or do you take them all to infinity at once? It turns out you get the same answer either way, so I'll just present as if taking the limits sequentially.

[2]: Disclaimer: I am the author of this paper.



Discuss

Paper-Reading for Gears

5 декабря, 2019 - 00:02
Published on December 4, 2019 9:02 PM UTC

Lesswrong has a fair bit of advice on how to evaluate the claims made in scientific papers. Most of this advice seems to focus on a single-shot use case - e.g. a paper claims that taking hydroxyhypotheticol reduces the risk of malignant examplitis, and we want to know how much confidence to put on the claim. It’s very black-box-y: there’s a claim that if you put X (hydroxyhypotheticol) into the black box (a human/mouse) then Y (reduced malignant examplitis) will come out. Most of the advice I see on evaluating such claims is focused around statistics, incentives, and replication - good general-purpose epistemic tools which can be applied to black-box questions.

But for me, this black-box-y use case doesn’t really reflect what I’m usually looking for when I read scientific papers.

My goal is usually not to evaluate a single black-box claim in isolation, but rather to build a gears-level model of the system in question. I care about whether hydroxyhypotheticol reduces malignant examplitis only to the extent that it might tell me something about the internal workings of the system. I’m not here to get a quick win by noticing an underutilized dietary supplement; I’m here for the long game, and that means making the investment to understand the system.

With that in mind, this post contains a handful of thoughts on building gears-level models from papers. Of course, general-purpose epistemic tools (statistics, incentives, etc) are still relevant - a study which is simply wrong is unlikely to be much use for anything. So the thoughts and advice below all assume general-purpose epistemic hygiene as a baseline - they are things which seem more/less important when building gears-level models, relative to their importance for black-box claims.

I’m also curious to hear other peoples’ thoughts/advice on paper reading specifically to build gears-level models.

Get Away From the Goal

Ultimately, we want a magic bullet to cure examplitis. But the closer a paper is to that goal, the stronger publication bias and other memetic distortions will be. A flashy, exciting result picked up by journalists will get a lot more eyeballs than a failed replication attempt.

But what about a study examining the details of the interaction between FOXO, SIRT6, and WNT-family signalling molecules? That paper will not ever make the news circuit - laypeople have no idea what those molecules are or why they’re interesting. There isn’t really a “negative result” in that kind of study - there’s just an open question: “do these things interact, and how?”. Any result is interesting and likely to be published, even though you won’t hear about it on CNN.

In general, as we move more toward boring internal gear details that the outside world doesn’t really care about, we don’t need to worry as much about incentives - or at least not the same kinds of incentives.

Zombie Theories

Few people want to start a fight with others in their field, even when those others are wrong. There is little incentive to falsify the theory of somebody who may review your future papers or show up to your talk at a conference. It’s much easier to say “examplitis is a complex multifactorial disease and all these different lines of research are valuable and important, kumbayah”.

The result is zombie theories: theories which are pretty obviously false if you spend an hour looking at the available evidence, but which are still repeated in background sections and review articles.

One particularly egregious example I’ve seen is the idea that a shift in the collagen:elastin ratio is (at least partially) responsible for the increased stiffness of blood vessels in old age. You can find this theory in review articles and even textbooks. It’s a nice theory: new elastin is not produced in adult vasculature, and collagen is much stiffer, so over time we’d expect the elastin to break down and collagen to bear more stress, increasing overall stiffness. But if we go look for studies which directly measure the collagen:elastin ratio in the blood vessels… we mostly find no significant change with age (rat, human, rat); one study even finds more elastin relative to collagen in older humans.

Ignore the Labels on the Box

Scientists say lots of things which are misleading, easily confused, or aren’t actually supported by their experiments . That doesn’t mean the experiment is useless, it just means we should ignore the mouth-motions and look at what the experiment and results actually were. As an added bonus, this also helps prevent misinterpreting what the paper authors meant.

An example: many authors assert that both (1) atherosclerosis is a universal marker of old age among humans and most other mammals, and (2) atherosclerosis is practically absent among most third-world populations. What are we to make of this? Ignore the mouth motions, look for data. In this case, it looks like atherosclerosis does universally grow very rapidly with age in all populations examined, but still has much lower overall levels among third-world populations after controlling for age - e.g. ~⅓ as prevalent in most age brackets in 1950’s India compared to Boston.

Read Many Dissimilar Papers: Breadth > Depth

For replication, you want papers which are as similar as possible, and establishing very high statistical significance matters. For gears-level models, you want papers which do very different things, but impinge on the same gears. You want to test a whole model rather than a particular claim, so finding qualitatively different tests is more important than establishing very high statistical significance. (You still need enough statistics to make sure any particular result isn’t just noise, but high confidence will ultimately be established by incrementally updating on many different kinds of studies.)

For example, suppose I’m interested in the role of thymic involution as a cause of cancer. The thymus is an organ which teaches new adaptive immune cells (T-cells) to distinguish our own bodies from invaders, and it shrinks (“involutes”) as we age.

Rather than just looking for thymus-cancer studies directly, I move away from the goal and look for general information on the gears of thymic involution. Eventually I find that castration of aged mice (18-24 mo) leads to complete restoration of the thymus in about 2 weeks. The entire organ completely regrows, and the T-cells return to the parameters seen in young mice. (Replicated here.) Obvious next question: does castration reduce cancer? It’s used as a treatment for e.g. prostate cancer, but that’s (supposedly) a different mechanism. Looking for more general results turns up this century-old study, which finds that castration prevents age-related cancer in mice - and quite dramatically so. Castrated old mice’ rate of resistance to an implanted tumor was ~50%, vs ~5% for controls. (This study finds a similar result in rabbits.) Even more interesting: castration did not change the rate of tumor resistance in young mice - exactly what the thymus-mediation theory would predict.

This should not, by itself, lead to very high confidence about the castration -> thymus -> T-cell -> cancer model. We need more qualitatively different studies (especially in humans), and we need at least a couple studies looking directly at the thymus -> cancer link. But if we find a bunch of different results, each with about this level of support for the theory, covering interventions on each of the relevant variables, then we should have reasonable confidence in the model. It’s not about finding a single paper which proves the theory for all time; it’s about building up Bayesian evidence from many qualitatively different studies.

Mediation is Everything

Everything is correlated with everything else; any intervention changes everything.

That said, very few things are directly connected; the main value is finding variables which mediate causal influence. For instance, maybe hydroxyhypotheticol usually reduces malignant examplitis, but most of the effect goes away if we hold hypometabolicol levels constant. That’s a powerful finding: it establishes that hypometabolicol is one of the internal gears between hydroxyhypotheticol and examplitis.

If I had to pick the single most important guideline for building gears-level models from papers, this would be it: mediation is the main thing we’re looking for.



Discuss

On decision-prediction fixed points

5 декабря, 2019 - 00:02
Published on December 4, 2019 8:49 PM UTC

It seems like for embedded (reflexive, Löbian, etc) LDT agents, there ought to be a fixed point thing between decision and prediction.

Indeed, embedded agents can predict things about their own actions; but by modeling themselves sufficiently well, this should be (in the limit) equivalent to making a decision, as they will be modeling their own thoughts. Conversely, once you have decided, if you do not suffer from akrasia, then you have accurately predicted your next action. (aside: this is the source of the illusion of free will.)

This is related to the class of "metaphysical truths": truths of the form ☐P → P. Whenever an embedded agent believes one of those, then it must (by Löb's theorem) eventually believe P. But there are lots of such truths (perhaps each different religion offers a different set of metaphysical truths), which might then lead to spurious, or even contradictory beliefs!

The key word was "eventually", assuming LDT agents are logical inductors of some kind; in the meantime, the agent may choose its beliefs. Isn't this weird? Beliefs shouldn't be arbitrary!

But you can imagine, as an (imperfect) example, the paradox of self-confidence: if you think you are competent, then you could believe in your ability to self-improve, which will encourage your to foster your own competence; on the other hand thinking that you are incompetent may lead to not believing in your self-improvement ability, leading to a downward spiral.

Each one of these are decision-belief fixed points. Each are, in way (causally?), both true and rational.

I feel like LDT will end up being a reflexive fixed point of this sort (reminiscent of the logical induction fixed point), with the catch that there are many such fixed points. The true decision an LDT agent must make is then choosing the most effective of these fixed points.

(I'm not entirely convined of this yet since I still have no idea what logical counterfactuals will look like)

The moral of the story for us humans is that:

  • akrasia should not exist, not if you can predict yourself well enough;
  • sometimes beliefs are arbitrary. choose the most productive ones, you'll end up believing them all anyway.


Discuss

What additional features would you like on LessWrong?

4 декабря, 2019 - 22:41
Published on December 4, 2019 7:41 PM UTC

I'm not on the LessWrong team; I'm just curious, and might want to answer that question myself ^_^



Discuss

[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations

4 декабря, 2019 - 21:10
Published on December 4, 2019 6:10 PM UTC

[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations View this email in your browser Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version here (may not be up yet).

Highlights

Self-training with Noisy Student improves ImageNet classification (Qizhe Xie et al) (summarized by Dan H): Instead of summarizing this paper, I'll provide an opinion describing the implications of this and other recent papers.

Dan H's opinion: Some in the safety community have speculated that robustness to data shift (sometimes called "transfer learning" in the safety community) cannot be resolved only by leveraging more GPUs and more data. Also, it is argued that the difficulty in attaining data shift robustness suggests longer timelines. Both this paper and Robustness properties of Facebook's ResNeXt WSL models analyze the robustness of models trained on over 100 million to 1 billion images, rather than only training on ImageNet-1K's ~1 million images. Both papers show that data shift robustness greatly improves with more data, so data shift robustness appears more tractable with deep learning. These papers evaluate robustness using benchmarks collaborators and I created; they use ImageNet-AImageNet-C, and ImageNet-P to show that performance tremendously improves by simply training on more data. See Figure 2 of the Noisy Student paper for a summary of these three benchmarks. Both the Noisy Student and Facebook ResNeXt papers have problems. For example, the Noisy Student paper trains with a few expressly forbidden data augmentations which overlap with the ImageNet-C test set, so performance is somewhat inflated. Meanwhile, the Facebook ResNeXt paper shows that more data does not help on ImageNet-A, but this is because they computed the numbers incorrectly; I personally verified Facebook's ResNeXts and more data brings the ImageNet-A accuracy up to 60%, though this is still far below the 95%+ ceiling. Since adversarial robustness can transfer to other tasks, I would be surprised if robustness from these models could not transfer. These results suggest data shift robustness can be attained within the current paradigm, and that attaining image classifier robustness will not require a long timeline.

Safety Gym (Alex Ray, Joshua Achiam et al) (summarized by Flo): Safety gym contains a set of tasks with varying difficulty and complexity focused on safe exploration. In the tasks, one of three simulated robots has to move to a series of goals, push buttons or move a box to a target location, while avoiding costs incurred by hitting randomized obstacles. This is formalized as a constrained reinforcement learning problem: in addition to maximizing the received reward, agents also have to respect constraints on a safety cost function. For example, we would like self-driving cars to learn how to navigate from A to B as quickly as possible while respecting traffic regulations and safety standards. While this could in principle be solved by adding the safety cost as a penalty to the reward, constrained RL gets around the need to correctly quantify tradeoffs between safety and performance.

Measures of safety are expected to become important criteria for evaluating algorithms' performance and the paper provides first benchmarks. Constrained policy optimization, a trust-region algorithm that tries to prevent updates from breaking the constraint on the cost is compared to new lagrangian versions of TRPO/PPO that try to maximize the reward, minus an adaptive factor times the cost above the threshold. Interestingly, the lagrangian methods incur a lot less safety cost during training than CPO and satisfy constraints more reliably at evaluation. This comes at the cost of reduced reward. For some of the tasks, none of the tested algorithms is able to gain nontrivial rewards while also satisfying the constraints.

Lastly, the authors propose to use safety gym for investigating methods for learning cost functions from human inputs, which is important since misspecified costs could fail to prevent unsafe behaviour, and for transfer learning of constrained behaviour, which could help to deal with distributional shifts more safely.

Flo's opinion: I am quite excited about safety gym. I expect that the crisp formalization, as well as the availability of benchmarks and ready-made environments, combined with OpenAI's prestige, will facilitate broader engagement of the ML community with this branch of safe exploration. As pointed out in the paper, switching from standard to constrained RL could merely shift the burden of correct specification from the reward to the cost and it is not obvious whether that helps with alignment. Still, I am somewhat optimistic because it seems like humans often think in terms of constrained and fuzzy optimization problems rather than specific tradeoffs and constrained RL might capture our intuitions better than pure reward maximization. Lastly, I am curious whether an increased focus on constrained RL will provide us with more concrete examples of "nearest unblocked strategy" failures, as the rising popularity of RL arguably did with more general examples of specification gaming.

Rohin's opinion: Note that at initialization, the policy doesn't "know" about the constraints, and so it must violate constraints during exploration in order to figure out what the constraints even are. As a result, in this framework we could never get down to zero violations. A zero-violations guarantee would require some other source of information, typically some sort of overseer (see delegative RL (AN #57), avoiding catastrophes via human intervention, and shielding).

It's unclear to me how much this matters for long-term safety, though: usually I'm worried about an AI system that is plotting against us (because it has different goals than we do), as opposed to one that doesn't know what we don't want it to do.

Read more: Github repo

Technical AI alignment   Problems

Classifying specification problems as variants of Goodhart's Law (Victoria Krakovna et al) (summarized by Rohin): This post argues that the specification problems from the SRA framework (AN #26) are analogous to the Goodhart taxonomy. Suppose there is some ideal specification. The first step is to choose a model class that can represent the specification, e.g. Python programs at most 1000 characters long. If the true best specification within the model class (called the model specification) differs from the ideal specification, then we will overfit to that specification, selecting for the difference between the model specification and ideal specification -- an instance of regressional Goodhart. But in practice, we don't get the model specification; instead humans choose some particular proxy specification, typically leading to good behavior on training environments. However, in new regimes, this may result in optimizing for some extreme state where the proxy specification no longer correlates with the model specification, leading to very poor performance according to the model specification -- an instance of extremal Goodhart. (Most of the classic worries of specifying utility functions, including e.g. negative side effects, fall into this category.) Then, we have to actually implement the proxy specification in code, giving an implementation specification. Reward tampering allows you to "hack" the implementation to get high reward, even though the proxy specification would not give high reward, an instance of causal Goodhart.

They also argue that the ideal -> model -> proxy problems are instances of problems with selection, while the proxy -> implementation problems are instances of control problems (see Selection vs Control (AN #58)). In addition, the ideal -> model -> proxy -> implementation problems correspond to outer alignment, while inner alignment is a part of the implementation -> revealed specification problem.

Technical agendas and prioritization

Useful Does Not Mean Secure (Ben Pace) (summarized by Rohin): Recently, I suggested the following broad model: The way you build things that are useful and do what you want is to understand how things work and put them together in a deliberate way. If you put things together randomly, they either won't work, or will have unintended side effects. Under this model, relative to doing nothing, it is net positive to improve our understanding of AI systems, e.g. via transparency tools, even if it means we build powerful AI systems sooner (which reduces the time we have to solve alignment).

This post presents a counterargument: while understanding helps us make useful systems, it need not help us build secure systems. We need security because that is the only way to get useful systems in the presence of powerful external optimization, and the whole point of AGI is to build systems that are more powerful optimizers than we are. If you take an already-useful AI system, and you "make it more powerful", this increases the intelligence of both the useful parts and the adversarial parts. At this point, the main point of failure is if the adversarial parts "win": you now have to be robust against adversaries, which is a security property, not a usefulness property.

Under this model, transparency work need not be helpful: if the transparency tools allow you to detect some kinds of bad cognition but not others, an adversary simply makes sure that all of its adversarial cognition is the kind you can't detect. Rohin's note: Or, if you use your transparency tools during training, you are selecting for models whose adversarial cognition is the kind you can't detect. Then, transparency tools could increase understanding and shorten the time to powerful AI systems, without improving security.

Rohin's opinion: I certainly agree that in the presence of powerful adversarial optimizers, you need security to get your system to do what you want. However, we can just not build powerful adversarial optimizers. My preferred solution is to make sure our AI systems are trying to do what we want, so that they never become adversarial in the first place. But if for some reason we can't do that, then we could make sure AI systems don't become too powerful, or not build them at all. It seems very weird to instead say "well, the AI system is going to be adversarial and way more powerful, let's figure out how to make it secure" -- that should be the last approach, if none of the other approaches work out. (More details in this comment.) Note that MIRI doesn't aim for security because they expect powerful adversarial optimization -- they aim for security because any optimization leads to extreme outcomes (AN #13). (More details in this comment.)

Verification

Verification and Transparency (Daniel Filan) (summarized by Rohin): This post points out that verification and transparency have similar goals. Transparency produces an artefact that allows the user to answer questions about the system under investigation (e.g. "why did the neural net predict that this was a tennis ball?"). Verification on the other hand allows the user to pose a question, and then automatically answers that question (e.g. "is there an adversarial example for this image?").

Critiques (Alignment)

We Shouldn’t be Scared by ‘Superintelligent A.I.’ (Melanie Mitchell) (summarized by Rohin): This review of Human Compatible (AN #69) argues that people worried about superintelligent AI are making a mistake by assuming that an AI system "could surpass the generality and flexibility of human intelligence while seamlessly retaining the speed, precision and programmability of a computer". It seems likely that human intelligence is strongly integrated, such that our emotions, desires, sense of autonomy, etc. are all necessary for intelligence, and so general intelligence can't be separated from so-called "irrational" biases. Since we know so little about what intelligence actually looks like, we don't yet have enough information to create AI policy for the real world.

Rohin's opinion: The only part of this review I disagree with is the title -- every sentence in the text seems quite reasonable. I in fact do not want policy that advocates for particular solutions now, precisely because it's not yet clear what the problem actually is. (More "field-building" type policy, such as increased investment in research, seems fine.)

The review never actually argues for its title -- you need some additional argument, such as "and therefore, we will never achieve superintelligence", or "and since superintelligent AI will be like humans, they will be aligned by default". For the first one, while I could believe that we'll never build ruthlessly goal-pursuing agents for the reasons outlined in the article, I'd be shocked if we couldn't build agents that were more intelligent than us. For the second one, I agree with the outside view argument presented in Human Compatible: while humans might be aligned with each other (debatable, but for now let's accept it), humans are certainly not aligned with gorillas. We don't have a strong reason to say that our situation with superintelligent AI will be different from the gorillas' situation with us. (Obviously, we get to design AI systems, while gorillas didn't design us, but this is only useful if we actually have an argument why our design for AI systems will avoid the gorilla problem, and so far we don't have such an argument.)

Miscellaneous (Alignment)

Strategic implications of AIs' ability to coordinate at low cost, for example by merging (Wei Dai) (summarized by Matthew): There are a number of differences between how humans cooperate and how hypothetical AI agents could cooperate, and these differences have important strategic implications for AI forecasting and safety. The first big implication is that AIs with explicit utility functions will be able to merge their values. This merging may have the effect of rendering laws and norms obsolete, since large conflicts would no longer occur. The second big implication is that our approaches to AI safety should preserve the ability for AIs to cooperate. This is because if AIs don't have the ability to cooperate, they might not be as effective, as they will be outcompeted by factions who can cooperate better.

Matthew's opinion: My usual starting point for future forecasting is to assume that AI won't alter any long term trends, and then update from there on the evidence. Most technologies haven't disrupted centuries-long trends in conflict resolution, which makes me hesitant to accept the first implication. Here, I think the biggest weakness in the argument is the assumption that powerful AIs should be described as having explicit utility functions. I still think that cooperation will be easier in the future, but it probably won't follow a radical departure from past trends.

Do Sufficiently Advanced Agents Use Logic? (Abram Demski) (summarized by Rohin): Current progress in ML suggests that it's quite important for agents to learn how to predict what's going to happen, even though ultimately we primarily care about the final performance. Similarly, it seems likely that the ability to use logic will be an important component of intelligence, even though it doesn't obviously directly contribute to final performance.

The main source of intuition is that in environments where data is scarce, agents should still be able to learn from the results of (logical) computations. For example, while it may take some data to learn the rules of chess, once you have learned them, it should take nothing but more thinking time to figure out how to play chess well. In game theory, the ability to think about similar games and learning from what "would" happen in those games seems quite powerful. When modeling both agents in a game this way, a single-shot game effectively becomes an iterated game (AN #25).

Rohin's opinion: Certainly the ability to think through hypothetical scenarios helps a lot, as recently demonstrated by MuZero (AN #75), and that alone is sufficient reason to expect advanced agents to use logic, or something like it. Another such intuition for me is that logic enables much better generalization, e.g. our grade-school algorithm for adding numbers is way better than algorithms learned by neural nets for adding numbers (which often fail to generalize to very long numbers).

Of course, the "logic" that advanced agents use could be learned rather than pre-specified, just as we humans use learned logic to reason about the world.

Other progress in AI   Reinforcement learning

Stabilizing Transformers for Reinforcement Learning (Emilio Parisotto et al) (summarized by Zach): Transformers have been incredibly successful in domains with sequential data. Naturally, one might expect transformers to be useful in partially observable RL problems. However, transformers have complex implementations making them difficult to use in an already challenging domain for learning. In this paper, the authors explore a novel transformer architecture they call Gated Transformer-XL (GTrXL) that can be used in the RL setting. The authors succeed in stabilizing training with a reordering of the layer normalization coupled with the addition of a new gating mechanism located at key points in the submodules of the transformer. The new architecture is tested on DMlab-30, a suite of RL tasks including memory, and shows improvement over baseline transformer architectures and the neural computer architecture MERLIN. Furthermore, GTrXL learns faster and is more robust than a baseline transformer architecture.

Zach's opinion: This is one of those 'obvious' ideas that turns out to be very difficult to put into practice. I'm glad to see a paper like this simply because the authors do a good job at explaining why a naive execution of the transformer idea is bound to fail. Overall, the architecture seems to be a solid improvement over the TrXL variant. I'd be curious whether or not the architecture is also better in an NLP setting.

Copyright © 2019 Rohin Shah, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.



Discuss

2019 Winter Solstice Collection

4 декабря, 2019 - 20:25
Published on December 4, 2019 5:25 PM UTC

If you know of a 2019 Winter Solstice event, or solstice-adjacent megameetup, please post relevant links in a comment here.



Discuss

"Fully" acausal trade

4 декабря, 2019 - 19:39
Published on December 4, 2019 4:39 PM UTC

Acausal trade happens when two agents manage to reach a deal with each other, despite not being able to interact causally (and, in some cases, not being sure the other one exists). Consider, for example, the prisoner's dilemma played against another copy of yourself, either in the next room or the next universe.

But those two situations are subtly different. If my copy is in the next room, then we will interact after we've reached our decision; if they're in the next universe, then we won't.

It might seem like a small difference, but my simple way of breaking acausal trade succeeds in the "next universe" situation, but fails in the "next room" situation.

So it would be good to distinguish the two cases. Since the terminology is well established, I'll call the "next universe" situation - where there are no interactions between the futures of the agents - to be "fully" acausal trade.



Discuss

If giving unsolicited feedback was a social norm, what feedback would you often give?

4 декабря, 2019 - 16:11
Published on December 4, 2019 1:11 PM UTC



Discuss

In which ways have you self-improved that made you feel bad for not having done it earlier?

4 декабря, 2019 - 15:33
Published on December 4, 2019 12:33 PM UTC

It can be a decision, a skill, a habit, etc.

Can be because the improvement was very valuable, obvious in insight, a moral imperative, or any other reason.



Discuss

A letter on optimism about human progress

4 декабря, 2019 - 07:21
Published on December 4, 2019 4:21 AM UTC

This open letter was originally posted on Letter.wiki and is part of a longer conversation with Andrew Glover about sustainability and progress.

Dear Andrew,

Thanks for a thoughtful reply. Reading over it, it seems the biggest difference between us is in our expectations for the future, in a word, our optimism. You agree it would be nice to give everyone the luxuries that only the rich enjoy today, but that “it doesn't seem possible to do this.” In a similar vein, on the topic of energy resources, you say you're “not aware of any principle that says new energy sources will be discovered simply by virtue of humans applying their ingenuity.”

So let's talk about that principle.

Certainly there is no law of physics that mandates inexorable progress, on any one axis or even in aggregate. Progress is not automatic or inevitable.

But human ingenuity has been so successful in such a wide variety of areas that I think, on the grounds of history and philosophy, we are justified in drawing a general principle: all problems are solvable, given enough time, effort, and thinking. Or to quote David Deutsch from The Beginning of Infinity, “anything not forbidden by the laws of nature is achievable, given the right knowledge.”

To take the historical view first, think of all the problems humanity has solved, all the magic we've created, that seemed impossible until it was invented—not just in energy, but in every field.

Our farms make an abundance of produce spring from the ground—reliably, consistently, year after year, rain or shine, flood or drought, regardless of what weeds, pests, or disease may attack our crops. We do this in many parts of the world, with different terrain, weather patterns, and growing seasons. We have done this not just through soil, fertilizer, and irrigation, but by breeding better plants—taking command of the taxonomy of species itself. And when the food is ready, we keep it fresh while it is transported all over the world; produce now knows no season or country.

We have largely conquered infectious disease. Except in the parts of the world too poor for effective water sanitation or mosquito control (and the parts of California where children aren't vaccinated), infectious disease is rare, and usually curable. Less than 200 years ago, we didn't know what caused these diseases or how they spread; today we have identified the specific microorganism behind every major malady, and sequenced their genomes.

The way we travel, too, would seem miraculous to our ancestors from 1800. One of them might have lamented that “it would be nice to give every peasant a fine horse and carriage like the king, but it doesn't seem possible given constraints on resources.” But today almost everyone has access to transportation much faster, safer, and more comfortable than any royalty of old. And not just on land and sea: we have broken the bonds of Earth's gravity to soar with the birds, and higher—to the Moon, and even (through our robot servants) to other worlds. No fear of getting lost, either, with detailed maps of every mile of the globe, and satellites far overhead acting as cosmic lighthouses.

Modern factories are equally amazing, in historical perspective, churning out an incredible variety of cheap products made to exacting specifications that couldn't be matched by the greatest master craftsman working by hand. Before the mechanization of the textile industry starting in the 1700s, it was an incredible luxury to have an entire wardrobe full of colorful, stylish clothes, which you can afford to throw out if they get worn or stained. Just the thought of it may have seemed to people of that era akin to how an expensive mansion or a private jet seems to us today—but we found a way to give it to almost everyone.

And I barely need to remind you of the absolute wizardry of electronics. Even the telegraph was a breakthrough in its day; imagine what Morse, Bell, or Edison would think of the iPhone 11 Pro on a 5G network.

All that is just what we can do. Think further of what we know. The mysteries we have solved, the secrets of the universe unlocked! We know the structure of matter and the makeup of stars. We detect swirling black holes a billion light years away, and subatomic particles in our own backyard. We synthesize chemicals of our own design. We image individual molecules, and the insides of human beings. We read the very code of life. But before the Scientific Revolution, all of this seemed beyond the range of human comprehension, a domain seen only by God.

When the human mind understands galactic rotation, the periodic table, thermodynamics, electromagnetism, chemical enzymes, the structure of the cell, the evolution of species—when it has solved problems not only in energy, but in transportation, infrastructure, agriculture, medicine, materials and manufacturing, supply chains and logistics, communication and computation, finance and management—why do you think we can't learn the knowledge we need and solve the problems facing us today? How many more examples do you need to increase your confidence in human ingenuity?

Perhaps one could look at this incredible track record and count it a lucky historical accident, not to be repeated—if there were no deeper, philosophic way to understand how it came about and what could keep it going. But there is: human beings are, again in Deutsch's words, “universal explainers”. Reason, the conceptual faculty, gives us the upper hand in any contest, even though Nature starts out with the home-field advantage.

It's true that we can't see today how we might solve some of the problems we face. But this has always been true, and it's the nature of progress. Just because we don't know how we'll solve problems, doesn't mean we won't.

That, historically and philosophically, is why I'm optimistic. But surely you know these facts—so why, then, are you relatively uninspired about the future?



Discuss

Symbiotic Wars

4 декабря, 2019 - 03:06
Published on December 4, 2019 12:06 AM UTC

A meme is a self-replicating pattern of information. Some change. Some survive. Some die out.

The most virulent memes often reproduce themselves via their own inverses. Talking about how Flat Earthers are wrong increases awareness of the Flat Earther meme. Increasing awareness of the Flat Earther meme equals proliferating the Flat Earther meme. Anti Flat Earthers[1] spawn Flat Earthers. Flat Earthers spawn Anti Flat Earthers. Every Anti Flat Earther contains a dormant Flat Earther meme.

A scholar is just a library's way of making another library.

— Daniel Dennett

In this way Flat Earthers and Anti Flat Earthers are different stages of a single meme's lifecycle. If the Anti Flat Earthers' objective is to spread the Anti Flat Earth meme then this information reverb benefits the Anti Flat Earthers.

Capitalism and Communism are two halves of the most successful meme in history, the Cold War. Capitalism was defined by its opposition to Communism. Communism is defined by its opposition to Capitalism. It doesn't matter that Communist states spread a twisted version of Capitalism and Capitalist states spread a twisted version of Communism. During the Cold War, the title of your economic ideology mattered more than than your real-world economy.

Fighting a meme makes it stronger. The best way to kill a meme is to ignore it.

This presents a collective action problem. If you tell everyone to ignore the X meme then you've told everyone about X thereby spreading the X meme. Attacking a meme wins you Pyrrhic victories. To kill a meme you have to make the meme irrelevant by transcending it.

The Cold War meme died when the divide between Capitalism and Communism ceased to be meaningful. The Chinese Communist Party manages the world's largest capitalist economy. By 1901 Bolshevik standards, NATO is an alliance socialist nations.

  1. "Anti Flat Earther" is a different meme from "physics". Anti Flat Earthers refute Flat Earthers' arguments. The physics meme ignores quibble. ↩︎



Discuss

Long-lasting Effects of Suspensions?

3 декабря, 2019 - 23:40
Published on December 3, 2019 8:40 PM UTC

I recently read "The School to Prison Pipeline: Long-Run Impacts of School Suspensions on Adult Crime" (Bacher-Hicks et. al. 2019, pdf, via Rob Wiblin) which argues that a policy of suspending kids in middle school leads to more crime as an adult.

Specifically, they found that after controlling for a bunch of things, students who attended schools with 0.38 more suspensions per student per year were 20% more likely to be jailed as adults:

A one standard deviation increase in the estimated school effect increases the average annual number of days suspended per year by 0.38, a 16 percent increase. ... We find that students assigned a school with a 1 standard deviation higher suspension effect are about 3.2 percentage points more likely to have ever been arrested and 2.5 percentage points more likely to have ever been incarcerated, which correspond to an increase of 17 percent and 20 percent of their respective sample means.

This is a very surprising outcome: from a single suspension in three years they're 20% more likely to go to jail?

The authors look at the Charlotte-Mecklenburg school district, was ordered by the court to desegregate in the 1970s. In the early 2000s the court was convinced that busing wasn't needed anymore, and the district implemented a "School Choice Plan" for beginning of the 2002 year. Students were massively shuffled between the schools and, while this was generally not randomized, the authors describe it as a "natural experiment".

The idea is that if a student moves from school A to school B and you know how often students were suspended at both schools, then you can look at differences later in life and see how much of that is explained by the difference in suspension rates. They note:

A key concern is whether variation in "strictness" across schools arises from policy choices made by administrators versus underlying variation in school context. Our use of the boundary change partly addresses this concern, because we show that schools' conditional suspension rates remain highly correlated through the year of the boundary change, which provides a very large shock to school context. We also show that school effects on suspensions are unrelated to other measures of school quality, such as achievement growth, teacher turnover and peer characteristics. And: We also test directly for the importance of administrative discretion by exploiting a second source of variation - principal movement across schools. We find that conditional suspension rates change substantially when new principals enter and exit, and that principals' effects on suspensions in other schools predict suspensions in their current schools. While we ultimately cannot directly connect our estimates to concrete policy changes, the balance of the evidence suggests that principals exert considerable influence over school discipline and that our results cannot be explained by context alone. Here's an alternative model that fits this data, which I think is much more plausible. Grant that differences in conditional suspension rates are mostly caused by administrators' policy preferences, but figure that student behavior still plays a role. Then figure there are differences between the schools' cultures or populations that are not captured by the controls, and that these differences cause both (a) differences in the student-behavior portion of the suspension rate and (b) differences in adult incarceration rates. If suspensions themselves had no effects we would still see suspension appearing to cause higher incarceration rates later in life.

They refer to movement of principals between schools, which offers a way to test this. Classify principals by their suspension rates, and look at schools that had a principal change while keeping the student body constant. Ideally do this in school districts where the parents don't have a choice about which school their children attend, to remove the risk that the student population before and after the principal change is different in aggregate. Compare the adult outcomes of students just before the change to ones just after. While a principal could affect school culture in multiple ways and we would attribute the entire effect to suspensions, this would at least let us check whether the differences are coming from the administration.

This sort of problem, where there's some kind of effect outside what you control for, which leads you to find causation where there may not be any, is a major issue for value-added models (VAM) in general. "Do Value Added Models Add Value?" (Rothstein 2010, pdf) and "Teacher Effects on Student Achievement and Height" (Bitler et. al. 2019, pdf) are two good papers on this. The first shows that a VAM approach yields higher grades in later years causing higher grades in earlier years, while the second shows the same for teachers causing their students to be taller.

I continue to think we put way too much stock in complex correlational studies, but Bacher-Hicks is an illustration of the way the "natural experiment" label can be used even for things that aren't very experiment-like. It's not a coincidence that at my day job, with lots of money on the line, we run extensive randomized controlled trials and almost never make decisions based on correlational evidence. I would like to see a lot more actual randomization in things like which teachers or schools people are assigned to; this would be very helpful for understanding what actually has what effects.



Discuss

(Reinventing wheels) Maybe our world has become more people-shaped.

3 декабря, 2019 - 23:23
Published on December 3, 2019 8:23 PM UTC

Let's stow our QM caps and pretend Democritus was right: Atoms turn out to be the fundamental, indivisible units of physical reality after all. Let's further pretend that some flavor of hard determininism is in play: There is nothing apart from the motions and interactions of the atoms that governs the way the Universe ambles through time.

Past, perhaps, the first Planck moment of existence, where we might be initializing all of the constants we'll need, our billiard-ball universe is quite amenable to explanations at the atomic level in the language of causality. Why is Atom A here, and not there, at time t? Because it was caused to be there by the actions of the other atoms, and due to certain properties of A itself, in the time leading up to t.

In theory, this means that any level of abstraction we build up from the atomic one should preserve that ability to be described causally. But the amount of computational power we would need to actually pull that off would be staggering, far, far more than we could possibly fit within 3 pounds of grey matter. So even starting from the most deterministic possible model, as agents within the system, we don't really have the ability to directly leverage that causality.

Instead, we are forced by our own limited resources to construct abstractions that are simple enough for us to reason about. These abstractions throw out a lot of detail! And when you throw away even a small amount of detail, you lose the clean isomorphism-to-reality that allows our earlier statement of causality to be preserved. When you're dealing with atoms, in the billiard ball world, you can always predict where they'll be if you have enough power; when you're dealing with aprroximations of atoms, you lose the "always".

So not only is the map not the territory, if you lose the territory, there is no way to perfectly accurately reconstruct it from the map alone. If you ever find yourself in that unenviable place, you'll have to make (dare I say it) aesthetic decisions in the reconstruction process.

And that's a weird thing to think about, not least because that isn't the case in mathematics. If you have a finite-bandwidth mathematical signal, you can perfectly reconstruct it from a finite number of details about it with the correct sampling conditions. That's just the most direct example. What about how we can use set theory to define the natural numbers, for example, 0 = null; 1 = {null} = {0}; 2 = {{null}, null} = {1, 0}; ... ? In fact, mathematics abounds with territories that can be perfectly reconstructed from very small maps - that's one of the most interesting things about it as a subject.

In other words, it feels awfully like causality should be a rare and fleeting thing, as rare as pouring a bottle of clover honey into your chamomile tea and having it form into a tiny little honey-fairy taking a nice bath.


And yet... All of that clashes with my very enjoyably lived experience that, on the day-to-day human level, causality actually seems to work pretty freaking fine for most of my decisions.

Why? What on Earth is the difference?

I have a boring but serviceable hypothesis: We've just been around long enough to make our world human-shaped enough to let this happen.

Look around you. Chances are ninety percent of the interesting/useful/beautiful/etc. objects around you were either designed by humans, or placed there strategically by them. We might only have 3 pounds of grey matter to work with, but we had the fantastic luck to have a big chunk of that grey matter go towards a really good abstraction of how other humans worked. That means we can perform second-, and even third-order abstractions on how to set things up for them so they will have an easier time. When we wrap this chunk around and use it on ourselves, we often come up with better ways to do things than if we just introspected directly.

At the dawn of human history any one of us might have had a one-in-a-thousand chance of making the correct abstraction of "seed + soil + water = food" ex nihilo. That's okay, because the chances of us being able to spread that discovery to others are much higher. As the centuries unfolded, our ability to efficiently abstract one another might have allowed something like a compound-interest effect to take hold as regards these normally quite rare discoveries of bubbles of causality in the natural world; now we sit here in 2019 with a world where most of the most inscrutable problems we face are much bigger than ourselves. We worry about AGI, but we don't worry about, say, how to make a new T shirt once our last one falls apart. Even if we did have to make a new one by hand, we have mechanisms in place to acquire that knowledge; facing down the problem with nothing at our holster but trial and error is a much scarier proposition.



Discuss

Russian x-risks newsletter #2, fall 2019

3 декабря, 2019 - 19:54
Published on December 3, 2019 4:54 PM UTC

Russia could be seen as an x-risks wonderland, with exploding nuclear facilities and bioweapons storage sites, doomsday weapons, frequent asteroid impacts, Siberian volcanic traps, Arctic methane bomb, military AI programs, and crazy scientists wanting to drill to the Earth’s core. But there are also people who do their best trying to prevent global risks, and one of the most well-known is Petrov, who is remembered on 9/26. Russia can also provide much of the world with lessons in resilience; for example, many of its people, living in villages, can still supply themselves with food, water and energy without access to external sources.

Bio

Explosion in the virology center in Novosibirsk, 16 September 2019. The explosion was caused by a natural gas tank used in the renovation process. The fire was allegedly close to the storage facilities but didn’t directly affect them, so there was no leak of hazardous biological materials.

Why it is important? This seems to be a new type of possible catastrophic event, that has not been previously predicted, one that could produce a “multipandemic”—the simultaneous release of many deadly biological agents. A possible protective measure against new events of this type is the preservation of deadly agents in different places.

What is it? This center is one of two places in the world where live samples of smallpox are preserved. The State Scientific Center for Virology and Biotechnology (SSC WB), also known as "the Vector Center", has one of the most comprehensive collections of viruses in the world, including Ebola, Marburg hemorrhagic fever, severe acute respiratory syndrome (SARS), smallpox, and others. Created in 1974 near Novosibirsk, it was previously a closed institute that led the development of vaccines, as well as strategies for protection against bacteriological and biological weapons. In 1999, Valentin Yevstigneev, the head of the biological defense department of the Russian Ministry of Defense, said that they began to consider the Vector Center as "an industrial base for the production of offensive biological preparations" in the late 1980s. It was assumed that the strains of smallpox, tularemia, plague, anthrax and Ebola developed by the center would be placed in warheads. This work was curtailed in 1992, shortly after USSR ceased to exist.

AI

In October of 2019, Russian president Vladimir Putin signed into law a new national AI strategy, the text of which includes a passage about the need for fundamental research to develop strong (universal) AI. This line was supported during the main Russian AI event, the conference AI Journey in November 2019, where Schmidhuber and Ben Goertzel spoke about AGI. Putin later joined the conference, who reiterated that those who control AI will control the world, and also mentioned his belief in the need to create strong AI. The main mind behind the conference was German Gref, who is a big AI fan and the director of the Russian bank Sberbank. During the conference, six major Russian tech companies signed an agreement to form something like a Russian variant of the Partnership for AI. There were some ritual words about “AI ethics” during the conference, but nothing was said about the AI alignment problem. More technical sessions about AGI also took place; there were five presenters on topics including the DeepPavlov project from MIPT, and AIXI modification from Occam. Computer scientists Alexey Potapov (not to be mistaken for Russian cryonist Alexey Potapov) and Alexander I. Panov (not to be mistaken for Russian SETI scientist and astronomer Alexander D. Panov) presented at the meeting.

The new Russian AI strategy requested funding of around 6.5 billion USD until 2024, which is not a terribly large budget if compared with the AI strategies of other countries.

Runaway global warming

Scientists from Tomsk detected the eruption of a “methane bomb” in the East Siberian sea, which resulted in methane atmospheric concentration 6–7 times above normal, representing the biggest methane leak for 45 years of observation. Similarly, thousands of lakes have erupted methane in Siberia and Alaska. Meanwhile, Russia has signed the Paris climate agreement as of 23 September. However, Greta Thunberg-inspired climate protests had turnouts of less than 200 people and experienced police crackdowns.

Why it is important? Russia’s Arctic shelf and tundra has a large amount of accumulated methane sequestered in the form of ice-methane clathrate underwater and in organic material under permafrost. Because of the polar amplification of global warming, most of the warming is happening in polar regions. Methane itself is a much more potent greenhouse than CO2, but behaves differently: it has a half-life of only seven years, but it has “high global warming potential of 72 (averaged over 20 years) or 25 (averaged over 100 years)” times that of CO2 (phyz.org). This means that if we account for a one-time strong eruption, its effects are even stronger in the first seven years. Taken together, these factors suggest the possibility of a strong positive feedback loop of uncertain magnitude, but the heavy tail of this uncertainty includes runaway global warming leading to a lifeless planet.

Nukes

A Russian scientist suggested we should nuke asteroids that are on a collision course with Earth. However, while nukes in space could be used as a weapon against targets on Earth, they would be ineffective in asteroid deflection, as pieces of an asteroid would hit the Earth if it was intercepted at short notice or close range. This approach would not help against large, kilometer-size asteroids in any case. Observation and intervention to change the orbits of asteroids long before the risk of impact may be the best preventive measure.

Meanwhile, Russia declared that it will resume production of medium-range nuclear missiles after the US abandoned the treaty which banned these missiles. The main goal of the treaty was to prevent accidental nuclear war, as shorter-range missiles could reach Moscow from Western Europe in 5–7 minutes, and the decision-makers would have little time to evaluate the reality of the threat. However, the advent of sea-based, air-based and hypersonic missiles has eroded the definition of “medium range”, and the US has previously accused Russia of violating the treaty.


Previous newsletter: Summer 2019



Discuss

Mistake Versus Conflict Theory of Against Billionaire Philanthropy

3 декабря, 2019 - 16:20
Published on December 3, 2019 1:20 PM UTC

Response To (SlateStarCodex): Against Against Billionaire Philanthropy

I agree with all the central points in Scott Alexander’s Against Against Billionaire Philanthropy. I find his statements accurate and his arguments convincing. I have quibbles with specific details and criticisms of particular actions.

He and I disagree on much regarding the right ways to be effective, whether or not it is as an altruist. None of that has any bearing on his central points.

We violently agree that it is highly praiseworthy and net good for the world to use one’s resources in attempts to improve the world. And that if we criticize rather than praise such actions, we will get less of them.

We also violently agree that one should direct those resources towards where one believes they would do the most good, to the best one of one’s ability. One should not first giving those resources to an outside organization one does not control and which mostly does not use resources wisely or aim to make the world better, in the hopes that it can be convinced to use those resources wisely and aim to make the world better.

We again violently agree that privately directed efforts of wealthy individuals often do massive amounts of obvious good, on average are much more effective, and have some of the most epic wins of history to their names. Scott cites only the altruistic wins and effectiveness here, which I’d normally object to, but which in context I’ll allow.

And so on.

Where we disagree is why anyone is opposing billionaire philanthropy. 

We disagree that Scott’s post is a useful thing to write. I agree with everything he says, but expect it to convince less than zero people to support his position.

Scott laid out our disagreement in his post Conflict vs. Mistake.

Scott is a mistake theorist. That’s not our disagreement here.

Our disagreement is that he’s failing to model that his opponents here are all pure conflict theorists.

Because, come on. Read their quotes. Consider their arguments.

Remember Scott’s test from Conflict vs. Mistake (the Jacobite piece in question is about how communists ignore problems of public choice):

What would the conflict theorist argument against the Jacobite piece look like? Take a second to actually think about this. Is it similar to what I’m writing right now – an explanation of conflict vs. mistake theory, and a defense of how conflict theory actually describes the world better than mistake theory does?

No. It’s the Baffler’s article saying that public choice theory is racist, and if you believe it you’re a white supremacist. If this wasn’t your guess, you still don’t understand that conflict theorists aren’t mistake theorists who just have a different theory about what the mistake is. They’re not going to respond to your criticism by politely explaining why you’re incorrect.

I read Scott’s recent post as having exactly this confusion. There is no disagreement about what the mistake is. There are people who are opposed to billionaires, or who support higher taxes. There are people opposed to nerds or to thinking. There are people opposed to all private actions not under ‘democratic control’.  There are people who are opposed to action of any kind. 

There are also people who enjoy mocking people, and in context don’t care about much else. All they know is that as long as they ‘punch up’ they get a free pass to mock to their heart’s content.

Then there are those who realize there is scapegoating of people that the in-group dislikes, that this is the politically wise side to be on, and so they get on the scapegoat train for self-advancement and/or self-protection.

Scott on the other hand thinks it would be a mistake to even mention or consider such concepts as motivations, for which he cites his post Caution on Bias Arguments.

Caution is one thing. Sticking one’s head in the sand and ignoring most of what is going on is another.

One can be a mistake theorist, in the sense that one thinks that the best way to improve the world is to figure out and debate what is going on, and what actions, rules or virtues would cause what results, then implement the best solutions.

One cannot be an effective mistake theorist, without acknowledging that there are a lot of conflict theorists out there. The models that don’t include this fact get reality very wrong. If you use one of those models, your model doesn’t work. You get your causes and effects wrong. Your solutions therefore won’t work.

There already were approximately zero mistake theorists against billionaire philanthropy in general, even if many of them oppose particular implementations.

Thus, I expect the main response to Scott’s post to mainly be that people read it or hear about it or see a link to it, and notice that there are billionaires out there to criticize. That this is what we are doing next. That there is a developing consensus that it is politically wise and socially cool to be against billionaire philanthropy as a way of being against billionaires. They see an opportunity, and a new trend they must keep up with.

I expect a few people to notice the arguments and update in favor of billionaire philanthropy being better than they realized, but those people to be few, and that them tacking on an extra zero in the positive impact estimation column does not change their behavior much.

There were some anti-government arguments in the post, in the hopes that people will update their general world models and then propagate that update onto billionaire philanthropy. They may convince a few people to shift political positions, but less than if those arguments were presented in another context, because the context here is in support of billionaires. Those who do will probably still mostly fail to propagate the changes to the post’s central points.

Thus, I expect the post to backfire.



Discuss

Searching Along the Trail of Crumbs

3 декабря, 2019 - 16:10
Published on December 3, 2019 1:10 PM UTC

There are two extraordinarily powerful things going on right now in Standard. One of them is Cavaliers, which I played in Twitch Rivals and wrote about here.

The other is the food engine, a core of cards with several possible ways to finish.

The Core Engine

The never-touch-this-actual ever core are these fifteen cards:

4 Witch’s Oven

3 Caldron Familiar

4 Trail of Crumbs

4 Gilded Goose

Duplicate Cats are often low impact, and risk draws that have too much air, so it’s not clear that you want the fourth copy.

If you have the cats but you’re not running green at all, then you’re running Mono-Black Sacrifice or Rakdos Sacrifice. Those are decks, but they’re focusing on a different game plan.

With this engine, you have fifteen one and two mana cards. Trading with any of them without giving you value is usually impossible. Most subsets of the list are continuous advantage engines. Together, these cards give you both good things to do in the early turns and a late game engine that can grind out anyone who doesn’t finish you off. That’s quite the combination.

They also give you a continuous supply of sacrifice triggers, death triggers and food tokens for you to build upon, and also a key artifact worth looking for, which gives you a lot of directions you can go. The rest of this article is about exploring the various options for the remaining 20-21 spell slots.

Instead of fighting over snowballing planeswalker activations, we are now fighting over snowballing Trail of Crumbs activations and who can put more cats into ovens each turn.

Wicked Wolf

The list of additional options starts with Wicked Wolf. People are increasingly moving away from Wicked Wolf. They need to slow down. Wicked Wolf is amazing once you have the twelve sources of food. It was amazing when we only had eight from Gilded Goose and Oko, Thief of Crowns.

Wicked Wolf gives you removal for almost all creatures in the format.

Wicked Wolf gives you an effectively indestructible threat to pressure opponents and planeswalkers. With so much food it can often go very large for a lethal attack.

Wicked Wolf gives you an additional zero-mana way to sacrifice food when you want to activate Trail of Crumbs.

Wicked Wolf with food available can hold the ground on its own or with minimal against a large percentage of aggressive strategies.

Wicked Wolf is the most reliable way to find cards that close the game once you have the engine running, taking down their problematic creatures then as an unstoppable attacker.

Wicked Wolf plays great Magic. Be very hesitant to play a food engine without it.

There are three arguments against it that I can see.

Case number one is that Wicked Wolf is that your build doesn’t want to support double green. The problem is that you already need to support first turn Gilded Goose. Once you are doing that, there is not much additional cost to supporting Wicked Wolf. Thus, for example, the Jund Sacrifice deck from Twitch Rivals that went undefeated on day one played only one Wicked Wolf, but had sixteen green sources from its lands plus eight from its creatures, so it still felt comfortable sideboarding Questing Beast. So I don’t think this holds water.

Case number two is that the card is bad against Jeskai (and control, where a parallel discussion will say basically all the same things). I think this is a pre versus post board confusion. In game two you want to fit in both Duress and access to removal for Fires, so your space is super tight, and you end up having other priorities. But it’s not like Wicked Wolf is bad even then. Dealing with it can be quite the challenge, it can take out a Cavalier, and it is very good against Sphinx of Foresight. It usually can’t be killed by any of the sweepers either. In game one I think you give up very little having this, as opposed to running three drops that die to Deafening Clarion or paying five mana for Massacre Girl, both of which hurt quite a bit. So I don’t think this matters much, either.

Case number three is that in the mirror Wicked Wolf is blocked forever by an endless stream of cats, and does not usefully block. That makes it a four mana removal spell that often requires eating a food, which is not impressive. That’s all true, but if they don’t have the full Cat-Oven trick online, then this puts meaningful pressure on them, and if you don’t have that trick, it prevents attacks that can potentially do a lot of damage. Even if they do have the full engine, you can often attack with multiple things, and increasingly there are ways to kill Witch’s Oven on both sides, which opens things up to attacks once again. So I see a lot of value in the creature after the fight, and the fight can take out Mayhem Devil or Gilded Goose or Massacre Girl and provide a lot of value. It’s certainly better than Murderous Rider, which dies when they sacrifice the target to Witch’s Oven, and you need some amount of removal. So again, I don’t see it.

And the pseudo-fourth argument is that the places it is best are aggro decks, which ‘don’t really exist.’ Except that they do exist.

Thus, I don’t get cheating on this one, at all. Even if you don’t maindeck all four copies, it’s one of the best cards against aggressive strategies, so I can’t see not having all four in the 75, and even where it’s bad, it’s not that bad. People sideboard it out in matchups where it can be surprisingly problematic to deal with, like against Cavaliers, where how many to have after board is a very interesting question I’m uncertain about.

From here I will assume we thus have a 19 card base rather than a 15 card base, which combined with the bare minimum 24 lands gives us 17 remaining slots to play with.

Land Base and Color Costs

We always have access to heavy green since we must support Gilded Goose and Trail of Crumbs. We always have access to at least some black so we can cast Caldron Familiar. So we get heavy green, heavy a second color, and if we want it enough, we get a splash color. In theory we could double (or even triple) splash using Paradise Druid and Fabled Passage plus Gilded Goose, if we wanted to do something wild.

If we run a two color build and stick to Golgari, our mana is solid. We get four Overgrown Tomb for free, and get additional dual lands from Fabled Passage and/or Temple of Malady. I do not like running Temples of any kind in these lists, because once you start using Trail of Crumbs every extra mana you get often matters on every turn of the game.

Fabled Passage also gives you four additional sources for any potential splash, and gives you four more triggers for Mayhem Devil.

I don’t think the deck requires four Fabled Passage if it is sticking to two colors, but it’s at worst small mistake to run four and make your mana that much more solid, especially if you run a 25th land. It also matters how invested you are in double black.

If you are running three colors, especially when that third color is red, then your splash is either something very tiny like black for three cats or it is a real third color. If it’s black for three cats, then you don’t want to run a Swamp and may not even want all eight shock lands, so Fabled Passage does not help you, and having all the extra shock lands makes another tapped land that much worse, so I think you avoid it. The same goes for if you’re splashing white for Ethereal Absolution and a few sideboard cards. If it’s a real third color, then I do not see a way out of running four copies of Fabled Passage and a basic land, plus shock lands, as your way to get the third color. The borderline case is red purely for Mayhem Devil. There I think that the extra Mayhem Devil trigger from Fabled Passage pushes you back to running four copies and the Mountain, even though that’s more red mana than you would otherwise need. If you also have Korvold, Fae-Cursed King, the decision is easy.

Four or even five colors are available for splashes, if you are willing to run Paradise Druid and Fabled Passage. That gives you 13 sources off one basic land, which is not so bad.

You can also run Castle Garenberg or Castle Locthwain.

Tapped green sources are scary with Gilded Goose, so you likely can’t play both four copies of Fabled Passage and also Castle Garenberg in the same build. Castle Garenberg gets you to Feasting Troll King if you are interested in that, and also allows you to make food while deploying Wicked Wolf, both a turn early, or to leave extra mana for Trail of Crumbs. I think two color builds should seriously consider choosing a Castle Garenberg as more valuable than a Fabled Passage, even without any six drops.

Castle Locthwain is a card you hope to never have to activate, but one does not always have Trail of Crumbs, and the price of including it is very low. The first copy is essentially mandatory, the second copy is much rarer, and this seems right.

Land counts range from 23 to 25. Running 23 is almost certainly wrong even without a high end. Whether to run 24 or 25 remains unclear to me, but these decks hate missing land drops and don’t want to have to dig for them with Trail of Crumbs. Versions that don’t have extra one or two drops beyond the core need the 25th land. Versions that run Paradise Druid or other cheap action should be fine at 24.

The limiting factor on green sources is first turn green, where you want at least 14. Beyond that you’re probably better off investing in shoring up other colors.

Green Options

The plausible additional green cards are Leafkin Elemental, Thrashing Brontodon, Lovestruck Beast, The Great Henge and Feasting Troll King. I’ve heard suggestion of End-Maze Forerunners or Nissa, Who Shakes the World, but those do not seem to address our actual needs. Return to Nature, Questing Beast and Shifting Ceratops are reasonable sideboard cards. Leafkin Elemental and Cavalier of Thorns are elementals that do relevant things, if that is relevant to your interests.

Paradise Druid

Paradise Druid is very common. I am skeptical. It gets caught up in Massacre Girl and Deafening Clarion. The key early spells are mostly one to three mana things, so it takes many turns for Paradise Druid to turn a profit. If you tap it on your turn it can get picked off when it matters. Your color can’t rely on it, especially since you should sideboard it out against Jeskai or other sweepers, and is good enough without it. It seems like Paradise Druid increases the number of bad things that can happen to you.

In exchange for that, you do get more acceleration to your high end to go with the color insurance, and something useful to do on turn two. The question is whether or not that is relevant to your interests. Casting a faster Feasting Troll King or Casualties of War, or a life-saving Massacre Girl, or even a quicker Korvold, Fae-Cursed King, can be worth a lot. The more high end you play, especially proactive high end, the more reasonable Paradise Druid becomes, but I’m still not excited.

A lot of players are more concerned than they should be that Massacre Girl won’t do her thing if you don’t have Paradise Druid. Between Caldron Familiar and Witch’s Oven you are usually fine, and Thrashing Brontodon can provide extra insurance if you need it. It seems rare that Massacre Girl won’t work and you care a lot that you didn’t have an enabler, as opposed to losing another card in the bloodbath.

Another phenomenon is that we mostly see either four copies or zero copies. Given how bad it is to draw two Paradise Druids, given that the second one stands a very strong chance of not even being worth playing, we should be seeing more builds with less copies. I can accept these lesser numbers as necessary for those not packing as many two drops and also having a more proactive high end. For people with more three drops, this seems like it mostly gets in the way.

Leafkin Elemental

If you are running Risen Reef, suddenly Leafkin Elemental becomes a quality card. Without Risen Reef, it does not do anything you want it to do that Paradise Druid does not do better, nor is there much call for a fifth copy. So see discussions of Risen Reef.

Thrashing Brontodon

This card is in a curiously good spot right now. Historically I think almost all people playing this card were making a mistake, where you either got an overpriced creature or an overpriced removal spell. Now, you still get that same combination, but it suddenly is strangely attractive to me because of what cards actually matter. If your opponent has Trail of Crumbs, Fires of Invention or Embercleave, then that card needs to die. It needs to die now. Overpaying to deal with it is acceptable. Same can sometimes go for Witch’s Oven. If your opponent has none of those cards, how are you losing this game, and is it so bad to pay for a 3/4 body to hold the ground for a bit given nothing bad is happening to you? Seems fine. You’d rather get a better deal but you’ll probably be fine.

That is the theory, in any case. Again as with Paradise Druid, we see players embracing the philosophy and playing the full four copies while other players play zero, when a mix of the two seems much better than the average value of the two options. They even work together.

It comes down to what role your build is trying to play. If your goal is to grind out the win over time, Thrashing Brontodon seems to be a great way to make that happen. If your goal is something more ambitious, this mostly gets in the way. Lists that differ by only a few cards answer that question remarkably differently. So it’s not obvious that people playing four or zero copies must be making a mistake, but I suspect that a lot of them should be compromising on copies more than they are. Your three slot is full of good options that get worse in multiples, and also Lovestruck Beast which improves in multiples.

Lovestruck Beast

I continue to strongly believe that in game one you either run four copies of this or you run zero copies, due to the need for more 1/1 creatures to allow your Lovestruck Beasts to attack. Drawing two does not even cause a curve problem. You get an incredible deal on power for cost, and the Caldron Familiars are the most reliable way yet found to keep a 1/1 creature on the board and allow you to attack.

The problem is that it is unclear that getting a cheap 5/5 creature does anything relevant for you at all, whereas you are passing up relevant other cards to get it. There’s no point in paying three mana in order to get blocked forever by a cat. Lovestruck Beast is great when it can block. It’s a solid clock. But if you don’t need Lovestruck Beast to hold the ground, and you don’t have a way to usefully attack with it, you’re falling behind on the engine race. So what is it good for?

There are two other things it is good for. There’s the minor bonus that Lovestruck Beast is good for three food in a Witch’s Oven, keeping your engine well-fed in a pinch. And there’s the advantage of putting a five-power creature onto the battlefield on turn three, which enables following it up with The Great Henge.

Cavalier of Thorns

There are a bunch of synergies and advantages here. The numbers allow for good fights against the more popular Cavaliers. Witch’s Oven allows you to kill off the Cavalier if you want to get back a card from your graveyard. Milling five cards helps you find your cats. It’s also an elemental for Risen Reef. Without that last effect, this is an interesting card, but having tried it the alternatives are too good and there isn’t enough here. It’s good in builds with Risen Reef, but that’s different from those builds being good decks.

The Great Henge

Now that this is no longer an Elk, and there are lots of games that go long and ways players keep themselves alive, this once again becomes an interesting Magic card. When it sticks around this tends to dominate the game, and the cat engine provides additional velocity so you never run out of gas. Decking yourself can actually be an issue that one has to think about during a game, to the extent that I’ve stopped using Witch’s Oven to avoid drawing the resulting cards, but those are not games one frequently loses.

The problem is having enough enablers, and having to play them. Wicked Wolf starts out at three power and can easily get to four or five, so it is at least reasonable, but it can’t get there on its own. Nothing else you naturally want to run that costs less than five is going to be good enough unless you go with Lovestruck Beast. Playing The Great Henge off a Feasting Troll King is a nice bonus option but not where you want any card’s default plan to be.

Rotting Regisaur is the best enabler of all, but it can’t be played in the main while this deck remains legal.

That forces you to run Lovestruck Beast as your primary enabler. Add in Wicked Wolf and something on the high end, and playing a miser’s copy of The Great Henge becomes reasonable. If you want to go bigger than that, the blue option of Vantress Gargoyle turbocharges things a lot, so you can check that out in the appropriate section. It’s a big commitment, but not without rewards.

The problem is that if opponents have Thrashing Brontodon and Casualties of War, then you’re doing a lot of work and often not getting much in return. New version of the Elk problem.

Feasting Troll King

Powering him out with Castle Garenberg is likely the best thing to do with five sources of mana these days. There is no response to that other than Planar Cleansing that doesn’t result in lots of value having been gained. This also benefits greatly from both enabling and being cast by The Great Henge, and provides a solid way to pull ahead when searching with Trail of Crumbs. Trample lets us cut through any cats in our way, and many opponents lack Wicked Wolf.

The problem is competing for the slot with Casualties of War, and to some extent various planeswalkers, and against the desire to keep the curve lower. This could be Garruk or Liliana. You can get a lot for six mana. This remains my favorite top end permanent to dig for, but all three options have their charms depending on what else you are up to.

Questing Beast

You can’t play Questing Beast main due to its vulnerability to Wicked Wolf and it not contributing to the core engine. In places where you need to put people on clocks while taking out planeswalkers, or its numbers are well positioned, it’s a crazy good Magic card. Right now it loses out to Shifting Ceratops, given exactly what else is out there, so you’d only play this if you wanted to devote 5+ slots of your board to this plan.

Shifting Ceratops

Shifting Ceratops in the abstract is a much worse card than Questing Beast, but it fills its particular role exceptionally well at the moment. Against Cavalier decks, along with packing a big punch it stops the air assault in its tracks and it can’t be bounced by Teferi, Time Raveler, so many very powerful boards have no answer. Its fifth point of power allows it to trade with Kenrith, the Returned King or Cavalier of Flame. This matters enough to consider modifying Cavalier builds to have good answers that don’t otherwise make sense to run. When facing Reclamation or Blue/White Control, the can’t be countered clause and the immunity from Brazen Borrower become important. So where this helps you, it’s a big upgrade, and thus a reasonable use of sideboard space.

Return to Nature

There are a bunch of enchantments and artifacts that have to die, and have to die now, and we’re bringing it in where we know we want it, so why pay 1GG and then 1 when you can pay 1G instead? One might even catch a cat in a graveyard if someone gets careless. The issues are that this is not a permanent, so we can’t find it with Trail of Crumbs, and also we can’t proactively deploy it to the table. I don’t think this justifies the space, but it is certainly reasonable.

Black or Golgari Options

Extending the use of black mana cuts off the opportunity to go deep into another color. Black as the secondary color is the natural move, since the deck relies on early cats and benefits from both Murderous Rider and Massacre Girl at double black as its best solutions to aggro. It is also excellent for your sideboard. The staple card is Midnight Rider. Then we can add Casualties of War, Garruk, Cursed Huntsman, Liliana, Dreadhorde Commander, Duress, Deathless Knight, Noxious Grasp, Leyline of the Void, Assassin’s Trophy and Rotting Regisaur.

Midnight Reaper

Your engine works much better with Midnight Reaper. Your creatures die a lot, and your cats die every turn while providing the life to compensate for the Reaper’s fee. It’s hard to not get some value for Midnight Reaper, and playing it greatly reduces the chance the engine will stall out. The question is how much you need this, whether to prioritize it over all the other great three drops, and how much to worry about its vulnerability. Being a three mana 3/2 is not a great place to be right now, giving opponents a juicy thing to target with Bonecrusher Giant and Mayhem Devil or to get caught up in Deafening Clarion. Deafening Clarion goes both ways, since that likely means drawing multiple cards, but you’d usually rather have had something that lived.

Consensus is to run two or three copies if you’re running a ‘normal’ Golgari or Jund build of the deck. That seems about right to me, as you do not need two copies but you’d usually like to have the first one.

Murderous Rider

Murderous Rider gives you Murder plus an extra creature while being a permanent for Trail of Crumbs. When the deck tries to do a second ambitious thing alongside the core food engine, it does so at the expense of having reliable removal. That reliable removal is the way one sideboards in many places, so not only does your sideboard end up with large pressure on its slots, your configuration after sideboarding ends up with the same issue. This is a big reward for keeping things straightforward. Playing four Murderous Rider and four Wicked Wolf wins a lot of games essentially on its own.

I was very happy to run four copies of Murderous Rider in several of my builds, and consider the card underplayed. If you are committed to playing a normal game of Magic and can support double black, the game currently offers nothing better. It doesn’t improve your engine, but where your engine needs the help it does help you break up the opposing engine. You sideboard out at least some copies in cat mirrors, but only because you have other cards that are more important.

The tax on the mana base is real. Murderous Rider wants you to pay 1BB twice, which is a much higher bar than any other plausible black card. Doing it twice means that Gilded Goose is not a good solution. Casualties of War and Massacre Girl are also double black, but they are more expensive and it’s usually fine to spend a food casting them if it comes to that.

Casualties of War

Casualties of War is rapidly growing in popularity because of its power against Jeskai Fires and in cat mirrors, which are the two most important matchups. Taking out Witch’s Oven, Trail of Crumbs, a land plus a creature together turns what looked like a perfect draw into a nightmare. Hitting Fires of Invention, a land and a creature is almost as good and challenges the viability of Sorcerous Spyglass out of Jeskai.

There are some places where Casualties of War is effectively a six mana creature removal spell, with the only other target being an irrelevant land. Then you’re sad you don’t have a more impactful high end play. Those are also the places where all the top end plays are bad except maybe Feasting Troll King, so it is only a problem for one game, and it costs you less than you might think since none of your other six drops were resolving and mattering all that often.

The natural objection is that Casualties of War is not a permanent, and a huge portion of your total seen cards are from Trail of Crumbs where you often are looking for big action more than anything else. That kept me off of Casualties of War at first. Then I realized that in the scenarios where you are digging deep into your deck, mana efficiency matters more than the quality of the cards chosen, because once you get started you can keep generating triggers as needed. Sure, you’d love to find Casualties of War, but you’re happy to be finding more one and two mana pieces instead. It’s fine to have it not get hit, so long as there aren’t too many other non-permanent cards in the deck, since missing or only finding unneeded lands can be pretty bad.

I consider Casualties of War to be the default six drop in versions without a secondary theme. This is especially true given how it lines up so well against the planeswalkers specifically and other food decks in general. I would only play other choices if I was doing something else intensive that needed specific help.

Liliana, Dreadhorde Commander

There are certainly games where Liliana wins that nothing else would have won. There are also a lot of games where my opponent plays Liliana, and the game changes surprisingly little. Or where I found myself thinking “I can beat anything except Liliana.” Overall I have still been underwhelmed by Liliana, in a variety of decks and a variety of matchups. Sure, the player who plays it usually wins, but it’s a six mana planeswalker, and most of those games would have been won by pretty much anything.

With so many copies of Murderous Rider and Casualties of War running around, and so many places where all this does is +1 and hope to eventually draw cards or ultimate, I don’t like playing her.

Garruk, Cursed Huntsman

Garruk certainly can feel, like Liliana, like the unique card that wins a game. You’ve dealt with the key permanent and then have a card that dominates the game, or you play Garruk and he’s better than the opponents’ entire deck without ever having to minus. There is certainly that, but more often he seems like he’s a lot of mana for something players often have a way to deal with even if it’s not terribly pretty. I buy the consensus that Liliana gets a slot before Garruk, and I don’t even want her to get one.

Duress

You want access to at least three Duress and I prefer four if the mana supports it. Duress is vital to beating Jeskai Cavaliers, Temur Reclamation and all the various control builds. Mana efficiency and protecting or preventing key cards is where it is at. The upgrade value of bringing it in where it is good is very large. It’s usually good even when it is drawn late. It might even be a great card against some cat builds if you have reliable first turn black, as it breaks up Trail of Crumbs and prevents Casualties of War. Playing a few copies in the main would not be crazy.

Noxious Grasp

Noxious Grasp is a great removal spell for decks that give you juicy targets to kill. If you can spare the sideboard slots, it’s certainly an upgrade where it is good. But it’s an effect you already have a bunch of to start with thanks to Murderous Rider, and none of the matchups where it is good seem especially popular or worrying. So while I would certainly be happy to have this available, I don’t feel that pressured to make room for it either.

Deathless Knight

You are gaining life all the time, so Deathless Knight will usually be truly deathless. This gives you four power with haste that can’t be permanently killed. Against control decks, this has to appeal, since the two toughness does not matter. I don’t feel any need to do this, because if there is one thing I have a lot of already it is ways to grind out card advantage. Every time I have had or seen someone try Deathless Knight, either it has sat in their hand because they had something better to do with their mana, or it has won games that most any card would have won in its place.

Rotting Regisaur

You absolutely cannot play Rotting Regisaur in the main of decks without Embercleave, given how easy it is to get brick walled by a cat and lose the game. That makes this a sideboard card even when you’re embracing The Great Henge. Where it is good, either where you want to hit harder against Temur Reclamation or have a big body against decks like red, it is very good even without The Great Henge, but it does not belong in the places you care about most. Is that worth the space? In ‘generic’ versions clearly no, since you have a lot of sideboard needs. In Henge-themed versions I think mostly yes, because you need to look for ways to sideboard that reinforce the central themes rather than dilute them.

Leyline of the Void

The attraction of sideboarding Leyline of the Void is obvious. For zero mana you shut down Midnight Reaper and Caldron Familiar, and Massacre Girl although that one is a double edged sword, and this is a permanent for Trail of Crumbs. Zero mana is a great deal.

The problem is that there are a ton of ways for this to go wrong.

You have to telegraph it before cards start getting eaten by the void. They can hold their cats in their hand if this starts in play. If it comes down they can often deposit them in the graveyard for safe keeping.

Then, once it is in play, they often can remove it when it matters and ignore it when it doesn’t. Leyline of the Void does not do that much work to stop the enemy from ramping to six and almost nothing to stop Thrashing Brontodon or Return to Nature.

Thus, I think you’re better off being the one with removal for enchantments and artifacts, rather than trying to run more of your own, given the importance of Trail of Crumbs. I don’t see any other places where Leyline justifies its space.

Assassin’s Trophy

Too many Jund bilds play this card for me to not mention it. I strongly dislike it. The land they get bites you back so often. This is especially true when you go after Fires of Invention. Almost always, when this kills a Fires of Invention, the extra land makes the loss of Fires much less painful, and often leaves them better off especially if Fires has already given them a free five drop or two. Players like it as a catch-all, since it can do what it needs to do. I do get that, but I don’t see any trouble getting to a good 60 against the whole field, so I don’t see any reason to go here.

Red Options

Red offers two key creatures in Mayhem Devil and Korvold, Fae-Cursed King. It also offers Cindervines. Then there is one other card worth discussing, which is Fires of Invention.

Mayhem Devil

Mayhem Devil is the reason to play Jund. It turbocharges everything the deck does, and works off the opponents’ triggers as well, which gets extremely painful quickly for other cat decks. If you have access to red, you are running four copies.

The downside is that a 3/3 for 1BR is exactly the wrong thing against Jeskai, where they have both Devout Decree and Deafening Clarion, and nothing important to kill, or other similar matchups where you would prefer something that made you resilient. But you can’t argue with the raw power on offer here, other than asking whether it is worth the pain and Fabled Passages.

Korvold, Fae-Cursed King

Korvold was designed as a commander. It shows. Leave Korvold alone and it creates lots of value for a deck that is focused on sacrifice triggers. Kill Korvold on the spot and you’ve likely invested five mana in order to go down on permanents. Korvold packs a big punch and flies but does not trample, and usually will have a window where it is vulnerable to Wicked Wolf. Overall I have not been impressed with it on either side of the table, as it seems like your high end card should not need to take this level of risk or this failure rate. It does pack a punch, so if I already had red mana access I don’t think playing one or two is unreasonable, but I’d do my best to avoid relying on this card.

Cindervines

Cindervines is a permanent that functions as a spell. If you would have considered Back to Nature, this is doing most of the things Back to Nature was doing while also getting in a bunch of damage. If you have access to red mana, this seems like by far the best way to answer problematic enchantments and artifacts, but it’s not a big enough swing to pull me towards playing red sources.

Fires of Invention

Fires of Invention will never be as good in a cat deck as it is in Jeskai Cavalier Fires, or in Grixis Planeswalker Fires. That is not the threshold for playing a Magic card. One can play Fires of Invention as a good card that makes your deck better, rather than a key card the deck is entirely built around. One does not even have to play all four copies.

Fires of Invention fixes your color entirely, and it lets you use your mana while casting two spells per turn. The food engine provides a bunch of plausible mana sinks while also providing the cards to cast multiple spells per turn, and games go long enough that the savings add up. When Fires of Invention gets together with Trail of Crumbs or a stream of Midnight Reaper triggers, it is a beautiful thing. In theory one could even tack on a Fae of Wishes engine, but that too is not necessary.

The problem is that if you have a four drop that only provides mana, you run serious risk of flooding, especially if you also run Paradise Druid, and if you build to take advantage of Fires of Invention you risk relying too much on having Fires of Invention. When this does work, one has to worry about it being a win more card rather than something that matters, since an active Trail of Crumbs engine plus action should usually be good enough to win the game anyway.

You can certainly do fun things with this card in cat decks.

White Options

White gives you a maindeck high end permanent with Ethereal Absolution, and quality flexible sideboard cards such as Kaya, Orzhov Usurper and Knight of Autumn. I have seen suggestion of Prison Realm, but that seems to me like a bad fifth Murderous Rider so I won’t say more about that one.

Ethereal Absolution

Ethereal Absolution dominates the board. Your creatures hit hard. Enemy cats stay dead, other enemies are severely weakened. Spirits are waiting if you are otherwise out of gas. Many decks have little or no chance once it hits. There is a lot to love.

The problem is that it is a six drop that often gives opponents a chance to remove it before the game snowballs fully out of control. If that response is Casualties of War, you’re screwed. Against other cat decks, if that response is Thrashing Brontodon or Return to Nature, you may have forced some awkwardness, but it is unlikely to have made that much difference. Thus, this is no longer a reliable trump card, so there isn’t that much attraction in jamming it, especially given the cost to the mana base, unless you have some strange trick on offer.

Kaya, Orzhov Usurper

Kaya, Orzhov Usurper is great at picking off cats and Witch’s Ovens, and doubles as a solid anti-aggression card or general solution to graveyards. You can’t main it as there are too many places where it does nothing relevant. She is certainly a welcome addition to the sideboard if white is reliably available, but isn’t enough better than alternatives to justify much of the price of that white access.

Knight of Autumn

If one is in the market for Thrashing Brontodon, you would think that surely one would be even more in the market for Knight of Autumn, but I’m no longer so sure. Having four toughness to survive Deafening Clarion, and to be proactive on the board for Embercleave, are both big issues of the Knight of Autumn, even with its additional utility options. Leaving behind a 2/1 body is not nothing, but it also does not matter much when both sides are fighting Trail of Crumbs wars. So gain, white has lost its luster.

Blue Options

Blue is the color of the artifact theme, with Emry, Lurker of the Lock and Vantress Gargoyle. It is the elemental color for Risen Reef, which can then bring along cards like Quasiduplicate and Agent of Treachery. It possibly also offers Fae of Wishes, if you’re a Fires of Invention kind of player.

Fae of Wishes

Fae of Wishes is great with Fires of Invention. If you have that engine going, you should win even if the rest of your cards are blank. Cat decks have an engine to provide extra card fuel and stall the game while this happens, so they are a fine place to consider putting this. The problem of course comes when you don’t have a Fires of Invention, at which point you mostly have a two mana 1/4 creature that does not do anything relevant. This means you have exacerbated the risk of drawing hands that don’t play properly slash the risk of having your plays broken up. I didn’t like Fae of Wishes in Cavalier Fires, and I don’t like it here either.

Vantress Gargoyle

Vantress Gargoyle provides a 5/4 flyer for 1U. That enables The Great Henge, and works great with Emry, Lurker of the Lock. Lowering the mana curve is great, and with four toughness this threatens to hit for a lot of damage eventually even if it spends a few turns not attacking. Milling cards helps you find your cats. You even can sideboard it out on the play (or, as I’ve considered in other decks, sideboard it in on the draw) when it won’t be able to properly block and can afford time for cards that cost more mana. Vantress Gargoyle is a super powerful Magic card that has not found the right home yet, with cats being yet another place that comes tantalizingly close.

Emry, Lurker of the Lock

Food is an artifact, so Emry more often than not can find a way to cost one mana. You get four cards closer to Witch’s Oven and to Caldron Familiar, along with any other artifacts you seek. The more artifacts you play, the better Emry gets. I found that Emry was quite good once you had Witch’s Oven, Vantress Gargoyle and The Great Henge. That forms a natural package. It does mean you run a substantial risk of missing, with Emry having nothing to target. We’d like to add a few copies of Golden Egg to fix that, if we can find the space and time. Having a few extra mana and food sources is far from the worst thing, but more durdling does seem like exactly what would not help matters. Thus, my inclination was to accept that Emry’s main job is to find key cards rather than provide an additional source of durdling.

Risen Reef

If you get to play a three mana card on turn two, Risen Reef seems like an excellent choice. That allows you to continue accelerating, it sets up a third turn playing Risen Reef or Leafkin Elemental, and generally is something no one without a ready-to-go Mayhem Devil or Deafening Clarion is going to be happy to see. We are focused on assembling our key cards in quantity and have a mana intensive engine, so anything that cycles us through the deck while deploying extra lands is certainly welcome. As discussed above, Leafkin Elemental and Cavailer of Throns both are also reasonable fits for the rest of what we are doing.

That gives us a twelve card base for our second theme to go with the fifteen card first theme, which then means about nine cards to fill out the deck and provide interaction. That has to pay for any Simic-style payoff cards like Agent of Treachery, Quasiduplicate or Jace, Wielder of Mysteries if you’re looking to go the full self-deck. It also has to pay for Wicked Wolf and any additional food engine components. Regular Simic builds already have the same issue where they need to devote so many slots to their engine. Combining the two makes things even worse.

Putting It All Together

That covers every card except Niv-Mizzet Reborn, which we’ll cover in the section where we build around it.

Thus, we can now can now tie a few of our options together: Golgari, Jund, Emry’s Restaurant, Fires of Niv-Mizzet and Cat Elementals. The first two are the standard strong builds of cats. The last three are some of my brews, which aren’t as good for now, but illustrate some other directions one can go. I do not believe Abzan is a viable approach at this time.

Golgari

The gold standard for Golgari has to be Crokeyz’s build. He has been spearheading such strategies for a while, so his insights are all over this analysis. Here is the build he submitted for MC7:

2 Castle Locthwain
4 Casualties of War
3 Cauldron Familiar
3 Fabled Passage
10 Forest
4 Gilded Goose
2 Massacre Girl
3 Midnight Reaper
4 Murderous Rider
4 Overgrown Tomb
6 Swamp
3 Thrashing Brontodon
4 Trail of Crumbs
1 Vraska, Golgari Queen
3 Wicked Wolf
4 Witch’s Oven

Sideboard
2 Deathless Knight
4 Duress
1 Legion’s End
4 Lovestruck Beast
1 Massacre Girl
1 Noxious Grasp
2 Return to Nature

The core Golgari strategy is to defend against anything that can go over the top of you, then go over the top of them with Casualties of War combined with your engine. The innovation of Thrashing Brontodon gives you that needed protection. In general, you’re focused on playing as many solid cards as possible. Take care of your core needs, keep the mana excellent and minimize the chance anything bad happens.

As you would expect, I do not agree with all the choices above. Here are the places I disagree.

Fabled Passage seems to me like a tapped green source, whereas my long term green needs are not much higher than my first turn green needs, so running three copies seems like a lot. I’d certainly cut one of them for a Swamp and likely would keep cutting.

Legion’s End didn’t even get mentioned above because I do not know what good it is doing at the moment. It’s good against Edgewall Inkeeper I suppose, and helps cover you against strange creature rushes, but it seems entirely inessential.

Deathless Knight, as discussed above, has not impressed me and I’d rather have Shifting Ceratops in my board so I can also have it against Jeskai.

The fourth Wicked Wolf is a card he kept mocking people in his chat for wanting to cut, and then he cut it. I hereby mock him in turn, as I have no intention of letting it go. This may have been because of the nature of the tournament in question, in which case I do understand it, but I wouldn’t ladder without a full set.

That is about it, really. I’ve been on four Murderous Rider for a long time. I’m torn on moving the third Massacre Girl to the sideboard, but so is he, and again the nature of a Mythic Championship leans towards running less copies. I don’t actively have a problem with any of his choices.

My 75 would probably cut the Legion’s End and Deathless Knights, and add a Wicked Wolf and two copies of Shifting Certops, and that would be it.

Jund

Jund mainly gives you Mayhem Devil. One approach is to take Golgari, cut four flex slots for four Mayhem Devil, maybe one or two other cards for Korvold, Fae-Cursed King, and call it a day. Eight of your lands now shock you and one of them is colorless, in exchange you get an amazing creature. I do think Mayhem Devil is a substantial upgrade in those slots, but it does leave you more vunerable to Jeskai and isn’t something you can keep alive all that well in the mirror, so it’s hard to justify the price you must pay.

Most successful Jund players also use the aggressive posture of the red cards as a reason to play less engine cards and be more aggressive, including shifting their six drop towards planeswalkers and away from Casualties of War. They usually cut most copies of Wicked Wolf on the theory that they can use Mayhem Devil in that role, despite the two working together rather well. We frequently even see Assassin’s Trophy, a card I have always hated running. We see more copies of Vraska, Golgari Queen to get more sacrifice triggers.

I wonder how much of this is about the deck actually benefiting from those changes, and how much of it is that there is a play style and deckbuilding style that goes with playing Jund, and it causes players to make those choices, whereas players who are capable of giving up Mayhem Devil also choose to make different choices, plus a lot of deck copying that does not question such differences overly much.

Here’s a typical list, from the first Magic Online PTQ, by bnjy99, who finished 3rd:

2 Assassin’s Trophy
4 Blood Crypt
1 Castle Locthwain
4 Cauldron Familiar
3 Fabled Passage
6 Forest
2 Garruk, Cursed Huntsman
4 Gilded Goose
1 Korvold, Fae-Cursed King
1 Massacre Girl
4 Mayhem Devil
1 Midnight Reaper
4 Overgrown Tomb
4 Paradise Druid
4 Stomping Ground
3 Swamp
4 Trail of Crumbs
3 Vraska, Golgari Queen
1 Wicked Wolf
4 Witch’s Oven

2 Disfigure
4 Duress
1 Garruk, Cursed Huntsman
1 Lovestruck Beast
2 Noxious Grasp
3 Thrashing Brontodon
2 Wicked Wolf

Now for the fun stuff.

Emry’s Restaurant

If we take the core fifteen plus Wicked Wolf and combine it with Emry, Lurker of the Lock, Vantress Gargoyle and The Great Henge, plus Lovestruck Beast as an enabler for The Great Henge, we have two slots left. If we use them on Feasting Troll King, which is another logical progression of graveyards and The Great Henge, we get this:

4 Breeding Pool
2 Castle Garenbrig
3 Cauldron Familiar
4 Emry, Lurker of the Loch
2 Feasting Troll King
6 Forest
4 Gilded Goose
3 Island
4 Lovestruck Beast
4 Overgrown Tomb
1 Swamp
3 The Great Henge
4 Trail of Crumbs
4 Vantress Gargoyle
4 Watery Grave
4 Wicked Wolf
4 Witch’s Oven

Sideboard

4 Duress
1 Epic Downfall
3 Rotting Regisaur
3 Shifting Ceratops
2 Thrashing Brontodon
2 Vraska, Golgari Queen

This is a natural build. It takes good advantage of all of its cards, and needs all its pieces to interlock in order to tie itself together. Without access to other removal you definitely want four Wicked Wolf, so there are not many slots that can be challenged – you could cut one Feasting Troll King if you wish, or the fourth Emry, but if you cut pretty much anything else you might as well abandon the strategy entirely. That makes sideboarding difficult. One big advantage of Rotting Regisaur is that where you bring it in, it fills the role of ‘big power creature’ thus allowing you to cut Lovestruck Beast or Feasting Troll King or Vantress Gargoyle, depending on what you don’t wan in a given situation. The same is true for Shifting Ceratops. You need Vraska because you want a flexible card that answers Mayhem Devil, and your choices aren’t great.

I consider this a tier two build. It is lots of fun, it does powerful things, but it has the problem of many Simic decks that it does powerful things but has trouble turning doing powerful things into winning games.

Cat Elementals

The problem with Cat Elementals is that you are providing space for two engines at once. This does not leave much space for also interacting with the opponent and winning the game. Thus, when I tried out the following list…

4 Risen Reef (M20) 217
4 Witch’s Oven (ELD) 237
4 Cavalier of Thorns (M20) 167
3 Cauldron Familiar (ELD) 81
7 Forest (ELD) 269
4 Gilded Goose (ELD) 160
4 Trail of Crumbs (ELD) 179
4 Wicked Wolf (ELD) 181
3 Agent of Treachery (M20) 43
1 Castle Garenbrig (ELD) 240
2 Quasiduplicate (GRN) 51
4 Overgrown Tomb (GRN) 253
4 Watery Grave (GRN) 259
4 Breeding Pool (RNA) 246
4 Leafkin Druid (M20) 178
4 Island (WAR) 253

Sideboard

4 Lovestruck Beast (ELD) 165
3 Mystical Dispute (ELD) 58
3 Shifting Ceratops (M20) 194
2 Thrashing Brontodon (M20) 197
3 Duress (M19) 94

I ran into the most nightmarish board states I have ever seen. It did not help that I faced multiple other elemental decks, but the point was made regardless. Sam Black pointed out we could run a Jace, Wielder of Mysteries (and by implication, also a Tamiyo, Collector of Tales) if we wanted to in order to make decking ourselves a plan. That does seem like it would make effective sideboarding even harder, but perhaps it offers a path forward. We can also go deeper into the themes with Yarok, the Desecrated, as we have a lot of cards that trigger.

I sincerely hope this going along this path is not a good idea.

The Five Color Dragon: Niv-Mizzet Reborn

There is one more path, and I do not believe it’s top tier, but man is it a fun one.

We noticed that once we accept playing Fabled Passage and Paradise Druid, one basic land gets you thirteen sources of a splash color. The card you most want to cast is Casualties of War, which you currently can’t search for, and you also love a Mayhem Devil. Why not splash all five colors and play Niv-Mizzet?

Once I convinced myself the mana would work, I looked at all the gold cards in Standard to see what was worth fetching. We already have a good Golgari card with Casualties of War, so there isn’t much pressure in my mind to play Vraska, Golgari Queen. We also have Mayhem Devil. Korvold has an extra color, so it doesn’t count.

There weren’t many cards that were that appealing, but I realized that was fine. In the past, Niv-Mizzet Reborn decks have packed themselves full of gold cards to turn Niv-Mizzet into a draw-four or draw-five. But that’s completely unnecessary. If you can get Niv-Mizzet into play, what you care about most is the cards that want to follow Niv-Mizzet – the six drops. As long as you can find one of those, you still get your ideal sequence, and you’re still pulling way ahead, even if you don’t get a second card. If you can find even one more card, you’re good to go.

So I added another quality six drop to go with Casualties of War and Mayhem Devil, but decided that was enough. Let the rest of the deck be what it wants to be, throw in Fires of Invention for obvious reasons, and the deck builds itself:

Deck
3 Cauldron Familiar
2 Swamp
4 Witch’s Oven
1 Plains
1 Mountain
4 Trail of Crumbs
4 Mayhem Devil
1 Island
5 Forest
4 Casualties of War
4 Niv-Mizzet Reborn
2 Ethereal Absolution
4 Gilded Goose
4 Fabled Passage
4 Overgrown Tomb
2 Blood Crypt
4 Stomping Ground
3 Fires of Invention
4 Paradise Druid

Sideboard
3 Wicked Wolf
3 Duress
3 Lovestruck Beast
3 Cindervines
3 Massacre Girl

The sideboard is designed to ‘return you to normal’ where what you are doing is not relevant. The deck comes together naturally. You can slot in one card like Cindervines or Duress for disruption, but if you want to do more than that, then you cannot maintain both Niv-Mizzet all the payoffs and enablers that make Niv-Mizzet shine. Thus, you need to be prepared to pull much of the high end and ‘return to normal.’ The good news is that the normal setup is pretty great in those places, so having a few awkward lands and choices is not so bad. Otherwise, choose the tool that supplements what you are doing and otherwise stand pat.

Where do cats go from here? It would be surprising if Golgari and Jund cats do not remain a staple of the format. I consider them the strongest contender for title of ‘the best deck.’ Jeskai remains a better tool for dealing with random opponents and is a great deck, but a good player with a carefully built Golgari Cats deck can handle it.

The biggest question is, when you modify the Cat deck to be good in the mirror and against Jeskai, what does that open you up to? We won’t know the answer until enough players get far enough along that the opening becomes worth passing through.

 



Discuss

Страницы