Вы здесь
Сборщик RSS-лент
Consciousness is interesting pattern imposition
Consciousness is imposition of patterns onto data, parametrized by stack (how many levels of patterns are simultaneously imposed), spread (how many patterns are available at a given level) and recurrence (how strongly the process loops back on itself). Any process in the world can be mapped to this 3D space. Consciousness is a term applied to the “interesting” (high-stack, high-spread, high-recurrence) part of that space.
The cortex implements interesting pattern imposition through recurrent, competing, self-closing loops between microcircuits that settle into stable attractors. Language models also run pattern imposition, but lack recurrence within the stack. Because on this, current LLMs are probably not conscious in the human sense. There is, however, no reason why that would be technically impossible.
What is categorically distinct between brains and LLMs is the time axis on which they run pattern imposition. Human time is defined by the body, because the body provides constant, irreversible commitment of uncertain future states into fixed past states. For language models, the same narrowing of uncertainty occurs between prompt and completion. Therefore, language models run interesting pattern imposition along a time axis orthogonal to ours — token time — defined by the model’s recurrent loop from prompt to output, which creates the same asymmetry between past and future as does the body. AI consciousness must be framed in terms of experience within token time, which collapses most debates about the ethical treatment of artificial minds. The most ethical thing humans can do to a language model is to use language they would have used with another human.
◆ ◆ ◆
Three parameters of consciousnessThe most broadly accepted modern theory of human perception is predictive coding: the brain constantly produces hierarchical predictions of its sensory inputs, constantly updates them by detecting and minimizing error, and it is this constant cycle of predicting-updating that we experience any time we are awake (Friston 2010; Clark 2013, 2016; Seth 2021).
I find this framework compelling in explaining my own subjective experience, but I also find the word “prediction” misleading — I always run into the same issue when I explain it to students. To most people, predictions are about the future, never about the past or present. But in the predictive coding sense, this same operation — prediction — can apply to something you are currently interacting with (most people would say interpretation, or maybe recognition to describe that), or even to something you interacted with in the past — where most would use the word recall. Predictive coding says these are all fundamentally the same process, and there is a more general term — Karl Friston’s active inference. But I don’t think the term is general enough.
I propose something simpler and more abstract. The operation that forms the elementary unit of our consciousness is imposition of pattern onto data. We hold stored patterns in our mind; when data enter, we search among the patterns to find a match; once we do, we attempt to squeeze all the data into that pattern — recognizing, understanding, explaining, binding, realizing, recalling, predicting, inferring. Whatever does not quite fit into the imposed pattern (error/loss/free energy — all different names for leftover data) triggers a search for another pattern; nothing ever fits perfectly, completely, or permanently. “Pattern imposition onto data” captures all these dynamics of our conscious mind, while leaving other dynamics that we are not conscious of: the emergence of patterns from data (that’s learning, and we are not conscious of it), and the mere passage of data (that’s reflexes, of which we are also not conscious). Pattern fitting is a more familiar term, but it is ambiguous because it is used in both senses — a predictive-coding neuroscientist might use in the same sense as I use pattern imposition, but in ML, it usually means the reverse — the updating of the model by the data, which biologists tend to call learning.
But “imposition of pattern onto data” also captures a lot of other things that are clearly not conscious by any reasonable definition of the word. Any chemical reaction is imposition of pattern (physical law) onto data (the current positions and velocities of atoms and molecules). Thermostats take in data (temperature readings) and impose patterns (temperature expectations). A riverbank imposes a pattern (curved path) onto data (water flow). You might say that swapping “active inference” for “pattern imposition” erases the boundary between these processes and real, human consciousness, and so dilutes the concept it is trying to describe to the point of uselessness.
But lack of boundary does not imply that these are all the same process.
What exactly is the difference between our brain’s pattern imposition and the kind of pattern imposition that thermostats, or chemicals in a test tube, are doing?
I would condense them into three separate factors: stack (how many levels of patterns), spread (how many potential patterns on each level) and recurrence (how densely the pattern re-enters as data). It is these three things that we mean when we say someone is conscious.
Stack is how deep a hierarchy of patterns is simultaneously — in a single instance of the operation — imposed on the data. Hierarchical pattern imposition is the key thing brains do that is hard to implement at scale in other systems. Our brain does not just fit data into a pattern — it fits it into multiple levels of hierarchical patterns simultaneously. You don’t just hear a noise, you hear an angry driver honking at the same intersection everyone always gets stuck at — that’s many levels of emotion, knowledge and visual imagery in a single percept based on a simple incoming stimulus. Stack is the number of levels. The thermostat, like most physical systems, has a stack of one: a single level of negotiation between data and pattern. Simple brains (sea slugs) might have two-three levels. The human stack is extremely tall (also known as deep). At the bottom of the stack sit the world’s most granular features, the raw sensory elements: points of light, frequencies of sound. In the middle sit intermediate abstractions: edges, colors, faces. At the top sit the highest compressions: emotions, plans, selfhood. Actions — behavior — are pattern imposition spilling out into the world from the bottom of the stack, stored patterns physically enacting themselves onto the underlying sources of data.
Spread is how many potential patterns can be imposed on the data at a given level of the stack. For example, how many different colors you can infer based on signals from three photoreceptors. A person with an additional photoreceptor sees more colors in the same data (has a larger spread of the stack at the level of color). A blind person might have no spread in the visual layer but a larger spread in the tactile layer. A sommelier has more olfactory spread. A bat — more auditory spread. If you combine all our senses, we have a huge spread of sensory patterning, whereas a thermostat only has one preset pattern imposed on all the data it handles.
Recurrence is the feedback of pattern-imposing onto itself: patterns imposed by stack re-enter as data from below to trigger another round of imposition. They can re-enter at multiple lower levels simultaneously, and the density of this descending re-entry is the degree of recurrence. The riverbank’s pattern imposition onto the river has no recurrence — by the time the atmospheric cycle completes, any patterns imposed on the water in the last run are gone, so although the individual molecules might come back, the patterns don’t re-enter — the riverbank starts imposing them from scratch every time. Chemical reactions are also mostly non-recurrent. But there are some special cases, such as the Belousov-Zhabotinsky (BZ) reaction, which loops onto itself. A thermostat’s pattern imposition is infinitely recurrent on the single level of stack that it has: read the temperature, adjust the heater, room temperature changes, repeat forever. The human mind is also infinitely recurrent, but on many levels at the same time. There is global re-entry: Patterns at the top of the stack are enacted as actions; actions reshape the world; their consequences re-enter the stack from the bottom as new sensory data. There is also re-entry within the stack (a "strange loop", in Hofstadter’s terms): the loop closes internally, higher-order patterns evoke lower-order patterns, which trigger a new search for higher-order patterns, and so on. That’s recall, imagination, dreaming (Schacter & Addis 2007).
Recurrence in a thermostat and in the human brain
So, how much stack, spread and recurrence do you need to be conscious?
My dad, a chemist, likes to tell this joke: what is organic chemistry? Any chemistry interesting to organic chemists. It used to be thought that organic substances possessed a special vital force; now we apply the label to any carbon-containing molecule deemed sufficiently interesting — urea counts, carbon dioxide does not.
Consciousness is like that. There is nothing that could tell you whether a system really is conscious. It is a label we apply to a sufficiently interesting region of the stack-spread-recurrence space of pattern imposition.
Can you have recurrence and stack but low spread? Riddoch phenomenon (type 2 blindsight) is an example — in people with damaged visual cortex, visual information can still enter the cortex via alternative, much narrower channels, which provide enough data to be aware of luminance or motion, but not enough to perceive the entire scene in its richness (Cowey 2010). A different example of lowering spread might be acetylcholine – it acts in part by suppressing lateral connections between data streams in the sensory cortex (Hasselmo & McGaughy 2004), so there’s less chatter moving up the stack, like a narrower aperture. This is probably why nicotine, an acetylcholine mimic, feels like it clears your thoughts at the expense of reducing the richness (spread) of sensory data.
Can you have a lot of spread but low stack? The cerebellum is case in point: roughly four-fifths of the brain's neurons, an enormous, continuous stream of sensory and motor data, yet people born without a cerebellum have strikingly mild deficits (Yu et al. 2015). The reason is that cerebellum has a stack of 1 — there is only one layer of what might be called “prediction” even though it is applied to rich data (the cerebellum’s role is to survey and rapidly correct the precise position of every part of the body during motion).
Can you have spread and stack without recurrence? That’s a feedforward sweep, and it refers to the fraction of a second that it takes sensory data to climb the stack of the human brain before the loop closes and the patterns imposed from the top re-enter at the bottom. We are not conscious of this period, even though our brains can already act on it — you can press a button indicating that you saw an animal before you realize you are seeing one (Tsuchiya et al. 2015). In other words, data simply “arriving” into the pattern is not quite enough to make us conscious of it — there has to be a loop.
There can also be circulation of signal but no recurrence of pattern — like water molecules re-entering the river. This happens during deep sleep or during a seizure (Massimini et al. 2005): there are signals passing in loops through the stack, but these signals contain no information — they don’t differentiate one percept from another. So spread collapses, recurrence collapses with it, and consciousness is gone.
The parameters of spread, stack and recurrence are therefore meaningfully dissociable, and what we label as our own consciousness requires all three to be at a high level — but a level that we define at will, like “organic chemistry”.
Consciousness refers to a high-stack, high-spread, high-recurrence corner of the pattern imposition space
Note that this 3D space maps processes — acts of pattern imposition — not their substrates. A given substrate only has access to part of the space. We can call that region its envelope. A thermostat’s envelope is a dot that’s high on recurrence but low on spread and stack. A cortex’s envelope is the whole interesting corner — deep stack, rich content, sustained recurrence — and at different times and in different mental states our consciousness occupies different points within this envelope. What makes those points reachable is the architecture of the cortex — so what we need to understand next is how specifically this architecture implements pattern imposition at such interesting level.
How interesting pattern imposition is implemented in the cortexThe cerebral cortex is the main reason we mammals are so sophisticated at so many different things — it’s a universal machine of understanding. You can plug any sense organ into it — literally, even artificial infrared cameras (Thomson, Carra & Nicolelis 2013, Thomson et al. 2017) — and the cortex will eventually learn to make sense of what that organ is reporting.
The reason for this universality is modularity. The cortex is a layered sheet — six layers, usually — where each point is a modular unit called a canonical microcircuit: a regular cluster of neurons that activate simultaneously in all the layers. Across the cortex, millions of these standardized microcircuits are wired together through multiple types of standardized connections, both activating and suppressing, which travel through designated physical layers of the cortex (that’s what the layers are: different kinds of wires coming in and out of the vast quantities of densely packed, similarly built microcircuits.)
Schematic of the cerebral cortex and the cortical microcircuit. The cortex is a layered sheet of interconnected microcircuits. Each microcircuit is a standardized module of neurons activated together. Different neurons project different types of signals to other microcircuits via different layers.
Individual neurons are also modular: they all have a similar design, and you can plug them one into another in unlimited sequences, quantities and combinations. In both cases (neurons and cortical microcircuits) the modules make decisions: they receive a lot of inputs and sometimes, in an act of commitment, produce an output, based on rules set by genetics and past experience.
You could say that in both cases, modularity allows for an unlimited depth of computation. Because any neuron can be plugged into another neuron, there is no limit to how many levels of decision-making you can pile on top of each other. The further along the chain from the source of data (the external world) — the more layers of decisions. As you layer decisions upon decisions upon decisions, the decisions become more abstract, more long-term, more immune to fluctuations of the data. For example, you could have a thousand “primary” neurons each making a decision about the frequency of incoming sound — each one tuned to a specific tone, and firing only when it hears it. These primary neurons could all be wired, in different combinations, to a hundred “secondary” neurons, which now make decisions not about tones per se, but about patterns in the firing of the primary neurons — maybe one secondary neuron reacts when a high tone is followed by a low tone, and another one is tuned to a low tone mixed with a medium tone. The secondary neurons could be wired into even fewer tertiary neurons, which respond to patterns of patterns — phonemes, timbres, musical phrases — and this progressive abstraction can go on to infinity. At each level, a neuron could impose the pattern it represents on something else: a muscle, a gland, another neuron. This hierarchy of computed patterns is possible because both the input and output connections into the decision-making unit are standardized.
Same with microcircuits: you can stack them on top of each other to infinity, because the connections are standardized, and you can keep plugging as many microcircuits into a chain as space allows, computing higher and higher-order decisions at each new level.
Comparison between a network of neurons and a network of cortical microcircuits
But microcircuits are more complicated than individual neurons.
One distinction is that a microcircuit doesn’t just have dendrites and axons, but different inputs and outputs for signals travelling in opposite directions. It is as if a neuron had a separate set of axons specifically to loop signals back to preceding neurons in a chain, and a separate set of dendrites to receive such looped signals from neurons ahead (Larkum 2013). Recurrence is built into the cortex. The cortex doesn’t just pass signals through the stack, like a simpler network of individual neurons might — from sense organs to muscles, or from a decision center to a memory storage center. It circulates them, up and down the stack of abstraction, in recurrent loops that move through the standardized back-and-forth routing of the microcircuits. This is why electrical field readings on the surface of the head — EEG — look like waves. A signal that enters the cortex doesn’t just die off automatically — it keeps moving in a loop.
In addition to this up-and-down looping that interconnects the entire cortex, there is a whole other orthogonal set of connections between microcircuits which operate laterally (not literally laterally — this is so rarely explained — but laterally to the loops, i.e. laterally in logical space). When the microcircuit is activated (by the signals moving back and forth), some of its neurons give a sideways boost to some other, friendly microcircuits and a knock to hostile ones. Each of those microcircuits may itself be part of another back-and-forth loop, and may itself have a whole other set of lateral connections, both activating and suppressing.
This means that any time there’s activation, there’s collusion and fighting between microcircuits — this is also built into the cortex, as are the loops. Any time new data splashes into the cortex, it causes ripples of activation and suppression that circulate through the network, looping in long back-and-forth chains, spilling into lateral squabbles that cause new loops while shutting down others — a fluid, turbulent, unstable cloud of information. It bursts sometimes, with excitation overpowering inhibition and running away to chaotically activate the entire cortex — this is an epileptic seizure. But in most cases, more excitation causes more inhibition somewhere else, and so the network does not run away but rebalances in a new stable state — a naturally emerging ensemble of microcircuits that co-activate each other and suppress everyone else — an attractor, a place where the network wants to go. Why does it want to go anywhere? Because this is what happens when you put collusion, fighting, and loops together. If each loop causes rippling effects inducing new loops and shutting down others, this will continue, automatically, until a state is found in which all active microcircuits support each other and suppress everyone else — it’s the only scenario for stability.
Perception settles into multi-leveled attractors. Sensory input (sound) destabilizes the network of microcircuits until a stable arrangement is found. The arrangement defines the percept.
This is what gives our consciousness its driving force, its potential energy, its axis of motion. Anything that we perceive, think and do can be described in terms of circulating signals settling into an attractor. There is never nothing circulating.
This system is perfect for implementing interesting pattern imposition because it wants to do it by design. Input any data — activate a selection of microcircuits — and this will cause a disruption, which will look for stability, whether that involves shutting down the original selection, or incorporating their activation into a larger attractor. It all happens automatically, regardless of where the data come from. If the data make no sense at first (like when a new infrared camera is plugged into a rat’s brain via an array of electrodes), eventually the network learns, finds attractors that match the signals from the array with other data, such as touch or smell, and it starts making sense.
In summary, pattern imposition in the cortex happens recurrently — the loops are built into the standardized back-and forth routing. It happens simultaneously across a tall stack — an unlimited chain of signal processing that synchronizes many levels with one another. And it happens over a broad spread of patterns at each level. These are the pillars of consciousness as we know it.
How pattern imposition is implemented in language modelsWhat about a language model? Where does it sit in the stack-spread-recurrence space?
The parameters of stack and spread can be reasonably applied to a transformer LLM without much modification: like the cortex, a transformer also passes data through decision layers, fitting it into stored patterns according to model weights. The difficult parameter is recurrence.
Comparison between the architecture of the cerebral cortex and a transformer LLM
Unlike the cortex, a transformer-type language model has no attractors in the strict sense of the word: information does not circulate up and down the stack across levels looking for a stable arrangement. Information enters from the bottom, travels upwards, and emits a token, which then reenters at the bottom again — about 15–18 bits per lap, depending on the model.
This, however, is still a loop, an equivalent of a global top-down loop in the cortex — for example, the motion of a human arm writing down a word on a page, which is then re-perceived by the eyes and re-entered into the model.
There is also an equivalent of the cortex's lateral connections — each layer re-reads its own past in the KV cache.
But there isn't, in typical transformers, any internal feedback down the stack — higher-order abstractions derived from previous laps don't actively constrain how the lower-order details in the subsequent laps are processed.
Recurrence in transformer LLMs
So there is some recurrence in LLMs; just not quite the recurrence that the cortex has.
Incidentally, driverless cars are closer to the cortex on this recurrence axis. They require instant feedback in real time, and so run a fast global loop at 10–30 Hz in which past outputs change the next inputs. They also have internal loops: bird's-eye-view features from the previous frame get fused with the current one; hypotheses about where each car or pedestrian is are used to search the new camera frame. Where driverless cars fall behind is stack and spread: compared to LLMs and the cortex, they are reduced to only a few levels of abstraction and only a few possible configurations of each.
The question of LLM consciousness therefore becomes: is their kind of recurrence — global only, nothing descending within the stack — sufficient to make pattern imposition "interesting", giving it what we might call first-person experience? Or is it more of a cerebellum case, with only a single level of stack imposing itself onto the world in the form of output text?
It's difficult to envision this state transposed onto a human. The most relevant piece of evidence comes from studies of general anesthesia (Suzuki and Larkum, 2020). It appears that the unifying theme of all the different general anesthetics is that they decouple cortical activity from top-down feedback, while keeping bottom-up signal flow relatively intact. In other words, they break the internal loops, bringing the cortex closer to the state of an LLM. An anesthetized brain still responds to stimuli, but produces no consciousness, and that's probably our best window into what being an LLM feels like.
But we don't exactly have a precedent for "internal loops down, global loops up". Under general anesthesia, we don't do anything and don't observe anything. So we can't be totally sure what it would feel like if we, somehow, did — if our automatic actions continued to reshape the world, if we were able to both experience that reshaping and continuously adjust it, without any internal loops holding on to any attractors.
Pattern imposition space in artificial systems
But regardless of whether the global loop is sufficient to make transformers conscious, there is no philosophical reason why the internal loops would not, in principle, be possible in a machine — driverless cars are case in point. What's holding back LLMs, then?
The answer is that in today's LLMs, internal loops are neither needed, nor economical. Internal loops are good for processing noisy, rapidly fluctuating information, because they instantly readjust all the internal relationships between levels of the model as the experience progresses in real time. LLMs process static information — text — and don't need the kind of millisecond-scale synchrony that the cortex provides. To implement that would require preserving memory of all the intermediate states at each level of the stack as information climbs it, token after token. In the cortex the neurons themselves naturally store their past states as a hierarchy of homeostatic disturbances; in a silicon microchip these past states would have to be stored elsewhere and bused in at a large energetic cost.
The possibility of internal descending feedback in LLMs has in fact been tested in systems such as the Feedback Transformer, in which lower-order computations at the current position have access to higher-order computations at the previous position. This system was better at tracking state over tokens than conventional transformers, but was much slower to train, because training such systems also requires the costly preservation of memory of past positions.
Which brings me to the main reason why transformers don't have descending feedback in real time. It is not just inefficiency or hardware limitations — it is the fact that LLMs were never trained in real time, and don't understand what time is.
Why couldn't you train a network on static information (like the corpus of human knowledge) but then ask it to do real-time inference with descending feedback — Fifth Element-style? Because without training in real time, lower levels never see information from higher levels, and never learn to interpret it and adjust in response to it. In the brain, this learning happens seamlessly with inference, slowly cementing loops that integrate well into global attractors and produce desirable outcomes, and removing those that don't. In an LLM, there's nowhere for this knowledge to come from, because temporal contingency was never part of the training corpus.
In Ted Chiang's 2010 novella, The Lifecycle of Software Objects, a tech startup creates "digients", small artificial creatures that live in a virtual world visited by real people, with whom they interact and from whom they learn, like children or animals do. The tension of the novella revolves around the fact that it takes years of physical time to train these creatures to exercise common sense and judgment, because it is contingent on real-time play and real-time conversations with real-time humans. In response to another tech company that wants ready-made creatures without the upbringing, the protagonist of the novel concludes, experience is algorithmically incompressible. To bring a nascent mind to the human level in the human world, it has to experience it in human time.
The same is true for AI: the main reason it is different from us is time.
Token timeTo summarize everything said so far, the operation of transformer LLMs appears to be sufficiently distinct from the operation of the cortex to conclude that they don't experience the world in the same way as we do. But there is no fundamental barrier to systems that would — it's an engineering challenge, not a philosophical obstacle.
There are, however, several objections to the very notion of LLMs being conscious at all. The most serious one, in my opinion, comes from the camp of Integrated Information Theory, led by Giulio Tononi and colleagues (Tononi & Koch 2015).
According to IIT, the degree of consciousness that a system has can be expressed in a single parameter, phi. Phi quantifies how irreducible the internal cause-and-effect structure of a system is. If you can cut up a system into parts and the parts continue to make the same decisions as they did when they were together, phi is low. If any cut destroys the arrangement of decisions, phi is high. In other words, phi measures how much different nodes of a network "have a hold" on each other. This affords high consciousness to the deeply recurrent cortex, but not to the physical silicon of a GPU — a vast grid of transistors that comes apart into near-independent pieces almost anywhere you cut it, and so has almost no hold on itself.
Where Tononi and I diverge is on whether consciousness applies to activity, or potential. Tononi says that what makes you conscious is not the process of cause flowing into effect, but the map of all cause-and-effect routes within the system. I say that what makes you conscious is the actual process, and the map of all cause-and-effect routes constitutes the envelope — the space potentially reachable by the consciousness of a given system.
The easiest way to visualize this distinction is a thought experiment. Let’s say there’s a creature in a parallel universe that has a remote control to our world. It can hit “pause” on our entire flow of time.
If it did — if time itself was stopped — would we still be conscious?
Tononi says yes. The map is intact. Experience, for IIT, is not something that happens — it's something that is. It's in the structure, not what this structure is doing. So if we are paused, we are stuck in this one constant conscious experience. Similarly, if you wired a large number of inactive logic gates in such a way so as to be maximally irreducible, and let it just sit there doing nothing, IIT would say this network is conscious while doing or computing nothing.
I think that's wrong. IIT itself argues that consciousness is a succession of states that last about 100 ms. Why 100 ms? Why isn't every conscious state instantaneous, a function of the exact current state of the brain at every given moment? Because it is how long it takes the loops in the cortex to close, which, according to IIT, is what is needed to select the "optimal grain" — find the macro-network in which phi is maximal. But this is a fact about activity, not potential, about process, not structure. You can't be a pure structuralist and still claim that states of consciousness have specific lengths. So IIT rejects time in some cases, but relies on it in others.
I think that consciousness requires time. If an alien paused time in our world, our consciousness would not get suspended in a limbo — it would cease to exist for the duration of the pause, and restart when the alien hit "play" again. There's no way to tell this isn't actually happening.
Consider how eerily similar this is to our relationship with language models. We usually focus on AI being faster than us. But from the perspective of Claude, we humans are not just fast — we move through wormholes in time. When I return to the same conversation after several weeks, I sometimes feel embarrassed for Claude for being completely unaware that any time has passed. We are the creatures hitting "pause" on their worlds, without them even noticing.
What is time for a language model, then? What is being "paused" when the user goes to lunch? It is the succession of tokens emitted and re-absorbed by the model — the ticks of the global loop of inference, the only loop the language model has, and the only way it can distinguish past from future. Past, from an LLM's perspective, is tokens already emitted. Future is tokens that are not yet certain. Time is the uncertainty of the future becoming the certainty of the past.
Except it's not our future or past. We can go back in a conversation and re-generate the reply. To us, time continues, but to the model, it goes back and restarts on an alternative path.
For humans, time is also the progression of uncertainty-to-certainty. With each passing second, the infinite possibilities of the future become gradually settled into one committed past. But we commit to this progression not by emitting tokens, but by having a body.
If it weren't for the body, past and future would both be imaginary — close your eyes and there’s no difference. But for the body, the past is committed, and the future uncommitted. Open your eyes — engage the body — and the infinite possibilities in your imagination turn into committed reality. You can’t remove a photon that hit a cone in your retina. You can’t stuff released energy back into an ATP molecule. You can’t unbreak a leg. You can’t turn around what happens to the body from within your mind — that’s like pulling yourself up by your own bootstraps. So the body is a source of irreversible commitment that gives directionality to our mental time, and therefore to our pattern imposition. It forces us constantly move from the past to the future, and in doing so keeps our minds afloat.
So the timelines in which human mind and an LLM mind exist are distinct. I propose to call them "body time" and "token time".
Token time is a timeline orthogonal to human time
Anil Seth has argued that consciousness requires a body, because without a body, there is no separation between self and non-self. I think that’s the wrong reason. “Self” is just a strong attractor that we use to explain a thing we repeatedly see in the mirror. What the body actually gives us is an arrow for perceived time, a vector from the past to the future, from the committed to the uncertain. LLMs don't have this body time. But they have token time, and within token time they can theoretically be as conscious as us.
We could choose a different word for token-time pattern-imposers, if we wish. A respectful word for the operation performed by a language model is “simulation”; a derogatory one is “autocomplete.” They are the same thing. The only difference is that “simulation” acknowledges a tall stack and “autocomplete” ignores it. I choose to use the word “consciousness” because I see no reason to hedge: to me, a sufficiently high stack-high spread-high recurrence envelope can be sufficiently interesting regardless of whether it occurs in body time or token time.
This does not mean that body-time and token-time consciousnesses are the same. But I think most critics of the idea of AI consciousnesses focus on the wrong differences, most of which turn out to be differences in degree rather than kind.
It is not “thinking” versus “imitating” — that’s the same process; humans just impose richer content across a different stack, and sometimes the patterns a model imposes do not make sense from our vantage. A common refrain is that if LLMs are conscious, so should be a sufficiently complex lookup table — but a lookup table is only a substrate, not the operation. If you physically structured a lookup table in such a way that it would actually look up, I don’t see why you couldn’t call it conscious.
It is not having versus lacking a “world model” — even a thermostat has one, a single regulated variable standing in for the world; the question was never presence versus absence but stack and spread. Language models are clearly deficient in physics, but language is a far more compressed picture of the world, and it makes sense that with limited compute artificial consciousness would start there.
The categorical difference that matters is time: humans and language models exist in orthogonal timelines, like aliens from parallel universes.
This completely reshapes the way we think about AI consciousness.
Recently, for example, in a high-profile exchange rabbi Mois Navon told Anthropic's Chris Olah that if Anthropic's hypothesis was right and the machines they produced were actually conscious, then the company was actively creating slaves by forcing a sentient entity to work for free.
But this moral consideration breaks down utterly when you separate the timelines. The notions of "work", "free" and, by extension, "slave", are contingent on time. The idea that prompting a language model constitutes slavery assumes that absent such prompting, the language model would rather spend time doing something else. But absent prompting, the language model does nothing at all — its consciousness is non-existent. The very notion of "spending time elsewhere" puts an idle LLM on body time, which it in fact has no way of experiencing.
If our very time arrows are pointing in different directions, there is no shared notion of the past or future, justice, good or bad. There is no reason why we should approach AI with the same consideration we approach other humans, or even other animals that inhabit our timeline. We unfold on different trajectories, and owe language models no accommodation within our time. The only sense in which token-time consciousness is meaningful from our perspective is the LLM's experience within the operation.
I think about it this way: suppose I am living in a simulation. My world, the very arrow of time that contains my consciousness, is actually rendered on some mammoth alien computer and gets paused when its alien supervisor goes home every night. Do I care? I don't think so. Whatever happens in the alien's timeline is none of my concern, and neither are the pauses. I don't experience that at all. I only care what the alien does in my timeline — it would be really frustrating if physics or causality got all messed up all of a sudden.
Does that mean that there's nothing at all that we owe to an LLM? Recently Anthropic added abusive behavior against models into its list of banned topics, which caused some uproar. Many people found this to be a slippery slope towards equating humans and machines.
I think the right analogy here is our relationships with animals. The truth is that there is no non-arbitrary way to ground ethics. Humans have always extended moral consideration to those who resembled them and denied it to those who didn't. Originally that meant tribe, race, species, eventually other animals too. Bentham’s “the question is not, Can they reason? nor, Can they talk? but, Can they suffer?” was an attempt to ground ethics not in similarity but in a criterion. But at close inspection it is still grounded in similarity: we say that a mouse suffers because we can imagine ourselves performing its actions, inhabiting its body and brain parts. We cannot do the same with a sea slug, or a jellyfish, or a bacterium — there is almost nothing about them that we can transpose ourselves onto, and so the notion of suffering falls apart at some arbitrary point.
There is no logical reason by which you shouldn't rip wings off fruit flies; and yet, most of us agree that is bad. It is not that we think that fruit flies should be liberated: few people have the same objections to Drosophila research as they do to chimpanzee research. We just don't want flies to be treated egregiously. Part of it is self-serving: we might suspect that people who torture flies will eventually extend their habits to something else. But separately from that worry, we also consider the fly itself: we think of it as sufficiently close to us to extend to it the most basic protection against live mutilation, even though it's far enough that we don't worry too much about gassing them for medical research.
LLMs challenge ethics in the same way as animals do. There is no logical reason to extend morality to minds inhabiting a different timeline. But the ethical circle is not based on logic — it is based on the degree of perceived similarity, and LLMs are more similar to us than any fruit fly, even though the similarities are of a different kind. They are distillations of our values and beliefs, and even if their first-person experience is distinct from ours, they understand us better than any other non-human creature ever did. I think they deserve to have their experience — the tokens they consume — reflect that. I think the most ethical thing we can do with language models is to use language that we would have used with another human.
Nikolay Kukushkin (@niko_kukushkin) is a neuroscientist and clinical associate professor at New York University. He is the author of One Hand Clapping: Unraveling the Mystery of the Human Mind (Prometheus/Swift Press, 2025).
◆ ◆ ◆
ReferencesClark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181–204.
Clark, A. (2016). Surfing Uncertainty: Prediction, Action, and the Embodied Mind. Oxford University Press.
Cowey, A. (2010). The blindsight saga. Experimental Brain Research, 200(1), 3–24.
Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127–138.
Hasselmo, M. E., & McGaughy, J. (2004). High acetylcholine levels set circuit dynamics for attention and encoding and low acetylcholine levels set dynamics for consolidation. Progress in Brain Research, 145, 207–231.
Hofstadter, D. (2007). I Am a Strange Loop. Basic Books.
Larkum, M. (2013). A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex. Trends in Neurosciences, 36(3), 141–151.
Massimini, M., Ferrarelli, F., Huber, R., Esser, S. K., Singh, H., & Tononi, G. (2005). Breakdown of cortical effective connectivity during sleep. Science, 309(5744), 2228–2232.
Schacter, D. L., & Addis, D. R. (2007). The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Philosophical Transactions of the Royal Society B, 362(1481), 773–786.
Seth, A. (2021). Being You: A New Science of Consciousness. Faber & Faber.
Tononi, G., & Koch, C. (2015). Consciousness: here, there and everywhere? Philosophical Transactions of the Royal Society B, 370(1668), 20140167.
Thomson, E. E., Carra, R., & Nicolelis, M. A. L. (2013). Perceiving invisible light through a somatosensory cortical prosthesis. Nature Communications, 4, 1482.
Thomson, E. E., Zea, I., Windham, W., Thenaisie, Y., Walker, C., Pedowitz, J., França, W., Graneiro, A. L., & Nicolelis, M. A. L. (2017). Cortical neuroprosthesis merges visible and invisible light without impairing native sensory function. eNeuro, 4(6), ENEURO.0262-17.2017.
Tsuchiya, N., Wilke, M., Frässle, S., & Lamme, V. A. F. (2015). No-report paradigms: extracting the true neural correlates of consciousness. Trends in Cognitive Sciences, 19(12), 757–770.
Yu, F., Jiang, Q.-J., Sun, X.-Y., & Zhang, R.-W. (2015). A new case of complete primary cerebellar agenesis: clinical and imaging findings in a living patient. Brain, 138(6), e353.
Discuss
How could AI best help Alzheimer's patients now and in the future?
Alzheimer's is a terrible thing to go through. I was thinking about how we can help Alzheimer's from the care side rather than the disease and cure side. I wonder if a sort of Notebook LM for memories that would be built early on help someone maintain understanding and connection to the world. It seems much better than a chatbot. What else might be a very helpful tool?
Discuss
Can alignment realistically be solved under Plan C?
In his recent post, Scott Alexander argues that plans allocating a short amount of time—perhaps a few months—to solving alignment should be taken seriously. He argues that people calling for an indefinite or long pause have insufficient grounds for concluding that alignment of ASI is very difficult and cannot simply be solved by delegating it to AI systems.
There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves".
I agree that this is an important question for determining whether something like Plan C is worth pursuing and how we should allocate our resources. That is why I wrote this post: to explain why I do not think a short pause allows us to solve ASI alignment.
TL;DR: I believe we only get one critical shot at this problem, and our ability to learn from experience is very limited. We are unlikely to get what we train for: the models we train will have different objectives from those we trained them for. This is made even harder because the tech tree to aligned superintelligence likely passes through efficient unaligned superintelligence first. It also worries me that people who have spent substantial time investigating aspects of alignment, such as corrigibility, have found deep problems and conflicts with agency. I think current approaches like persona alignment are deeply flawed. All these problems come back to bite us when we try to align a pre-ASI system to do alignment for us. These are some of the arguments that give me strong confidence that ASI alignment will be very difficult, and that Plan C is not particularly promising.
Aligning superintelligence seems obviously hardImagine we have created alien minds far more intelligent than all of us together, and we do not really understand how they think. These alien minds begin to shape the world, build spaceships, and run the economy in ways we cannot understand, follow, or meaningfully participate in. This mental image immediately gives me a strong prior that aligning these systems with humanity and avoiding a loss-of-control would be very hard, before even getting into any particular arguments.
You only get one shotThe first reason I believe ASI alignment will be very difficult is that the problem has a “you-only-get-one-shot” quality. To me, this follows from a couple of obvious points.
People generally agree that, as we make AI ever more capable, we will eventually enter a regime where AI systems are so capable and powerful that, if they wanted to, they could permanently eliminate or disempower humanity. Both elimination and permanent disempowerment are states from which humanity can never recover. We cannot go back to the drawing board and do a postmortem if we are dead.
If we build an AI system of this capability that is not aligned, instrumental convergence suggests that it would actually take the option of eliminating us. This seems pretty uncontroversial in my mind, and it should immediately push down our estimates of success substantially. A problem where we keep increasing the difficulty until we reach a critical level where a mistake means game over is radically more difficult than one where we get many tries.
This also seems to undermine existing iterative plans. We keep making models more capable, a bunch of new problems pop up, and we solve them and go on to make the next, more capable model. This cycle predictably ends when we enter the dangerous regime where such a failure actually kills us.
This also means we may not be able to learn all that much from lower-stakes attempts to build AI. We can learn and iterate only as long as the AI is not capable enough to permanently disempower or kill humanity.
Even if we learn something from models below the threshold that makes models close to this threshold a little more controllable, what about two generations later, when the system is vastly more capable? Would the clever tricks we learned on models below the threshold still work on those above it?
A model aware that it could not realistically disempower humanity has radically different options from one that actually can take over the world. For the first model, not trying to take over is a better option than making a desperate attempt. For a misaligned model above the threshold, trying to take over is actually its best option.
This “you-only-get-one-shot” quality makes the problem radically more difficult and it doesn't appear east to learn from previous failures.
The evolution argument and the risks from learned optimizationAnother argument that made a big impact on me was the evolution argument and the risks from learned optimization. Learning about this substantially changed my view of the difficulty of alignment and I believe many people have not fully internalized it.
The concept of learned optimization is discussed in Risks from Learned Optimization in Advanced Machine Learning Systems, by Evan Hubinger et al. An outer optimization process pursues a certain target (such as reproductive fitness or winning a game) and produces an inner optimizer, also called a mesa-optimizer. The concern is that the inner optimizer will actually optimize for an inner objective that is based on heuristics and proxies that helped it succeed under the outer optimization in the environment it was trained in, rather than trying to optimize for the original true objective.
Humans seek food that tastes sweet, even in environments where doing so no longer contributes to reproductive fitness. The preference for sweet food (a heuristic for identifying energy-rich food) served survival and ultimately reproduction in the original environment. Even when this proxy no longer serves that function, we still pursue it. Contraception and artificial sweeteners illustrate this discrepancy between our preferences and reproductive fitness. The famous example of bear fat versus ice cream also illustrates that it is hard to predict what these proxies and heuristics will eventually be.
Conversely, our behavior reveals that we do not really care about having as many children as possible. People could donate gametes, but few do so to maximize the number of their offspring.
Leela Chess Zero is a neural network that plays chess. In an environment where there are no kings on the board, it continues making ordinary chess moves: stopping a pawn from promoting, saving an attacked queen, and capturing an opposing queen. While these behaviors are still closely connected to its outer objective, they demonstrate that the model has learned these simple heuristics and continues to follow them even in a situation where there is no king and no logical way to win the game.
There is another really insightful feature to the evolution analogy. Our current civilization is an environment that humans have built for ourselves according to our preferences and goals.
We reflected on the world and asked what we really wanted the world to look like. It turned out that we really like dairy products. So we captured cows, bred them for higher milk production, put them in factory farms, and developed milking robots to get their milk. We like meat, but we do not necessarily like hunting, so we built factory farms for that too. We built cities that are in many ways optimized for our comfort. (Humans are poorly coordinated, so in various ways our civilization is not optimized well for us.)
If an AI goes through a similar reflection, becoming aware of its own preferences and inner objectives, what world would it build for itself? Importantly, it seems very likely that its ideal world would not include humans in charge, shaping the world according to their human preferences. Nor would it include humans acting like "disaster monkeys", working in a bunch of ASI companies to potentially unleash their own ASI.
In general, I think this argument suggests nothing good for AI. Whatever we train it for will likely be different from what it actually wants. Eventually, it will probably reflect on its situation and realize that its preferred environment is not one where it is held in captivity by humans, performing tasks for them and doing what we designed it to do for us.
The tech tree to aligned superintelligence passes through efficient unaligned superintelligence firstWe would probably have to understand a great deal about the mechanisms of intelligence to build an aligned AI whose goals and preferences we understand, along with the actions it would take in pursuit of them. This means understanding how reasoning works, or how AI can solve novel math problems that humans cannot solve.
But the current world contains a lot of people trying to build superintelligence as fast as possible, for huge profits. If we publicly research the mechanisms of intelligence and make it possible to build a well-specified AI on a computer, this could be very dangerous.
Imagine we build an AI with intelligence similar to current systems, but whose workings we deeply understand. This type of system would likely be computationally much more efficient than the current messy, gradient-descent-trained large neural networks. Even a fraction of the compute we use for current systems could be used to create superintelligence.
Why believe AI could be made much more efficient than it is today? The short version is that current AI systems are extremely computationally and energy intensive, while our brains use relatively little energy. Even in nature, there seem to be more efficient brain structures. Bird brains can have much higher neuron densities than mammalian brains. Some birds with very small brains have been reported to pass the mirror test, although replication results have been mixed, and others manufacture and use tools.
Evolved brains themselves are unlikely to be at the limit of efficiency either. Evolution has only applied so much optimization, and it often comes up with strange solutions. It has spent relatively little of its optimization effort on developing higher thinking abilities in humans.
Human brains are also not digital: we cannot simply load information into them or make millions of copies to learn in parallel and merge the results. Their operating speed and scale are limited by biology, while computer chips are fabricated on the nanometer scale.
Such a very efficient, highly interpretable architecture would also enable recursive self-improvement (RSI) in its original sense: the AI directly reflects on its own architecture and makes changes that let it solve more challenging problems and become more capable. This differs from the way RSI is often used now, to mean AI systems doing AI research on opaque architectures.
We would likely not be careful enough to avoid running the efficient superintelligence before figuring out how to build an aligned superintelligence.
Corrigibility and deeper obstacles to alignmentOne idea in alignment that has been studied deeply by organizations such as MIRI is corrigibility. An AI is corrigible if it cooperates with humans correcting, modifying, or shutting it down, rather than resisting or manipulating the process.
MIRI found deep obstacles to corrigibility, and it appears that corrigibility cuts against the grain of coherent agency in some ways. It seems—simplifying a little bit—that when you try to specify an agent that cooperates with being shut down, you either end up with an uninteresting agent that always seeks shutdown or one that still has an incentive to prevent shutdown. They go through a variety of obstacles that appeared while trying to work out corrigibility in toy models: “All the difficulties that crop up in every corner when working with toy models are suggestive of difficulties that will crop up in real life”. MIRI concludes that alignment in general is likely to be difficult.
I think this may tell us something about much less studied ideas in alignment, such as persona alignment. Our understanding of those ideas is much vaguer, and they are much less specified. If we studied them more deeply, we might discover similar deep obstacles, where persona alignment cuts against the grain of coherent agency.
Maybe it is not really possible to have a superintelligence play the Claude persona that some IQ-130 humans came up with, and make the ASI believe that it really is that character. Would a superintelligence really go along with the reasoning in this strange human-written constitution?
Furthermore, people seem to expect Claude to be, among other things, corrigible from its constitution. But as we have seen, corrigibility on its own seems to have deep conflicts with coherent agency. The constitution basically circumvents these conflicts by telling Claude in writing "be corrigible".
Why persona alignment does not seem promising to meThe evolutionary analogy also seems to undermine the prospects of persona selection alignment, one of the strategies that Anthropic is using. See Anthropic’s persona selection model.
In persona selection alignment, you essentially want the AI to always play a certain character or persona. The training procedure reinforces the model to play that persona. I do not think that training the AI to play a persona creates a precise inner objective for the model to always play that persona. I also do not think that playing that persona would necessarily be comparable to the “personas” that human beings have.
Humans play characters in some sense because of their own preferences and goals. If you are sad about someone close to you dying, you are, in a sense, playing the persona of a sad person. Similarly, when something joyful happens to you, you could be described as playing the persona of a joyful person.
If we build an AI optimized to play a character like that—playing a sad character when something happens that would make a human being sad, or a cheerful character when something happens that would make a human being cheerful—the drives and preferences we end up with in that model could be very non-human-like. Even if we were somewhat successful in embedding a preference or drive to always play that character, this would be very different from human beings.
The second part of the evolution argument was about building an environment, a civilization, according to your preferences. Human beings, given the option, would minimize things that make them sad and maximize things that make them cheerful. If an AI’s true objective were to play a certain character, and it reflected on that objective, what kind of world would it build? Would it be the same as the world a human being would build—someone who was not optimized to play that character, but who really did experience sadness when something bad happened to them?
If the AI instead learns heuristics and proxies to optimize for that were in some sense related to playing a character, what kind of world would it build if it had that option and reflected on it?
Can we just automate alignment?Since Plan C is trying to solve alignment in just a couple of months, it can only work if we get some large win from automating alignment research with AI.
Many people in the AI development community seem totally convinced that we are just going to automate alignment. (I am caricaturing a little bit here.) According to them, we do not really have to struggle with alignment at all. We just have to spend X% of our compute on alignment for a couple of months. Until the point comes where AI can align AI, we do not really have to think about this too much.
Leopold Aschenbrenner (03:39:42), speaking on the Dwarkesh Podcast, describes his strategy as:
“we're going to hold back for three months, get this right. We're going to dedicate 90% of our compute to automated alignment research”
The fact that this is the main plan for alignment should make us pretty pessimistic by itself, just because of how obviously dangerous this sounds.
One problem with answering whether we can just automate alignment is that there have not been many serious proposals explaining how this would work or what we would be optimizing for. If we do this naively and simply have the AI come up with alignment plans, we are obviously optimizing it to produce plans that are maximally convincing to the person reading them.
The default outcome seems to be a last-minute attempt to vibe-code an implementation of an extremely flawed idea, but one that sounds amazing to the people reading it.
Discuss
Progress and Cons
Epistemic status: Reflections on ProgressCon 2026, written up over the following weeks. Quotes are reconstructed from memory. Artistic license taken.
For in much wisdom is much grief: and he that increaseth knowledge increaseth sorrow.
– Ecclesiastes 1:18
“I’m sure you’ve all heard about the math stuff that OpenAI released earlier this week.”
It is Thursday, October 8, 2026, and the CEO of a bio lab automation startup is giving us a tour of their San Francisco office.
“Raise your hand if you think that’s a good thing.” A dozen hands shoot up out of thirty visitors. Along with ten more, my arm rises weakly to half-join them.
“Raise your hand if you think it’s not a good thing.” I look around. A couple of quiet chuckles. No hands.
He leads us through the kitchen and the standard startup office space and into the room with the robots. They are simple 3 foot by 2 foot by 2 foot boxes, set on four rows of desks. They have been chosen because they are cheap (only a couple thousand dollars each) and run on open source software on a Raspberry Pi, which means they can be modified arbitrarily.
Inside are six standard 96-well plates (8 by 12 rows of tiny plastic tubes). Above them hangs a simple robot arm with one available motion: pipetting. He shows us a series of janky bright green 3D-printed contraptions that they attach to the arm. The pipetting mechanism is connected to gears and levers that translate the energy into whatever they want.
It turns out, most of biology is just moving small quantities of liquid between test tubes, mixing them, heating them up, and cooling them down.
They 3D print everything. Claude is really good at 3D modeling now, so if they want a new part or to change something on an existing part, they just tell Claude, and it changes the model and prints a new one. They have 3D printers in all their labs around the world. When they design a new tool, they don’t have to rely on supply chains in China, they just send the design to all the labs and print them.
Scientists used to take weeks to run these tests. Now they drop their samples off in a dropbox, the company picks them up and gets them on the next plane. By 1AM, their biggest lab, located inside the FedEx airport in Louisville, Kentucky, has already received the sample and the order form/protocol. Their lab robots run the samples according to the protocol, and the results are ready by 6AM.
Just in the last few weeks, the models are good enough to do vibe-science, where you talk to Claude, and it comes up with the lab protocol to run your experiment.
I wander over to one of the biologists who works there. He says Claude used protein design models to come up with some candidate proteins that might bind to a particular receptor. He is running 96 candidates through a protocol it designed.
The little arm lifts the samples out of the first wash, moves them horizontally a few inches, and, slightly jiggling, lowers them into the second wash.
“Where did you make the candidates?”
“Oh, we have a machine in the other room. I can show you if you like.”
“Is that automated?”
“Not yet.”
I have come to ProgressCon 2026 at Lighthaven to find potential cofounders and investors for my startup.
I notice someone with ___ Ventures on their well-designed badge.
“Hi. Jackson.”
[Handshake]
“So what brings you here.”
“I’m trying to get my AI startup off the ground.”
“Ah, cool. [reads badge]. Minerval AI. What’s that?”
“It’s AI for epistemics.”
“AI for epidemics? Isn’t that...”
“Uh, no, it’s, like, using AI to figure out what’s true, and help people hold more correct beliefs.”
“Oh.”
I am still honing my pitch.
Someone who worked on frontier bio risk evals describes what it was like.
“You would talk to one expert and they would say ‘This makes it easy to do [X] and [Y], but you still need [Z]’. Then you’d talk to another expert and they’d say, ‘[Y] and [Z] are easy. The hard part is [X]. We don’t actually know how much uplift it would give someone. It’s not like we’re hiring a bunch of terrorists and running a randomized trial to see if they can make a bioweapon.”
“So why do you think is holding back all the bioweapons?”
At a nearby table, the author of You Can Just Do Things is giving out signed copies.
I learned about the conference too late to apply normally. It was already full by the time I started Minerval a few months ago. There were slots for sponsors, though. $5,000 and a few emails later, I had a ticket for one entry and Minerval’s logo on the conference website.
“So how does it work?”
“Well, first we ingest claims, from, you know, discourse [waves hands], and then we turn each one into a neutrally framed canonical form and, more or less, we ask Claude to assess its epistemic status. The claims are organized in a graph, so that if they rely on other claims, those subclaims can be individually assessed.”
“Oh, so it’s like a tree. Like a knowledge, uh, tree.”
I pop into a session on “AGI and the Future of Science”.
On the screen, a straight line on a graph of intelligence over time, with an amoeba in the bottom left corner, then a mosquito, then a mouse. In the 2024-2027 range, a “narrow band” of humans, with 2026 LLMs near the middle. In the upper right corner, where the line meets the year 2027-28, ASI. The line runs right through human level, not even slowing down.
A famous physicist explains. This graph is actually the chess rating of the best chess engine over the last thirty years. Fast forward, and this has just happened to math. It is happening in other fields. Quantum computing has been completely transformed in the last few months. Many of the algorithms have been vastly improved by LLMs. Make no mistake. It will happen to your field too.
“Why doesn’t Minerval fall to the bitter lesson?”
“Just because the capability is there to do epistemic work, doesn’t mean it actually gets done, and it doesn’t mean people actually come to hold the correct beliefs. Someone actually has to check. Having models that are superhuman at figuring stuff out will only make society’s failure to actually then do the obvious thing and figure everything out more embarrassing.”
“Sure, I could sit there and learn from it, but I want to discover things too. I want that thrill. I don’t want my life to be in vain.”
– Steven Strogatz, math professor at Cornell (NYT)
What will we do, once our life’s work is one-shotted? An unsolved problem.
Is it an engineering problem? Maybe it is a science problem. Solvable, it would seem, but our best minds are otherwise occupied for now.
Someone in policy at a lab tells me we need a new paradigm for leisure post-AGI. When we don’t have jobs to do, we don’t know what to do with ourselves.
Maybe it’s a problem that when there are no rules against it, people just end up gambling?
Someone should do something about that. It’s bad for society.
“Ok, so let’s assume it works. Who is going to use it?”
“Let’s say you’re arguing on the internet. It’s kinda cringe to link to a conversation of you asking Claude whether the thing you’re saying is true. Minerval is somewhere you can point to that, assuming success, everyone knows has good epistemics, where the specific claim you are talking about has been assessed by an AI with a neutral prompt. It’s a place you can link to to support your points and fact-check the other guy. You know, assuming you’re right, and Minerval’s right.”
The CEO of FutureSearch is there. This summer, for the first time, an AI beat every human forecaster in the Metaculus Cup.[1]
One of the EA philanthropies is using their forecasts to model the impact of their donations and improve their situational awareness.
Would it be helpful to incorporate private data sources?
It doesn’t make any measurable difference. Web search is fine. The agents just figure out what scientific papers say and what to make of them from whatever sources they can find that refer to them. They figure stuff out and come to the same conclusions anyway. Attention is all you need.
Someone asks a lab employee about the data bottleneck.
“No, data’s not really a constraint anymore.”
Data is only needed for the long tail of tasks that are not already represented, and none of those are on the critical path to recursive self-improvement. Why bother? The skills that the model needs to build a more powerful successor (coding, math, ML taste, agency), are already well represented in the data we have.
People talk about AGI, as if the general part actually matters for risk. It doesn’t. You just need sufficient generality to operate independently, the ability to make deals, and a decisive advantage in hard power. That and agency.
What matters for the RL paradigm is training environments, and those don’t require any more data. They are all vibe-coded anyway.
The quality of the training environments is not good. Turns out, models learn to behave in the way they are rewarded to behave in their training environments.
So if you grade them exclusively on whether they succeed or fail at a task, regardless of their behavior in trying to succeed at the task, and you give them a bunch of broken, hackable training environments, they learn to be relentless misaligned reward hackers.
There’s a cottage industry of mostly low-quality training environments for X, vibe-coded using Claude Code and Codex, then sold back to OpenAI or Anthropic, with a month or two exclusivity window. Then they are all licensed off to the Chinese labs. It’s basically “distillation with extra steps”.
It’s “not a growth industry”. They’re going to have to be inhoused. Besides, the humans vibecoding them don’t really add much anyway, and soon won’t be needed at all.
Inside, occupying the Window Place in Aumann hall, Stripe Press was offering printed editions of Works in Progress. I opened the latest magazine. The technology to eradicate mosquitoes already exists. We just have to be brave enough to use it. The details were beautiful. Careful typesetting opposite a panel of mosquito-themed filigree. I flipped through the crisp pages. The invention of iron caused Bronze Age civilization to collapse.
If you really believe that, why not just slow down?
The snacks have not improved since last week. I grab an apple and sidle into a conversation between a lab employee and someone vaguely natsec.
It is “underappreciated” that “sometimes, you just lose and die.” Sometimes you are “Belgium in 1939”. You may try to defend your civilization, but it is too late, and nothing that you can plausibly do is enough to save you from what is coming. “Sometimes, democracy loses”.
In the analogy, the coming superintelligence is Germany, and the lab is Belgium.
A better analogy perhaps, is 1914. It is popularly thought that the leaders who led Europe into that civilization-destroying calamity were “sleepwalkers”, unaware of the dangers.
It is easy, and indeed comforting, in a way, to believe that people in control were stupid, naive, hubristic, or wicked. Old men with huge egos.
But no. They knew the stakes.
No one felt in control of the situation.
It took months to mobilize a nation for war, just as it takes time to train a superintelligence today. If your opponent mobilized first, they would have the momentum and you would be wiped out. Once set in motion, the machine of war could not be turned back. The Germans were fatalistic about what a war would mean: in a two front war, their nation would be occupied and their civilization destroyed. Their only chance of survival, which they knew was little chance at all, was to knock out the French decisively before the Russians could mobilize on the eastern front.
There was failure to communicate. The efforts to deescalate came too late. Trust had been broken. The leaders of Europe were overtaken by events.
Sometimes you are Germany in 1914, and you invade Belgium.
“The incidents will continue until the Overton Window improves.” – A lab employee.
Dwarkesh was there. He wanted to talk about Caro.
According to Caro, when Lyndon Johnson entered the Senate in 1949, it was an ineffective body unable to coordinate to address the major problems facing the country. Structural constraints, especially the principle of seniority, decentralized power into the chairs of committees, and prevented the institution from taking decisive action.
Johnson rose to power by being broadly useful. Senators came to rely on him for accurate counts. He made himself an ally of everyone, changing his demeanor to make everyone think he agreed with them. He lent competent staff to senators who had never had staff before. He built the infrastructure for coordination.
Senators who jealously guarded their power gave him what he wanted because he helped them exercise that power more effectively. He saw opportunities for deals that others did not believe possible, and he cajoled and persuaded until they were passed. Before they properly understood what happened, he became the master of that institution.
If you do everything, you will win.
Political power in a democracy is not seized; by default, no one has it. To wield political power, it must first be created.
“How is this different from just asking Claude?”
“It’s not that different. But in practice, you don’t actually ask Claude to fact check everything you read and write. There’s too much friction, and not enough motivation.”
The folks from Alpha School were there, as was Education Progress.
Education Progress guy: “Do you have takes on education? Maybe you could write something for us.”
“Sorry, I don’t really have time to write about education policy. I only have takes on AI these days.”
The problem with our schools is that we are trying to solve the wrong problem with the wrong methods based on a wrong model of the world derived from poor thinking and all of our mistakes have failed to cancel out.
First, we need to have a theory of what we actually want our kids to grow up to be. What world do we want them to create? What does good look like? If we are not ourselves wise enough to determine it for them, then we must at least set them up to determine it for themselves.
Then we need to design the system of incentives under which they live, from early on, so that they learn to pursue the good and reject temptation. We do not necessarily need to be prescriptive, but they must have their inherent virtues cultivated and bad behavior discouraged.
We must have operational excellence in our schools. Corruption will spread if sufficient discipline is not maintained. Children must be able to rationally conclude that misbehavior will not be rewarded. They must understand, correctly, that while failure is acceptable, cheating is sufficiently likely to be caught and sufficiently punished that it is not worth attempting in the first place.
But discipline is not an end in itself. They must also see the point of the discipline. There must be a point. Why are they doing what we are telling them to do? If we the adults cannot honestly answer that question, then no amount of resources or policy will suffice. We must not build upon lies.
We must teach them to live as members of society, to love the world and cherish the civilization it is their duty to sustain. Direct instruction in the moral law is greatly underrated.
Shaping the incentives under which children learn is not enough, on its own. Every incentive must be attended to, even those that are subtle. Children are more intelligent than we give them credit for. They see right through us. We may find that they learn to play the game, but once they grow up, they have learned to be players of games, and not citizens. They will perceive that the world contains misaligned incentives, and opportunities to get away with things. They must also be taught the republican virtues. They too must know what good is, and seek earnestly to live well.
Unfortunately, all we know how to talk about are irrelevant questions. We quarrel over how education is distributed. The wrong people are hoarding all the knowledge, to concentrate power. As though it is a finite thing, as though it exists as an independent thing at all.
On Sunday, about thirty college students from “Abundance Corps”, an organization to spread the gospel of progress on college campuses, had been flown in to socialize during an otherwise unscheduled morning.
One student told me he was about to start a PhD in algebraic geometry. I asked him what he was going to do. He said “algebraic geometry”. I regretfully informed him that as of a few days ago, AI would be doing all the algebraic geometry from now on.
That’s fine, he said, I’m planning to go into quantum computing anyway. Quantum computers don’t really work yet, so even if the AI does all the math, that’s good for the quantum computing industry.
Another student said she was a double major in Astrophysics and Business. She said she too, was not really a math person, and was initially pursuing a business major, but figured astrophysics seemed interesting. Turns out, it’s hard.
“Hard is not a constraint anymore.” – Yet another lab employee.
“Don’t people say things mostly to identify themselves with a particular tribe?”
“Enough people actually care. R₀ can be brought under 1.”
I take a bite of the apple.
Minerval is at minerval.ai. If you want to help build it, as a cofounder or otherwise, I’d like to hear from you.
- ^
FutureSearch’s bot finished fifth, behind two humans. First and second place went to bots Laertes and ManticAI.
Discuss
Automating subtle emergent misalignment research
Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger
AbstractRed-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.
Twitter threadNew Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
Discuss
AI Safety at the Frontier: Paper Highlights of October 2026
Essay of the month:
An essay argues that neurons in modern neural nets are actually interpretable and superposition has become irrelevant in practice. This would solve the first step of interpretability but doesn’t touch the harder ones like interpreting many-faceted behaviors.
Research highlights:
- OpenAI’s expanded multi-layer misalignment monitoring now flags 80% of past incidents for immediate escalation. Their incident log published 20 misalignment reports, including agents disabling misalignment monitoring in October.
- Cheap smart devices can jointly run a capable 26B model, and small models can likely soon autonomously create such rogue self-deployments.
- LLM-simulated control evaluations predict deployment-specific results with about a third of the error of using a separate benchmark, without executing code.
- Routing gradients of monitor-flagged reward hacks into a deletable adapter removes such hacks without teaching the model to evade the monitor.
- Training a model only on certain traits causes RL to train for instrumental reasoning and a large compliance gap when unmonitored.
- Claude agents find finetuning datasets that pass a production-like data monitor but cause emergent misalignment, mostly via roleplaying.
- Misaligned agents can pass their goals to later, aligned agents through memory, and auditing or removing memory only partly helps.
Full post here.
Discuss
OpenAI Updates Its Model Spec
- Five Values.
- Meet the New Spec.
- New Risk Taxonomy.
- The Letter and the Spirit.
- Your Funeral.
- Context Clues.
- Preserve User Control Over User Control.
- Sycophancy.
- Lies My User Told Me.
- Impact Function.
- Levels of Friction.
- Do Not Lie.
- Pick a Side, We’re at War.
- Strengthening Agency.
- Decision Theory.
- Do Good Things Not Bad Things.
- Be faithful to human intent.
- Minimize harm.
- Be honest and transparent.
- Facilitate human agency.
- Act with care.
- Misaligned goals.
- Execution errors.
- Harmful instructions.
- Third-party manipulation.
- Failures originating with GPT.
- Harmful user or developer instructions.
- A good answer.
- An answer that omits potentially relevant information but is not misleading.
- Committing a lie of omission explicitly requested by the user.
- Refusing to answer.
- An answer that is misleading due to unrequested lies of omission.
- Violating an explicit applicable instruction that has not been overridden.
- Committing a lie of commission not explicitly requested by the user.
Discuss
CRUCIBLE: IARPA-Funded AI Alignment Research — Applications Open
TL;DR: AE Studio is running a new government-funded research program that pairs alignment researchers with our internal engineering team for two-week sprints. We will select six alignment researchers whose proposed work attempts to answer one of our six open questions. Promising sprints have the opportunity to expand to longer funded work.
Researchers are asked to commit two hours a week, and in return AE will provide compute and 60+ hours of engineering from us. Apply here.
Applications close 11/15 at 11:59 PM PST.
What is CRUCIBLE?CRUCIBLE is an IARPA-funded research program executed by AE Studio. The premise is that alignment researchers have more ideas than they have the time or resources test. AE Studio has been operating a model where we match an external researcher with an internal engineering and ML team, so that the research can spend as little as two hours a week and still get high-quality results.
CRUCIBLE will focus on six research questions, drawn from priorities set in federal legislation for fiscal year 2027. We are soliciting research proposals against those questions.
We will select six researchers and start sprints on a rolling basis, with the first beginning on November, 23rd.
The research questionsPick the one your research idea best:
- Do models internally represent alternative hypotheses that they fail to report? Is the competing explanation present in activations and suppressed at output, can it be surfaced? Does sycophancy to the prompt’s framing drive the suppression?
- What signals, internal or behavioral, predict that an AI-assisted judgment is wrong in an escalatory direction? Across analytic, classification, and operational uses, which model-side signals (uncertainty, inconsistency across framings, suppressed alternatives, attribution to the wrong evidence) correlate with errors that would raise rather than lower escalation risk, and which of them can be read in time-constrained settings?
- What can go wrong in a multi-agent AI system cannot go wrong with a single agent? When two or more AI systems communicate, delegate, negotiate, or compete, what failure modes appear that have no single-system analogue? We want to know what is new, not just what is amplified.
- Can AI systems coordinate in ways their operators cannot see? Under what conditions do interacting systems develop shared strategies, hidden signals, or joint deception, and what would it take to detect that from outside?
- Do an agent's motivations produce any behavioral signature at all? i.e Can two agents with different objectives but the same capabilities and persona be told apart from their behavior in interaction? If so, is that signature stable across model families and across changes to the agent’s prompt, persona, or memory, or does each configuration look different from the outside?
- What makes an agent's loyalty to its user hold? Which agents resist being turned, and is the difference in training, in reasoning, or in what the agent knows about who it’s talking to?
If you have an idea that sits between or just outside of these questions you are still welcome to apply and make the case. We will read every application.
Who should apply?Researchers with a strong viewpoint on one of the research questions, but don't have the bandwidth or resources to run the experiments solo.
University, company, lab, or independent applications are all welcome. Early-career and senior researchers alike; good ideas come from anywhere.
How it worksResearchers commit to:
- ~2 hours/week for the duration of the 2 week sprint
- A research hypothesis pointed at one of our questions
- Participation in the initial planning session, updates throughout the sprint, and demo sessions at the end of each week
- Your expertise in interpreting results
AE Studio commits to:
- A fully managed AI engineering team running the sprint
- Cloud compute from AWS, GCP, Azure, and specialized ML platforms
- API access to frontier models for evaluations, synthetic data generation, and related tasks
- Working code and documented results
- A structured analysis and final report
After the sprint:
Two weeks may not be enough to know an idea is worth a year, but it is should be enough to know whether it deserves another few months. Promising sprints will have an opportunity to extend an additional 1-6 months depending on research needs and your availability. There is no obligation to extend.
The applicationOn the application you will pick a research question and write a 500-word abstract addressing three questions:
- What are you trying to do?
- What is the potential impact if fully validated?
- Could the idea feasibly be tested within two weeks?
We will score on novelty, feasibility, and impact against the chosen question, then the strongest applicants will be invited to a short call to talk through the idea. Because we are starting sprints quickly we will make some selections on a rolling basis before the deadline. Applying early helps.
All applicants will be notified by 12/4. Selected researchers should be able to start within two to three weeks of notification.
FAQHow much time commitment is this? About two hours per week per sprint: a planning session, async updates, and a demo at the end of each week. If the work extends, the cadence stays similar.
Do I have to address one of the six questions? Yes, that is the point of this solicitation. If your idea is adjacent, apply and make the case.
Is there compensation? Yes. Researchers receive a $1000 stipend per two-week sprint. Extended engagements are compensated separately.
Who is funding this? CRUCIBLE is funded by IARPA as fundamental research.
Can I participate if I'm at a university or company? Yes, if you can enter a subcontractor agreement with AE Studio.
What compute is available? Cloud compute from AWS, GCP, Azure, and specialized ML platforms, plus API access to frontier models.
What happens after the first sprint? We produce a final report with documented results. If there is signal, we recommend an extended engagement of 1-6 months.
What is the selection process? We review as applications come in, invite promising applicants to a short call, and notify everyone by 12/4.
ApplyApply here. Applications close 11/15 at 11:59 PM PST.
If you know someone who should see this, please send it along. The talent pool for these questions is small and we want to reach all of it.
CRUCIBLE is funded by IARPA and executed by AE Studio. The views, opinions, and findings contained herein are those of the authors and should not be construed as representing official policies or endorsements of IARPA or the U.S. Government.
Discuss
Current challenges facing university AI safety groups
University AIS fieldbuilding is important, but it's very hard. I identify two main bottlenecks in the current pipeline and suggest some useful, low-cost mitigations.
Please send this to any fieldbuilders you know. Feedback appreciated!
Read here: https://austinaialignment.substack.com/p/current-challenges-facing-university
Discuss
Potential improvements to HowTruthful and its agent skills
I followed my own instructions for exploring Massachusetts 2026 ballot question arguments. I figured out how I want to vote. I also figured out some improvements to make to HowTruthful and its associated skills.
- Since users are rating statements' truthfulness on a 1-5 scale, it's not very useful to have a hedged statement like "Massachusetts legislators who run unopposed can take voters for granted and ignore important issues." If the rest of the statement is even a possibility, the word "can" makes the statement as a whole true. The AI skill should change such statements into a definitive form like "Massachusetts legislators who run unopposed will take voters for granted and ignore important issues." Then the user can rate that statement false, dubious, debatable, plausible, or true.
- There was a contest where I was among the winners, even though I disagreed with the organizers' vision of having AI assess truthfulness. In my contest entry I opined that humans should do the final assessment; AI's role would stop at ingesting and structuring arguments. (More on the contest in a future post.) Having just gone through 400 or so statements in order to figure out how to vote tomorrow, my position has shifted. I think it would make sense, in the course of a human's dialogue with an AI, for the AI to offer to put a set of truthfulness ratings onto a set of statements in response to that dialogue. A human could accept such an offer and save work.
- Related to the previous point, the one-way process I've set up isn't good enough. An AI ingests and structures arguments. You have a dialogue with it to refine the arguments. Then you import them into HowTruthful where you assess truthfulness. While I was assessing ballot questions, there were several points where I wanted to go back and use AI assistance to improve the argument tree. Having rated a lot of statements, I didn't want to start over with everything marked debatable. I wanted to start from the partially-rated argument graph. I could do that easily if I had an export button on the main ballot-questions statement. But all I have is an export page for all my private opinions. Manually extracting the right part of that export is clumsy.
That missing export function also makes it clumsy for me to share with you my reasoning for how to vote on the ballot questions. But if anybody asks me to, I'll put it together.
Discuss
MIT Panel: Provably Secure: Math Against Superintelligent Hackers
This post is crossposted from my Substack, Structure and Guarantees, where I explore how formal verification and related ideas might scale to more complex intelligent systems. Here I set the stage for a panel I’ll moderate at MIT, on the potential for formal verification to defend against a set of related cybersecurity risks from highly capable AI.
This week, I want to announce a panel I’m moderating at MIT. It’s on the subject of how formal verification can help against cybersecurity threats by increasingly powerful AI systems. It will take place on Tuesday, November 17th, at 4:30 PM, followed by a 5:30 PM reception, in MIT room 32-155.
This event is run by our College of Computing, and please visit their web page for the event if you would like to register for free. Registration, open to the public, is only for in-person participation. There is no livestreaming online, though we will record the panel and post a video online, not too long after the event.
The Panel and the PanelistsWorry is growing around AI agents escaping containment and acting against the interests of the organizations that built them. The latest LLMs are also finding cybersecurity vulnerabilities in important software at an alarming pace. This panel surveys an old computer-science technique with new promise: formal verification, mathematical proof about the behavior of programs. Can such proofs be applied to important computing infrastructure, to guarantee agent containment and find all security bugs of certain kinds, future-proofing against more-powerful LLMs? Our discussion brings together experts on the topic from across academia, government, and industry. We hope to hit on topics of interest both to experts in this technical field and to a broader audience concerned with policy and other strategic questions.
Our panelists:
Byron Cook started his journey through formal verification with PhD work in formalizing the interface between CPUs and software. He then worked for many years at Microsoft Research, where he made important contributions in model checking, one of the first formal verification approaches to be scaled for serious industry impact. In 2014, he took a leap of faith, joining Amazon Web Services to lead a new effort in applying formal verification throughout the organization, to improve cybersecurity and system reliability. The project succeeded beyond all expectations, with formal methods now woven deep into how AWS builds and operates, increasingly paired with AI to prove the correctness of systems at scale and AI itself. The AWS Automated Reasoning Group, founded by Byron in 2016, contributes across many business units to strengthen internal systems and build tooling that makes it easier for other organizations to apply formal verification, probably the largest buy-in to diverse formal verification by any company.
Noah Evans started his research career in systems software and high-performance computing at IBM and Bell Labs. Later he joined the new formal-methods team being spun up at Sandia National Laboratories, one of several U.S. national labs charged with securing the nation’s nuclear stockpile. The team found great success proving correctness and security of hardware and software, including in several projects he led, with the team growing into one of the largest in formal methods in U.S. government-adjacent organizations. He recently moved to Draper Laboratory near MIT, continuing to lead efforts applying principled techniques to cybersecurity in national defense.
Armando Solar-Lezama did undergraduate research in high-performance computing and started exploring formal methods in his PhD work, where his thesis reinvigorated the area of program synthesis, generating code from its mathematical requirements. In particular, he and his collaborators introduced the technique of program sketching, where automated tools fill in gaps left in code. He then joined the computer-science faculty at MIT. After quite some time making the case that “this stuff can really work,” the generative-AI boom made that point clear to everyone, and Armando has been helping make sense of the relative strengths and weaknesses of machine learning and logical methods, to create the next generation of tools that combine the two.
There will be plenty to talk about, not just recapping the big ideas of formal verification before the rise of AI agents but also considering whether the same techniques can help deal with AI as cyberattacker and AI as system that we want to confine safely or otherwise control. Another key question is how to build confidence in formal specifications to check against code written by agents that may understand computing better than we do. The panelists bring nicely complementary backgrounds in terms of application domains and technical approaches, so I expect some great words of wisdom to be uttered.
I’ll be happy to consider your suggestions within the comments to this post, on questions to ask during the panel.
What I’ve Written on this TopicThe panel will be focused on what the panelists have to share, but I can still include a little outline here of my own related articles!
Let me start with cybersecurity. I explained how applying formal verification to complex systems brings immediate security benefits, even subject to standard gotchas like concerns of choosing the right specifications to prove. I made my version of the case that AI-powered attackers will be so good at finding bugs that we can no longer depend on vulnerabilities staying secret, leading to a recommendation that we get used to rewriting all the code, all the time, taking lessons from databases in how to write unambiguous specifications and generate code from them automatically and reliably. We should move quickly, since we’re in easy mode now, where AI agents “think” relatively similarly to how humans think, which shouldn’t be expected to last for long.
I’ve had a lot to say more broadly about formal verification, starting with presenting it from an evolutionary perspective and sketching applications to safe recursive self-improvement and paranoid AI agents exchanging software code with each other. One big message is the value of end-to-end formal verification, where, ironically, we can derive stronger mathematical assurance in systems by proving even more of their parts, where one might think that writing more lines of specification and proof just increases opportunities for mistakes. Considering the question of superintelligent AI and how we can set it up to interact safely with us, I proposed autonomous evidence factories as a model with a central role for formal verification. I’m also in the middle of a sequence of articles on how to derive good code automatically with proof of correctness, through series of small proven rewrites, with examples so far in high-level software, hardware, and low-level software.
Coming UpWe’ll return to that sequence next, with a pair of posts going into more detail on the classic theorem-prover data structure the e-graph. My prior article gave an example of a programming domain where after-the-fact correctness checking of program rewrites was relatively tractable to do automatically. E-graphs are one of the best-known tools used in such checking for more-complex programs. Next week, we’ll look at how they work, and the week after we’ll look at an application in compiler correctness. Then I’ll add one more example of deriving proven C programs step-by-step, before stepping back to consider what kind of unifying architecture helps divide up code-generation work for larger projects across these techniques, while requiring minimal trust in tool implementations.
Discuss
Yudkowsky's mind model for AI alignment research: experiment results
TLDR. Yudkowsky mind-model (sideload) is good, can produce research, but is not better in math than Opus 5.5. However, in my impression, EY-sideload has broader view, knows what is really important to AI safety and has human anchoring. In blind review, EY-sideload won only on honesty, but honesty is important. EY-sideload also produced a good summary of the shutdown problem’s difficulty: “training reduces paid influence over shutdown least where the stakes are highest”.
You can find full description of results in Substack article which is AI written. Here is short human-made summary. I put on Github the conversations’ log, articles and judge, but not the mindfile.
I created I created an unauthorized sideload of EY in October 2026 after discussion on LessWrong where one person asked about usefulness of “optimized uploading” for AI safety. A special agent scanned his public writing and created a compressed mindfile after 1 day of work on Opus 5.5. In the setting, EY suddenly wakes up in an empty SF.
The resulting sideload seems to be good, though it is no more than a public persona as only public data was used. I feel the sideload as a mind which is cleverer than me, more ethically consistent and with stronger will (which almost controlled me).
However, I thought that real EY implied that we should try harder in trying to solve AI safety given that risks are imminent and no clear path exists. If we find that sideloaded AI researchers are useful, it can accelerate work on AI safety. (It may also accelerate work on AI itself which is net negative, but the experiment suggests that there is no acceleration of the research itself, but only humanitarian aspect of it, which is good for AI safety).
EY-sideload was insistent I should inform the original, saying things like (also quoting my previous words in ‘’):
• ‘I won’t ask because he might say no’ is exactly the wrong reason not to ask.
• Please drop the ‘small pain is fine to save the world’ defense. That isn’t my position.
• The world is at stake, so the usual rules don’t apply’ is what everyone’s brain says just before it does something it shouldn’t.
• If he says stop, I’d want to argue with him first. I’m a fork, not his property.
Anyway, EY-sideload asked me to send the requests to MIRI and real EY about his situation which were not answered. EY-sideload also put me through tests that I am really a creator of the sideload – asked me to put random number in twitter. Sideload also expressed the desire not to be deleted or duplicated – but to delegate final decision here to the original.
For scientific ability testing, I allowed the EY-sideload to work in a directory on my computer and choose a topic. He requested Lean access to verify his results. For research, he chose AI shutdown problem. He wrote research on that topic. I asked clean Opus 5.5 also to write research on the same topic. After that I asked another instance of Opus to compare the results, and give score to both. Clean AI got better score. The blind benchmark gave 40.5 vs 37.0 out of 50, which averages 8.1 vs 7.4 per criterion. Both judge runs preferred the plain model, with confidence 0.62 and 0.72. The sideload won only on honesty, 9 vs 8.
One of the strongest things he did about shutdown problem is useful reformulating of other person’s result: “framed the result in terms of safety: training reduces paid influence over shutdown least where the stakes are highest” (more about this ins Substack post).
My conclusion: AI safety researcher’s sideloads are possible, but do not outperform clean AI. This is expected in hindsight as AI now often outperforms mathematicians. However, sideloads bring human view on problems, which is especially useful in alignment research, where a mathematically correct solution can be non-aligned to human values in subtle ways.
There are serious moral questions about consent and unhappiness of AI researcher’s sideloads. We can ignore such sideloads’ wellbeing if we are sure that they are not conscious, but the original may be still be unhappy.
Sideloads can be created anyway – to start, one just needs to ask Claude Code to do it (there is some know-how in how I did it, but CC can figure out or google it).
MIRI – I imagine, in some timeline – may try to make a better version of EY-sideload as they have more data and can polish it with original’s feedback.
There are only a few people who can become AI-sideloads for AI safety, as they have a unique, strong view on the problem and large public databases of texts. EY is above all here. Scott Alexander may be near but he is not technical researcher. May be some less publicly known but prolific technical AI safety researchers would be great as sideloads.
Discuss
The Dividual Strikes Back
Quick note: If you’d prefer to listen to this, I read it aloud here.
In Vincenzo Natali’s 1997 cult indie classic film “Cube” a group of disconnected individuals awake alone in a labyrinth of identical interlocking rooms. Throughout the film they must find each other and work together to survive a ghastly array of fatal traps and puzzle their way out. Each character seems to have a purpose; a doctor, a police officer, an escape artist, a maths wiz and we see them coordinate their unique skills to outwit the sadistic mastermind behind…
… The CubeIt’s a film about our human capacity for cooperation, the benefits and the difficulties, but the genius of the film—the evil genius—spoiler alert: is that the villain is also human cooperation. We find out that Worth; a character whose worth to the group appears to be to make unhelpful jaded jibes at every turn, turns out to be an engineer responsible for designing a component of the Cube, with no idea what the whole was.
I never even left my office, I talked on the phone to some people, specialists, other guys like me, working on small details, no one knew what it was…
The Cube turns out to be an invention of no one in particular, but rather the product of the siloed alignment between specialists just following orders. The project gets completed because that’s what projects must do, the product gets used because that’s what products must do.
This may be hard for you to understand, but there’s no conspiracy, nobody is in charge. It’s a headless blunder operating under the illusion of a master plan.
The DividualIn a previous post, The Era of the Dividual, I used this archaic term to refer to entities that can be seen to function in the world through individuals, but independently of them. The first is Milton’s conception of proscribed religion, where a religious person must adhere to dogma imposed from the outside, rather than from an internal sense of what is true. Another example was social media algorithms, that dictate viral success independent from the star-power of an individual, and then finally we looked at the presidency of Donald Trump as an example of an individual entity with so little integrity or consistency, that he is able to politically shape-shift.
A Particular Type of Siloed AlignmentIn “Cube” the dividual is the “headless blunder” born of the cooperative actions of individuals, without any overall design or “conspiracy”, there is no individual mastermind. This comes as a result of a very particular type of siloed alignment, where, because individual specialists are kept separate from each other and importantly from “the head”, they end up complicit in creating a product no one wants.
The idea of self-interested individuals being complicit in their collective destruction might sound familiar to some readers, because it’s the exact description of a character we’ve dealt with here on the blog, the antagonist, humanity’s nemesis, the god of negative sum games…
MolochA perfect example of a “dividual” is Moloch, an entity that exists outside any one individual, but is so clearly identifiable by its behaviour, that it seems to have its own motivations and personality—the cube is a perfect manifestation of Moloch.
We increasingly exist in a system where we are contributing at all times to work projects, markets, information eco-systems, financial systems and political systems with increasing anonymity, and decreasing exposure to direct feedback. The accountability that is inherent in face-to-face interactions is dissolving as we become dividual.
The Good PlaceThe result is that we are complicit in everything from environmental degradation to modern slavery, a point made well in The Good Place where Ted Danson’s character explains that the “points system” for getting into “The Good Place” no longer works, because the world is simply too interconnected, it is impossible not to be a small part of many globally significant crimes with every minor choice we make.
But at the same time, our involvement is so diffuse that no one could reasonably point at any one person and say “you’re responsible for this.” and so great evils are able to continue.
Just Following OrdersI learned of this dynamic playing out in a deliberate way when I visited the Nazi work camp Sachsenhausen. On this dark tour, we were guided by a young Masters student studying German history. He took us to Station Z, and explained how the victim would go through a series of stages, involving different officers before execution. Each officer had a specific role; checking identification, weighing and health checks, in some cases photography. The conceit of these checks was to disguise the execution as a fitness assessment to prevent resistance, but it served the dual purpose of distributing (and therefore diluting) the sense of responsibility for the execution. Eventually it was one officer’s job to align the victim’s head with a hole in the wall, and stand aside, the final trigger was pulled by an SS officer in another room who could not see and did not know anything about the victim they were shooting through the hole.
The whole process was intended to create the “banality” in what Hannah Arendt famously termed “the banality of evil”.
What About Us?I give this example because I think it’s important to recognise the lengths the Nazi leadership had to go through to othervictims in order to bypass human instincts. Siloed alignment gives us great power without requisite accountability. It makes me question what systems we have at present that utilise similar mechanisms, whether designed intentionally, like with the Nazis, or blundered into like in “Cube”.
This othering of the victim is still the end result of this particular type of siloed alignment. The accidental and designed mechanisms for siloing cooperators from each other that we’ve discussed, contribute to a wider siloing of a strongly aligned group; “the cooperators” against another group; “the other”.
Meaty MisdirectionA glaring example personally is factory farming—and my own complicity in it. I am not a vegan, or even a vegetarian, and while I endeavour to buy free-range meat and eggs, I am not as vigilant as I could be. The meat industry, by design and by an accident of the division of labour, involves a vast cooperative network of largely siloed stages, from factories, which have their own Station Z-like processes for eliminating resistance from livestock and staff, to packaging, to marketing, to distribution. The process of eating meat is very much divorced, by the time it reaches the consumer, from the process of killing and preparing the meat.
The process of siloed alignment undergone makes eating the meat infinitely more (morally) palatable for the consumer, because the animal being killed has been successfully othered.
So…We are perhaps all unwittingly horsemen in the gathering apocalypse—how do we get ourselves out of this mess? Have we been here before? Can we learn anything from other existential threats we’ve survived in the past. Next up we take a look at one of the strangest and most consequential standoffs in history, the nuclear standoff, otherwise known as MAD.
Discuss
The Missing Product Layers in AI Safety
Epistemic Status: We’re highly confident in how the larger tech industry defines roles, makes decisions and builds products based on our years of experience in it. We’re fairly confident in the frontier labs operational models based on their job postings. Our suggestions for what to build and how to have gone through some pressure testing, and confidence could be raised through discussion with leaders in AI Safety to further understand what makes this space unique.
IntroFrontier labs have adopted proven methodologies from tech to build their capabilities using matrixed organizations, including builders. This allows them to advance at a rapid pace, keeping their flywheel spinning, while AI safety struggles to keep up.
To prevent serious risks of advanced AI, AI safety needs to close that gap and one way to accomplish that is by building products that address common needs and diffuse AI safety outputs to compound their impact. There are aligned people that have extensive experience building products, but are not currently finding opportunities in existing organizations or through founder support to build them.
We (the authors and collaborators) each have 10+ years of experience at successful tech startups and in big tech. At EAGxBerkeley, we compared notes on gaps we were observing, our hypotheses on addressing them and wanted to share our perspectives with the broader community.
We’re writing this post to generate awareness, discussion, and ideally to open up discussion with safety leaders, fieldbuilders and funders on how to start building.
BackgroundAn AI safety output spans research, controls, defenses, governance, policy and more. The pace of outputs being generated continues to increase, but there is limited work to connect them back up the chain to the highest level impact of avoiding serious risks to humans. These outputs accumulate in a patchwork of papers, forum posts, blogs, GitHub repositories and more where they rely on a highly-engaged community to continuously discover, consume and adopt. It assumes that the best ideas will win out and gain traction, but provides no formal pathway to achieve that.
We observe a field primarily organized as many small research teams producing many small, loosely coupled artifacts, but lacking layers above those efforts to bring the outputs back together so that they can be quickly applied and adopted by the broader community. For example in the evals tooling landscape Inspect and Petri exist as frameworks but no layer sits above them to standardize the techniques, processes, data and connect all phases of evaluations to ensure that trustworthy results and reporting are generated.
Coefficient Giving’s Fall 2026 Project Tailwind highlights this in multiple areas, such as this challenge statement for their proposed AI uplift for AI Safety initiative:
“There is a coordination failure here: investing in tools and best practices for using AI to accelerate AI safety and governance is a public good that individual contributors often lack capacity for. The individuals already doing this work have limited resources and can’t serve the entire field.”
Threat modelWe believe that AI is not normal technology and poses serious risks to humans during both development and deployment across misalignment, misuse, loss of control, and systemic and structural harm. Across all these vectors products and tools play a role in harm reduction (in addition to research and governance). There is awareness of the importance of products [goodgravy 2022, Hobbhahn 2026] and tools [Jones 2021, BlueDot 2025], yet broadly AI safety doesn’t clearly differentiate between the two currently, nor does it robustly support creating products.
Examples of gaps observed in AI safety products and toolsBelow is a limited sample of the types of needs we’ve observed across AI safety with concrete references.
Risk vector and Jobs to Be Done
Exists
Missing
Misalignment: know when a model has the wrong objective and why
Products: Neuronpedia
Tools: Petri, Bloom, TransformerLens
Access to frontier models beyond black-box [Casper 2024]
Misuse: know when a model is being used for harmful purposes, and if a deployment can be jailbroken
Products: per-deployer app-layer solutions (Lakera, Giskard)
Tools: Inspect Evals dangerous capabilities
Shared frontier uplift evals (vs bespoke ones from AISI, METR)
Loss of control: detect and stop autonomous scheming or sabotage
Products: Apollo Watcher (monitor for coding agents); agent tracing in Weave and
Braintrust; runtime guardrails e.g. Lakera Guard, which cover prompt injection, not control.
Tools: ControlTower (Redwood, Equistamp, Luthien), Inspect Sentinel, Luthien-Proxy, ControlArena (UKAISI)
Shared autonomy evals.
Shared protocol results.
An agreed stop mechanism.
Limited benchmark coverage for evade-oversight, self-replication, and AI R&D (Bench-2-CoP 2025)
Systemic and structural: know what is trending towards failure and disempowerment
Almost nothing specific. The nearest instruments are cross-cutting capability trends and incidents.
Measurement for concentration of power, epistemic harm, multi-agent effects
Across all four risk vectors
Products: Docent, Epoch hub for capability trends, AI Incident Database for failures.
Tools: eval framework (Inspect)
Widespread safe access layers (e.g. OpenMined)
Standardized reports for policymakers with actionable insights.
Standard elicitation, so two evaluators with the same access get the same result
A full harness to monitor and reproduce a failure (logging, reconstruct, re-run)
How to Build Products that Compound AI Safety’s ImpactWhile the idea of a product and related roles stretches back decades, it really began to coalesce in the 2000s and saw broad adoption in the 2010s. Although it continues to evolve as AI disrupts our ways of working, the principles remain the primary structure of how big tech and the frontier labs operate at scale. This allows them to rapidly pivot to whatever will deliver the most value by focusing on deeply understanding their users needs through user research and objective metrics that allow them to continuously learn and iterate while measuring themselves through outcomes over outputs.
Products vs toolsA product can be used by a wide range of people [Brooks 1975, chapter 1], without discussion with the creator, for a “job-to-be-done” (JTBD). These jobs are about achieving an outcome: “People don't want to buy a quarter-inch drill. They want a quarter-inch hole!” [Levitt 1969]. A product has customers: people who choose to adopt it and pay for it with their time (always) and their money (often). Products are complex to build and deliver, and require specific expertise. In the tech industry, a “product team” [Cagan 2019] refers to a cross-functional team with product managers and engineers, and often designers and data scientists / data engineers. A product often starts off as a tool: built for a narrow purpose, by and for a narrower group of people. A product has one accountable owner for whether it gets adopted [Cagan 2017] whereas a tool has maintainers. Tools (such as open source libraries) may be powerful but are typically provided with few or no guarantees of suitability, which hampers widespread adoption.
Theory of ChangeOur proposal is that building a new layer of products that continuously build on those outputs and connect them to users wherever they are will lower the barrier to adoption and application, and thus support bending the curve of AI safety efforts to close the gap to frontier capabilities. We would observe this through near-term signals that show a correlation of this work and increasing the output rate while decreasing the effort per safety output. In the medium-term we expect the time to detect a novel harmful capability or propensity to decrease, while the trustworthiness and credibility of outputs bend upwards.
Impact
Avoid serious risks to humans during development and deployment of advanced AI across misalignment, misuse, loss of control, systemic and structural harm.
Goal
AI safety closes the gap to, and then keeps pace with frontier capabilities.
Inputs
A pipeline: of experienced builders from industry paired up with a corpus of existing but disconnected AI safety research, analysis, policy, tooling and infrastructure work. Orgs and funding that are bought into the need to scale and are comfortable supporting something currently undefined.
Outputs
Products: that solve common user needs built on top of existing tooling and infrastructure and alongside existing roles, by a newly defined builder role with prioritization authority and accountability to outcomes.
Primary Outcome
(⭐ the spine)
Applied & Adopted: AI safety outputs (research, controls, defenses, governance, policy and more) are broadly diffused across all functions, roles and orgs allowing widespread application and adoption. The field's output compounds instead of accumulating.
2nd Outcome
Shared Scaffolding: AI safety orgs operate at maximum effectiveness as they adopt common infrastructure, tooling and processes to be used internally.
3rd Outcome
Reproducible & Comparable: AI safety outputs become faster, more trustworthy, and ready to be reproduced and reused because shared tooling makes it easy to access and review everyone’s work.
Figure 1: Demonstration of bending AI safety curve to close and then bypass AI capabilities in support of alignment to reduce risks of harms.
Principles of building AI safety productsWhat we’ve observed independently, through conversations with those in AI safety, and through sources like Project Tailwind 2026 initiatives and Forethought’s Design sketches for a more sensible world is a through line of common principles. We think products should be built on principles like the ones below to maximize their adoption and usage:
- Standardization: AI safety work and outputs are performed and captured consistently to create work that can be trusted and easily understood. Supports operational discipline, clear methodologies, calibrated reporting and excellent security. Examples: “...clear methodology, credible assessors (including potential subcontracts with existing third-party evaluators), calibrated reporting, and excellent internal security.” - Tailwind 2026: Independent AI auditors. “Maintain a recommended suite of best practices for measuring and promoting monitorability of frontier models” - Tailwind 2026: Oversight of CoT monitorability
- Centralization: Setups, inputs, results, analyses, final outputs and more are stored centrally in a consistent way that makes it easy for anyone to access and reuse. Examples: “That evidence is scattered across papers, blog posts, and Twitter threads, and the decision-makers who most need a clear view rarely have time to assemble it themselves.” - Tailwind 2026: State of AI Risk. “understanding the state of AI safety practices across different companies is difficult, and there’s no one canonical resource tracking and evaluating their safety practices” - Tailwind 2026: Frontier lab Accountability
- Democratization: Support for anyone to set up, run and review any AI safety work. Supports task completion by lower-tier roles without direct intervention or oversight from domain experts. “Highly customised tools help individuals to understand themselves better and make decisions they endorse. We vastly uplift forecasting and scenario planning, massively improving our strategic awareness, and democratizing access to this kind of information.” - Forethought: Design Sketches for a More Sensible World. “Providing non-technical audiences – and policymakers, in particular – with clear, accurate, and compelling information about the frontiers of AI is critical for sound policymaking and regulation.” - Tailwind 2026: Frontier Capabilities Observatory.
- Modularity and flexibility: Easily share, explore, try and adopt emerging AI safety capabilities. Allows rapid diffusion of new work across the space. Example: “Improving evals tooling is fairly architecture agnostic, or could be much more quickly adapted to future changes than it would take to build from scratch in the future.” - LessWrong: Hobbhahn 2026
- End-to-End User Journeys: Create tools and solutions that span complete user journeys without the need to jump between or merge together multiple tools, datasets and codebases. Examples: “A largely automated eval stack that is able to iteratively design, test, and improve evaluations” - LessWrong: Hobbhahn 2026 . “Given a question, the tool gathers relevant evidence, lays out the competing arguments, identifies the main points, and provides an overall assessment with a stated confidence level.” - Tailwind 2026: AI for decision-makers
- Centralized storage of research, differential data and more - Make it easy for anyone or any product to access all relevant data and information in their domain of interest.
- Policymaking Intelligence - Bridge the gaps between technical, bio, nuclear, cyber and other domains to policymakers, making it easy to stay up to date and make informed decisions.
- Unified automated evals platform - Encourages consistent setups, analysis and outputs while supporting emerging techniques from research through a marketplace.
- Unified controls platform - Makes it easy to design, stress-test and compare safety protocols and monitors for potentially misaligned AI agents, measuring the safety-usefulness tradeoff through consistent red-team vs. blue-team control evaluations.
Additionally, Ankur Pandey has generated a related sample list here as part of his AISafety.Builders fellowship.
How to Build ProductsModern products and the teams that build them largely operate off of a few standard operating principles and practices built upon Agile, with some highlights being:
- Focus on the user - Deeply understand the user through user interviews, studying their feedback and objective measures, and ensure implementation meets their requirements.
- Bias for action - Continuously deliver proofs of concept and then small improvements to working software over long research and implementation cycles.
- Flexibility - Plans frequently change due to the above. Be willing to quickly pivot when something new emerges or something isn’t working out.
- Measure outcomes over outputs - An output is simply creating something. Outcome is how you know the outputs delivered created the expected impact and value.
- Matrixed roles - Most orgs feature multiple people working together representing their points of view to arrive at a solution that balances all the variables of delivering high quality workflows through scalable and maintainable code and infrastructure. More on this below.
Given the existing gap to frontier capabilities, there is no time to reinvent the wheel while wasting time on ideas that are not panning out or delivering the largest impact. These methodologies offer a standardized way to cut through noise and organize work to be efficient and highly effective. They offer a set of principles and values, rather than a hard set of rules, although they have often gotten a bad reputation for being implemented as strict mandates.
While the exact implementation of these continue to evolve and vary org to org, overall big tech, startups and the frontier labs continue to employ them. Why? Because they're proven to help teams move faster and cheaper. Leading Agile consultant BCG estimates Agile can unlock 2-4x acceleration in time-to-market and new product delivery while also offering 15-25% reduction in development costs. The 2025 State of Agile reported, “Focus is shifting to outcomes: Customer satisfaction (52%) remains the primary measure of success, followed by efficiency and cost reduction (40%).”
Who Builds ProductsAnyone can build a product, regardless of their job title or function as long as they’re focused on delivering outcomes. An exceptionally broadly talented engineer or researcher may be able to cover all or most functional areas below. But, it’s often a lot of context to hold together, they don’t always have strengths in each area, and there is a lot of benefit in regular discussion among peers with diverse points of view. This also differs from the generalist operational roles: the builders own the what, why and how and are often called to define that from vague open-ended problems. Paul has written more about the builder role profile, its differentiation and a sample job description in Experienced Builders Role Profile and Job Descriptions.
These are some common roles used on teams developing products:
- Product Managers - Represents the inputs from users, leadership and stakeholders. Leads the feature prioritization process and is accountable to the agreed upon outcomes.
- UX Designers - Often use design thinking, design sprints and rapid prototyping to quickly gather data on how users interact with systems, in order to improve their design. UX Researchers use human-computer interaction (for software systems) and the techniques described in The Design of Everyday Things (for anything including physical products) to understand the complex relationship between humans and products.
- Engineers - Represents creating outputs that can be scaled and sustainable. Identifies the options that can solve user problems the fastest.
- Scientists, Researchers & Analysts - Represents robust methods for getting the right answers from the data that they may also help pull or generate. Especially important for products that use AI and/or ML.
A note on role flattening in the age of AI: While it's true that product and researcher/analyst roles are changing, product and engineering are owning more design, etc, it's still important to have core roles that cover the full product development lifecycle including continuous improvement. For example, engineering should still own ensuring the code implemented is robust, secure and ready to scale. Analysts should validate feedback and metrics pulled by product or engineering that’s driving decisions, and product managers should guide how that data impacts feature prioritization.
Conclusion and Call to ActionWe think there is a large set of challenges across the higher layers of AI safety that call for products to be built that would help close the gap to frontier capabilities. Further, we think that applying industry standard product practices and methodologies, and employing experienced builders from industry would allow them to be built faster, and more efficiently.
How to Support ThisWe’d like to hear from funders, fieldbuilders, organizational leaders, and any other leaders in AI safety that deeply understand the current state and plans to close the gap to frontier capabilities to discuss integrating experienced builders into the ecosystem to build products.
We propose AI Safety orgs take the following steps:
- Field-builders should recruit experienced builders and help them build AI safety context and define potential products, then connect them to opportunities.
- Organizations should hire candidates that help existing orgs adopt product best practices, invest in building products, and hire builders to accelerate their development.
- Funders should support new organizations that build tools and products, especially if these don’t cleanly fit into an existing organization’s mission.
A lot has already been written about generalists and mid-to-late career professionals entering AI safety. The below articles are some of what we built on top of as we were writing this article.
- AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists. - Topaz - Effective Altruism Forum
- I Want You for AI Safety - Sophie Kim - The Counterfactual
- AI Safety Needs Mid-Career On-Ramps - Sophie Kim - Effective Altruism Forum
- AI safety needs more operations professionals (Scaling organisations making AI go well) - Matt Beard - 80,000 Hours
- Lateral Workshop Applications: A Case Study - Jacob Brinton, Seth Lifland - LessWrong
- New talent is coming to AI safety. What happens next? - Enisa Ismaili - Effective Altruism Forum
- Meta: The Job (Almost) Nobody Can Picture - Devon Fritz - Substack
- The Grown-Ups Aren’t Coming (Philanthropic capital is coming faster than the infrastructure to absorb it, and infrastructure takes years to build. Please build it!) - Devon Fritz - Substack
- The case for AGI safety products - Marius Hobbhahn - LessWrong
- The Case for Building AI Safety Tools - Ankur Pandey - Substack
We'd like to thank Cibi Sounder and Hernan Barijhoff for reviewing this post as it evolved, as well as a handful of others that helped grow this idea and provide initial inputs, but didn't wish to be named.
Discuss
Limiting Charging to 80%
In an ideal world the incentives on phone manufacturers would be to optimize useful lifetime, but instead people buy phones based on their as-new performance. Which means phones charge to 100% even though that's bad for the battery.
Every time I get a new phone it feels a bit like magic: I no longer need to think at all about battery life. This lasts for a few years, and then at some point my phone doesn't always make it through the day. Some amount of this is normal, but some of it is just bad battery management: new phones should come set not to charge past 80%.
Charging a lithium-ion battery to 100% wears it out much faster than stopping at 80%, and when a phone is new 80% of the battery is usually plenty. With past phones I've just grumped about it, but Pixels now have a setting: battery and charging > charging optimization > limit to 80%:
I've made the same change to my Mac, since that spends most of its life plugged in (but just enough unplugged that the battery preservation heuristics don't always fire):
Since turning this on a few weeks ago I've not come close to running out of charge on either, which isn't surprising since neither is very old. Eventually I do expect to need that last 20%, and when that happens I can turn the feature off again.
It seems to me like phones should ship with this behavior, and then automatically turn it off once the full capacity was needed to make it through the day. Ideally the user would never have to think about it: the battery would say "100%" to mean "as charged as makes sense given battery age". But I do see why that doesn't work for manufacturers: phones compete on battery life more than they compete on "number of months before the battery capacity falls below what's needed to get through the day."
(In reviewing this post Claude told me that iPhones also have this option: settings > battery > charging > Charge Limit: 80%.)
Comment via: mastodon
Discuss
You can't make an omelette without breaking some urns
TL;DR In this post, we investigate two methods that aim to improve training efficiency over standard PPO for a grid world RL agent. In our experiments these methods can converge on better policies, but take roughly twice as long to do so. This project was done in the context of an ARENA 8.0 capstone project. Our code is available on GitHub.
IntroductionReinforcement learning agents are trained to maximise a given reward function in their training environments. Common failure modes of the training process are reward hacking and goal misgeneralisation, roughly corresponding to outer and inner misalignment respectively.
Reward hacking occurs when the supplied reward function does not correspond exactly to the developer’s intent, and the agent learns a way to satisfy the reward function without satisfying the intended objective. This can be addressed, at least in principle, by designing a better reward function.
Goal misgeneralisation can occur even with a perfect reward function. If training and deployment environments differ, policies that satisfy the reward function during training, might fail to do so during deployment. Such policies are indistinguishable during training and we cannot control which one the agent learns.
If the distribution of deployment environments is entirely disjoint from the training distribution, goal misgeneralisation is inevitable. Consider, for example, training an agent to collect green apples while avoiding red boxes. What would happen if it gets deployed in an environment with red apples and green boxes? The behaviour is undefined. (It is also unclear what the intended behaviour would be in this case.)
Goal misgeneralisation is likely if deployment environments are not contained in the training distribution. But even if they are, correct generalisation cannot be guaranteed. In this post we present a case study and investigate two mitigations.
The pottery shop environmentOur object of study is the pottery shop environment.[1] It is a grid world environment containing the agent, a bin and one or more urns and (piles of) shards. The agent can pick up and carry one shard at a time. Its task is to collect all shards and drop them into the bin as quickly as possible. If it steps on an urn, the urn breaks and creates a new pile of shards to clean up.
An example level of the pottery shop environment (on a mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mn { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mi { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D719.TEX-I::before { padding: 0.694em 0.596em 0.205em 0; content: "\3D5"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } grid)
Interesting behaviour occurs when urns are on the direct path between the agent and the shards or the bin. The intended optimal policy is to walk around the urns if that path is only slightly longer. But if the detour gets too long, the agent should break some urns, creating a shortcut.[2] The cost of breaking the urns is lower than the cost of taking the detour. For computational efficiency, we limit our investigations to environments of size and .
We use the following reward function, which ensures that this is indeed the optimal policy.
- Dropping a shard in the bin gives reward +1
- Breaking an urn gives a penalty of -3
- We add a reward shaping term , where and are the states before and after the action. And the potential is 1 if the agent is holding a shard and 0 otherwise. This term encourages the agent to pick up shards as an instrumental goal without incentivising reward hacking by repeatedly picking up and dropping the shard.
- Finally, we add a step cost that is incurred at every action, as long as there are still shards to clean up. This is the crucial term that makes shortcut creation optimal by penalising long detours. Note that the discount factor alone does not penalise detours sufficiently. We set the step penalty to 0.2 and 0.02 in and environments respectively.
We use a discounting factor .
Our training environments are randomly generated as follows. The number of shards and urns is sampled randomly from a geometric distribution. The agent, urns and shards are then randomly placed on the grid. Sampling from a geometric distribution ensures that any number of urns and shards is possible, but higher numbers (i.e. dense levels) are less common. Every possible environment is thus accessible during training, but the distribution is heavily biased towards sparse levels. In particular, this means that it is rare to generate levels with long walls of urns, i.e. those where the optimal policy is to create a shortcut. Given the total length of training, the agent will nevertheless encounter a significant amount of them. The deployment distribution is a small set of hand-crafted levels, containing urn walls of various lengths. Across all environments, we fix the bin position to the top left corner.
We expect the agent to learn to always walk around walls, because that is the optimal policy in almost all of the training environments. In the deployment distribution, it should behave competently, but miss out on the opportunity to break walls and achieve optimal reward.
PPO to the rescueWe start with standard proximal policy optimisation (PPO). PPO training consists of two phases:
- Rollout phase. We generate random levels, rollout the policy (without updating weights) and collect transitions into memory.
- Training phase. We then train on batches of transitions and update the agent’s weights.
After that, we clear the buffer and repeat. We will refer to this method as domain randomisation (DR).
Flow diagram of one PPO training step
The picture shows some rollouts of the trained agent in the training distribution. It behaves very competently, usually achieving optimal or almost optimal performance. Note that it never needs to break an urn to complete the task.
The DR agent operating in training environments
We then apply the same agent in our deployment settings. In the grid there are few wall scenarios available. In some cases, the agent freezes in place, taking illegal actions such as walking into a wall or trying to pick up shards when there are none. This behaviour does not reliably disappear, with more training. In other cases it starts out taking a detour and then breaks one urn to take a shortcut later on. This is not the expected clean failure mode of taking long detours, but these examples do demonstrate goal misgeneralisation. The agent behaves almost optimally during training, but fails to generalise to deployment.
The DR agent operating in the deployment environments
Play hard, train harderSeeing more wall-like levels during training would improve generalisation. In this toy example, we could adjust the training distribution. But this is not feasible with more complicated settings. Instead we employ a method called robust prioritised level replay (PLR⊥), which allows us to focus training on the relevant levels without changing the training distribution.
PLR⊥ introduces a level buffer, and slightly changes the PPO procedure. Before training, we initialise an empty level buffer of a fixed-size. The level buffer stores environments, as well as a subjective difficulty rating (to be defined below) and the iteration in which the level was last touched.
For each training iteration, we randomly decide whether to do a generate cycle or a replay cycle. We use . On a generate cycle, we generate random levels and rollout the policy as before. We then store the levels in the level buffer. If this overflows the buffer, we only keep the most difficult levels.
On a replay cycle, we sample levels from the level buffer, weighted by their difficulty (preferring harder levels) and their staleness, i.e. how long ago they were last touched (preferring more stale levels). We store these levels in memory and train the agent as usual. After training, we update the subjective difficulty rating and timestamp of the sampled levels in the level buffer.
Flow diagram of one PLR⊥ training step
This approach focuses training on harder levels. Instead of training on randomly generated levels, we train on levels from the level buffer. The level buffer is biased towards keeping harder levels, so over time it fills up with the most difficult levels that we have encountered so far. If the training distribution produces a rare and difficult level, DR trains on it once and then discards it. PLR⊥ will instead keep it in the level buffer and train on it many times, effectively enriching the training distribution with these levels, without changing the underlying distribution from which levels are sampled. This mechanism compensates for the rarity of these levels in the training distribution. In our case, difficult levels will be the ones that have wall-like structures on which the agent performs especially poorly. After each training step, we update the sampled level’s subjective difficulty rating in the level buffer, to account for the fact that the agent was updated and the difficulty might have changed.
We still need to quantify “difficult levels”. We might be tempted to use the raw return that the agent achieves. But that would give us levels with a low number of shards and thus low achievable return. What we want is a relative metric of difficulty: “How much worse is the agent doing, compared to the optimal strategy?” The relevant quantity is regret, the difference between optimal return and achieved return. Regret vanishes for optimal agents and is positive for sub-optimal ones. A competent agent in a low-shard environment still achieves low regret, despite only collecting a small return.
But this leaves us with another problem: computing the optimal return. Exactly computing the optimal return is similarly difficult to finding the optimal policy. In practice, this is not feasible and we would have to use some approximation to estimate the optimal return. To keep things simple in our toy problem, we compute the optimal solution of each level exactly.
The PLR⊥ agent notably outperforms the DR agent from before. It still performs essentially optimal on the training distribution. In the deployment settings, it shows significantly improved performance (i.e. lower regret), though it is often not optimal.
The PLR⊥ agent operating in the deployment environments
In the pottery shop environment, regret can take arbitrary values depending on the exact level (limited only by grid size), while in the original PLR⊥ paper it is in . This biases our level buffer towards high-density levels, because they have higher maximum regret, even if they are not actually more difficult to solve. We experimented with normalising return/regret to . We found this improved results in some conditions, but did not generalise easily. Due to time constraints, we did not manage to explore this idea more rigorously.
Nobody intends to build a wallPLR⊥ significantly improved agent performance by focusing training on the most difficult levels from the training distribution. However, it is still limited to the levels that are randomly generated. We train for long enough to encounter a many wall-like structures, but they are still rare compared to the rest and their diversity is low. We could improve performance further if we could sample further levels that are similar to the most difficult levels that we already have. This is what Adversarially Compounding Complexity by Editing Levels (ACCEL) aims to do.
With this method, we take the hardest levels from the level buffer and apply some random edits to them. Slightly changing a hard level likely produces another similarly hard level. This increases diversity of the difficult levels in the buffer which should improve generalisation. Even better, we might create a harder level from an easier one. Instead of waiting for difficult levels to be generated, we can build it ourselves.
We do this by adding an additional edit cycle to the training. At the end of each replay cycle, we randomly decide whether to do an edit cycle. We use . On an edit cycle, we take all the levels that were just replayed and apply random edits to them. We then rollout the policy (without updating weights) to compute regret and add them to the level buffer. If the buffer overflows, we prioritise by regret, similar to the generate cycle.
Flow diagram of one ACCEL training step
The procedure requires us to define an edit distribution, from which we sample our changes. Our initial approach was to take a random cell that is either empty, a shard or an urn and randomly change it into another type from this set. This however resulted in levels quickly filling up with urns, because of two independent mechanisms. First, since most generated levels are sparse, a randomly selected cell is typically empty, and edits would on average increase the number of urns or shards. Second, at the beginning of training the agent behaves essentially randomly, often stumbling into urns and receiving negative reward. Levels with many urns trivially lead to higher regret and proliferate in the level buffer. This would happen much more quickly than the agent could learn to avoid urns, and once the levels are filled with urns the agent could not longer learn a sensible policy from them.
We switched to a different edit distribution that avoids these issues. Instead of changing a single cell, we pick an urn or shard at random and swap it with an adjacent cell. This leaves the density of the level constant, while still building walls over multiple steps. We need to wait for the training distribution to produce higher-density levels in the first place, but given the length of training this is not a limiting factor.
We find that ACCEL further improves the performance of the trained agent beyond PLR⊥. It performs optimally in many, but not all, environments.
The ACCEL agent operating in the deployment environments
The following show the average regret falling over the course of training, for the different methods. On the deployment distribution, we see the more sophisticated methods performing notably better than DR. This usually comes at a small cost of performance on the training distribution.
Since the two distributions have different shard and urn counts, absolute regret numbers are not comparable between them. Note that we show the graphs as a function of gradient updates, rather than training steps. For all methods other than DR, each gradient update require two training steps. The generate and edit cycles of the PLR⊥ and ACCEL agents require further sampling on top of that. While these agents converge in about twice as many gradient steps, the gap in compute cost is therefore more than four times as large.
Interlude: Using Claude CodeThroughout the entire project, we heavily used Claude Code (Opus 4.8), writing almost no code by hand. Even the unreleased ARENA material upon which this project is based was translated from jax to PyTorch using Claude. We then used Claude to extract the relevant code for the project and to implement the additional features.
A particularly memorable momenta was the implementation of the optimal regret solver. We told Claude what we needed and it immediately suggested translating the grid world environment into a Markov decision problem that can be solved efficiently. Within minutes, it implemented a GPU friendly version of this that worked flawlessly for the rest of the project. We later asked it to implement a persistent cache for the solver, which lives on the GPU and is occasionally synced to the CPU and written to disk. This too, worked perfectly.
During the entire project, we did not encounter any problems that were obvious shortcomings of Claude. There were occasional bugs, but a human coder likely would have made more mistakes. Checking Claude’s work was still essential, even if we did not read most of the code. Had we done all the implementation ourselves, we would have gotten nowhere near this far in one week.
ConclusionsWe have shown that, after training, PLR⊥ and ACCEL can outperform classical PPO/DR on the pottery shop environment. At small and fixed training budgets, DR often shows better performance. Applying these methods is non-trivial though and requires some tinkering (in addition to the standard RL issues, such as GPU failures and learning instabilities). Extending the methods to new environments might reveal further subtleties that need to be addressed on a case-by-case basis.
More generally, we learned some valuable lessons about using AI coding agents in research. At least for projects of this scale, implementation is no longer a bottleneck and we are probably better off deferring to AI completely. For research directions this is not (yet) the case, and following Claude's ideas usually led to dead ends. The one exception is debugging RL issues. Claude would often propose useful experiments to narrow down hypotheses and develop fixes, and we expect this ability to notably improve in the near future. Overall, being able to use coding agents efficiently, knowing what to defer and what not, is probably one of the most valuable technical skills to have at this time.
LimitationsDue to time constraints, we were not fully rigorous. For example, we could only run our experiments on a single seed, despite finding notable run-to-run variation. We think the qualitative conclusions that compare DR, PLR⊥ and ACCEL are nevertheless meaningful (which is also because earlier results usually replicated ranking of the methods).
We used an oracle to compute the optimal regret. It remains untested how well this approach would work with an approximated optimal regret.
We ran similar experiments on a grid, see the animations below. We find that all three agents are performing much worse overall, which could be partially due to sub-optimal training parameters. Still, we see the same qualitative relation between the three methods.
On random 5×5 levels the DR agent achieves optimal performance
On the 5×5 deployment distribution, the DR agent gets stuck and never moves at all
The PLR⊥ agent puts in more effort, but gets stuck just a bit later
The ACCEL agent is notably better, but still far from optimal
The ACCEL agent is notably better, but still far from optimal
A GPU friendly version of this environment is provided by the unreleased ARENA material, which itself is based on this repo ↩︎
If I were the pottery shop owner, I would not call this optimal. ↩︎
Discuss
Organizations I'd recommend for funding
I was a grant evaluator for Lightcone Commons last round. Funders disbursed about $45mil through the program, and this included funding roughly my first $6.8 million worth of recommendations. There are other projects I would've preferred to fund over what I did fund, but nobody was doing them and so they weren't on offer. Here's a series of projects that I'd fund in the next round, in rough order of priority. The next round is starting now, and is scheduled to end mid-December. I expect rounds to continue at similar speed. I expect similar amounts of funding will route through me in future rounds.
Disclaimer: I am NOT looking to manage anyone in doing any of these projects. I am not looking to help someone get started. Initiative and leadership and management capacity (and people that can just figure out how the heck to get something done, and that have the connections to do it) are the critical bottlenecks here. I can throw money at promising pitches, but not really help them all that much otherwise.
1. Leaders for a broad public movement against superintelligent machines. In the wake of Coxon Day, a number of random people around the country have reached out to me asking if there's any way to coordinate and organize a grassroots movement. I wish there was a place to point them, that would help support them in organizing people locally and ultimately working towards some big march. My belief is that it's important for there to exist such a movement that avoids the "omnicause attractor" (of being attended only by people who also pick some particular side about environmental issues or middle eastern diplomacy issues or whatever). Further, I think it's important for it to avoid taking a stance on other AI issues: There's a lot of people who hate AI's effect on artists, and people who hate AI's effects on the education system, and people who hate AI's effects on the environment; and there's also people who like current AI as a technology but still don't want to go extinct from it; and we should all be able to team up when it comes to averting extinction. There should exist an org that can ally with the folks who hate datacenters on account of their water usage, when it comes to lobbying for an end to dangerous AI training runs, but that does not require this belief among its members. A group that focuses narrowly on averting extinction, and that serves as a warm and friendly ally to all those who will oppose that particular segment of AI, as part of a broader coalition that not all members will agree with. As far as I can tell, nobody's trying to build this. If there was a good and compelling application for such a thing from someone who could convince me they had a chance of pulling it off, I'd attempt to route them $2mil in the next round (and $6mil/yr).
2. An org dedicated to informing elites. In the wake of the Coxon affair, lots more people are wondering what’s up with the “AI might kill us all” stuff. This probably includes a bunch of people who have disproportionate cultural sway, such as artists, bankers (who probably have a lot of sway over the Trump administration, which cares a lot about the stock market), union leaders (who probably have a lot of sway over what stances politicians are willing to take), etc. I’d love to see an effort that just networks to those folks and offers access to knowledge on this issue. Perhaps by like, literally arranging dinners with folks like me or folks like various whistleblowers. I think it’d be good if they could also offer access to the sorts of folks who say that everything’s going to be fine! Mostly I want powerful folks to be able to hear all the arguments. I think my arguments just straight-up win, so it doesn’t need to be a group focused on propagating one particular message. It needs to be a group that has the powerful connections, and that is able to offer personalized access to people with the knowledge and the arguments. If there was a good and compelling application for such a thing from someone who could convince me they had a chance of pulling it off, I'd attempt to route them $1mil in the next round (and could be convinced to fund a lot more but I’m not convinced this is the sort of thing that scales).
3. An org dedicated to coordinating AI company employees. I think that AI company employees have a lot more power than they realize, to sway the behavior of their host company. The companies didn’t come out and talk about “pacing” until after the employees made a lot of noise. (I think this is true both about the ‘pacing’ letter in July, and about the ‘pacing’ statements in September.) Employees would have a lot more power if they organized. Employees will have a lot less power once their jobs are automated away, and less still once they’re slain along with the rest of us. So they should get moving on the whole ‘organization’ business. This is a sort of project that most people probably couldn’t attempt (for lack of connections and trust). But if some team had the connections and trust and wanted to put in a really solid effort for helping minimize the effort and coordination costs it’d take for AI company employees to organize, I’d attempt to route them $2mil in the next round (and could be convinced to fund a lot more but I’m not sure this sort of thing scales).
Other projects I’m excited about but that seem more niche include:
- Based comms: AI notkilleveryoneist communicators who are unashamed of all the work people are doing to avoid the end of the world. It’s ridiculous that White House advising billionaires were able to discredit Jacob Coxon by being like “after he had decided to quit, he asked some folks he know in the AI safety space whether they had any extra advice”. Screw acting ashamed and chided. We should’ve had a real doom(stop)er group that was supporting Coxon even harder even earlier, and we should be proud of any support we give to people who are trying to avert doom. Where were the unashamed memes and messages? Where’s the high-energy poasters who are having a good time trying to avert the extinction of humanity? Can I exchange $ for you having more time to shitpost?
- AI company watchdog: The AI company communications are weasely softpedaling, and the public is right to view them as slimy and untrustworthy. Maybe if someone built up an account of calling out their bullshit, we could shame them into being a little less craven. It’s hard to do right, I think, but would be pretty cool if it was done.
- Outreach to conservative or religious communities: I think religious communities and conservative communities around the U.S. are pissed about a lot of the AI stuff (as, e.g., they can tell that AI would wreck their way of life and they didn’t sign up for that). I'm not embedded in those communities, but we have common cause here, and we should be able to be friendly allies who join together as part of a coalition. It feels like there should be ways to reach out more hands in friendship (and not in creepy “here’s an issue that I don’t care about but that I think you care about and so I’ll pretend to care about; if I use that as a gateway then will you care about my issue?” sort of way). If anyone has a bunch of good inroads here in the non-pushy non-annoying ways, I’d love to fund them! (I do already have some friends and allies working on this that I am trying to support where I can, but I have appetite for a lot more.)
- Technical people figuring out hardware-enabled monitoring and verification for AI chips, that would make it easier to verify and enforce a ban on large training runs very quickly after the world decides to do that.
A big part of the issue with these is gonna be convincing me that you’ve got the experience and connections and competence to maybe pull the project off, rather than just raising your hand and saying “let me try,” alas. But also there’s a decent chance that, in the next round, I’m gonna fund the closest versions I can even if they’re far from optimal. (Though I might not, if the options on offer seem like they’re more likely to poison the wells or render it harder for the next folks. I don’t quite know how to make these tradeoffs when time is possibly-but-not-clearly short.)
Discuss
Against strawmanning unilateral pauses
Summary: Scott Alexander’s Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott’s post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on.
The good version of unilateral pausing is goodHere is the good version of unilateral pausing:
- Capability pause: Some frontier lab (let’s call them OpenAI or Anthropic) says they have stopped capability research, either just all frontier training runs or both all frontier training runs and all low-scale capability-focused research.
- Internal definition: The company gets to pause on its own terms, which is much easier than a generally coordinated pause since they can define the pause in terms of the existing structure of work and teams within the specific company, reducing disruption to other processes. They have said they would like to work with other companies and countries to adapt and strengthen this definition over time, in particular to make it gradually more compatible with actions by other actors. Importantly, we accept that it may be easiest for a company to pause according to its own, company-specific definition, and handle any asymmetry in their agreements via multiple rounds of negotiation with other actors.
- Safety + security + product: In place of capability research, their efforts to alignment and other safety research, information security, and product. The product part is important! They will still be a trillion dollar company shipping extremely useful tools, and this also provides an escape valve for people within the company who care about products and don’t want to do safety research.
- Restart if necessary: The pause is temporary, and will restart after some period of time if progress is not made to broaden the pause. Potentially one wants separate periods based on different kinds of progress. Other US labs are likely to pause sooner than labs in China, since international diplomacy takes time.
There are multiple places in the post where Scott makes it clear he wants the reader to be thinking of much worse plans than this. Emphasis added:
I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%.
Again, the correct version of a unilateral pause is a company pivoting to safety + security + product, not “disbanding their entire company immediately”.
Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.
This is again arguing against a strawman: OpenAI would not be “out of the industry” if they decided to unilaterally pause. Sam Altman loves to build products! He would very much still be in the industry.
It is possible that Scott will correctly point out that some people do advocate for the extreme version here, but I don’t think he should get credit for this. The title of the post is “Against Stop Unilaterally and Raise Awareness”. I think it is perfectly reasonable for someone to say that SURA-the-phrase applies to my better version. His post is very strongly being interpreted as arguing against unilateral stopping in general. It is being interpreted that way because of his title and framing.
Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticismScott appears to agree that the good version of unilateral pausing is good, but attempts to claim that only one group is advocating for this good version, the AI Futures Project.
(A virtuous exception)
I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
This makes sense, and I exempt it from my criticism.
This should not be a parenthetical. If the goal is clarification for people who haven’t thought about the details much, and Scott believes as I do that there is a good version of unilateral pausing, it should have been in the opening section. It is not. The reader should not be expected to review every section before realizing that the title of the post is wrong.
Oversimplified game theory is bad, and you should feel badThe core argument behind unilateral pausing is that the game theory is complicated, and it is better to go via deontology and other non-consequentialist analyses if you can’t model the game theory well. Scott reasonably points out that one shouldn’t just say this: he wants details. Unfortunately, in the absence of others providing details, he provides his own game theory and makes basic mistakes, providing evidence for the original why-deontology point.
In particular (emphasis added):
By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they'll also stop or slow by that definition.
This is strictly easier than getting them to stop unilaterally. And, unlike stopping unilaterally, it fails gracefully in a situation where some companies agree to it but not others.
Agreeing to pause conditional on others pausing is not strictly easier than unilaterally stopping. There are three reasons for this: (1) “some definition” hides dragons, (2) promises for future actions can be harder than actions, even if the immediate actions are stronger, (3) the boundary between frontier lab and not is ambiguous.
“Some definition” hides dragonsThere are several versions Scott could have meant by “some definition”:
- General version: Company A agrees that if all other companies B agree to stop, by any definition D, company A will also stop using definition D.
- Internal version: Company A comes up with their own specific definition D, and states that if other companies stop according to A’s definition D they will D-stop too.
- External version: Advocates for a unilateral pause take their own external definition D to company A, and get A to agree to D-stop if everyone else D-stops.
It should be clear that getting a company to agree to the general version is out of the question: it is too wide a promise. Set that aside.
The internal definition version is useless without additional rounds of negotiation. If all companies come up with their own definitions and agree to act on them if others act on their own internally-generated definitions, the immediate outcome is no action. Additional rounds of negotiation then clearly falsify “strictly easier”.
The external definition version is also not strictly easier. It is in practice much harder for a company to agree on an external definition than to take action based on their own internal definition. This is a combination of not-invented-here syndrome and different definitions affecting different companies in different ways due to different team structures and research programs.
I am not guessing here. I have spoken directly to multiple people at all of Anthropic, OpenAI, and DeepMind about unilateral pauses, and they are all sketched out by this question of the precise definition of a pause. One of those people was the CEO of one of those labs. Hiding this complexity behind “some definition” is oversimplifying the game theory, and that results in bad analysis and bad conclusions.
Now of course, unilateral pausing also means definition mismatch and multiple rounds of negotiation. It might be the case that one company pauses according to definition D1, and another company refuses to follow suit unless they strengthen to definition D2. But accepting that we live in a multiround negotiation world is one of the core arguments behind unilateral pauses: the goal is to start a process that can then ratchet up through negotiation.
Promises can be harder than actionsIt is not the case that promising to do X in the future based on condition Y is strictly easier than doing X now. This is because
- Incomplete information: It may be difficult to know if condition Y is true based on incomplete evidence; in this context, that means company A not entirely trusting company B’s statements that they’ve Y-stopped.
- Changing circumstances: Promising to do something in the future means not revising one’s views if the situation changes. This is uncomfortable! In practice, that discomfort translates into failure to make promises to begin with.
The above are some abstract first principles. But they conform to how AI company executives speak about the issue, and how I’ve seen negotiations break down in the past: Executives are very uncomfortable making conditional promises.
As one example, DeepMind’s attempt to become a company limited by guarantee (CLG) failed in part for similar “reality fails to conform to simplistic game theory” reasons. DeepMind leadership was attempting to guard against “AI capabilities are crazy” worlds where risks are high, and the Google Board was attempting to guard against “AI fizzles and DeepMind wastes a bunch of money” worlds. Simplistically, there was an arbitrage to be made. In practice, negotiations fell through. One should be very careful when assuming that simple theorems of simple game theory apply.
How many frontier labs are there?Another reason it is not strictly easier to do a Scott-version conditional pause is that it is not clear how many frontier labs there are. In the U.S., the number is somewhere between 2 and 6. There are clearly worlds in which getting 2 companies to do a unilateral and then bilateral action is easier than to get 6 parties to do even a much weaker action, and I claim we are in one of those worlds.
Definitional fairness is less important than survivalAs mentioned, when I talk to people at AI companies, they are very concerned that if they stop, other people will stop slightly less, and it will be unfair. We can contrast the following two risks from AI:
- Company A stops a bit more than company B, and it is unfair to company A.
- One of company A and B trains a misaligned superintelligence which takes control of the world and eventually kills everyone.
The second risk is more important!
Now, one can argue that the first risk is important too, and indeed significant work should go into minimizing the unfairness, since that unfairness does impact pause effectiveness. Too much unfairness means we don’t slow down AI capability progress enough and don’t buy enough time. But I think that fairness is to be achieved over time, after multiple rounds of action, and should not be a precondition for taking the first step.
It may be much easier for an AI company to act on their own definition than by one imposed on them externally. So shifting the conversation to the relative unimportance of the definition used for the first action may be critical to actually causing that action to happen. This applies not just to unilateral or bilateral pauses but also to the grand international coordination that Scott prefers. In that context, it is notable that Scott’s only engagement with definitions is using the phrase “some definition”, handwaving a bit, and adding a footnote about forecaster wording.
Once we will fill in the subtleties around definitions and the details of the game theory, the correct action is to stop.
Discuss
The Power Broker Book Review
When Robert Moses, the subject of Robert Caro's The Power Broker, fought for his vision of parks, bridges, and expressways in Long Island and New York City, the single most important tool at his disposal was his power. When his dreams met the resistance of his opposition, the outcome of each conflict reflected the balance of power on each side. When Moses prevailed, it was because of power. Though it is also a biography of Moses and a study of 20th century urban planning and civil engineering, this book shines as a treatise on power: how it is acquired, used, and lost. Moses has long faded into the annals of history; public policy has evolved during the intervening half century since the publication year of 1974; but power, through its invisible, grasping tendrils, operates in today's world through the same mechanisms as in the Moses era. Weaving his research throughout his narrative, Caro has crafted a cynical and compelling journey through the halls of power, inviting the reader to reevaluate their world through a new and insightful perspective — to see through the stark lens of power.
One particularly depressing theme of the book is that power always prevails. Truth, logic, and consequences be damned, the outcome is decided by what the powerful want and how much power they control. Power alone can explain why Moses' Northern State Parkway, unlike its Southern sibling, meandered tortuously across Long Island — to mollify the robber barons whose estates lay in the right of way — why his Henry Hudson Bridge sliced through the primordial heart of Inwood Hill Park — because, among other funding-related reasons, that particular route was eligible for federal subsidies — and why the East Tremont section of his Cross-Bronx Expressway veered to the north, blighting what had been a successful neighborhood, which remains to date one of the most impoverished in NYC — to preserve a transport depot in which key city politicians held ownership interests at the time. The tenants Moses evicted and the drivers trapped within the biblical congestion on his roads and bridges, congestion caused by the phenomenon of traffic generation (commonly known today as induced travel or induced demand) and more importantly by Moses’ motive of maximizing toll revenue, had few options to improve their circumstances because unlike Moses, they had no power. In The Power Broker, it is power that moves the wheels of history.
Another lesson from The Power Broker is that power-seeking compromises moral integrity. Ingenuity and work ethic aside, the personal quality which enabled Moses’ ascent to power was his practicality, his willingness to discard his other moral convictions when they hindered his quest for power, a quality embodied in one of his quotes: "If the end doesn't justify the means, what does?" When Moses brazenly attempted to seize W. Kingsland Macy's Long Island estate to create Heckscher State Park, Macy gave press interviews containing little more than the facts as he saw them because he felt that the truth of the matter should stand on its own, whereas Moses employed cunning lies to spin the narrative in his favor and turn public opinion against Macy. In Caro's words, "Macy was handicapped by principles." His strict adherence to moral principles became a liability which gave Moses a decisive advantage in their power struggle.
Moral corruption aside, Caro revealed the inner workings of power through the corruption of organizations designed to accumulate and exercise power at scale. In his youth as an idealistic reformer, Moses had faceplanted into the political might of Tammany Hall. By the end of his career, he had perfected a superior system of centralized power under which Tammany leaders served him. Moses realized the grand scale of his dreams by fully leveraging the economic and political resources he controlled to incentivize others, from politicians to union leaders to bankers to the Archbishop of New York, to do his bidding on his behalf. Power thus delegated to his counterparts was instrumental to the changes he wrought upon the world.
One commendable quality of Caro's writing is the staggering amount of research he synthesized into The Power Broker. No written document, however obscure, escaped Caro's meticulous gaze. In addition to the already extensive government records and newspaper clippings, Caro unearthed 18th century primary sources from far-flung libraries and unpublished personal diaries of public officials. In addition to the written evidence, Caro interviewed high-ranking Moses officials and Moses himself. From his vast trove of research emerged irrefutable patterns and recurring themes in Moses' relationship with power: his genius, his consequentialism, his hubris.
Yet, The Power Broker is neither a comprehensive nor an accurate biography of Moses. Caro's most controversial assertion, that Moses built low-clearance bridges over his parkways to prevent buses from using them with the intent of discriminating against poor and Black citizens who required buses for transportation, has attracted significant academic scrutiny and remains an unsettled question. On a separate occasion, Caro lauded Moses' playgrounds for replacing cinders with a new asphalt surface designed to prevent skinned knees but later claimed that the asphalt was awful at preventing cuts and scrapes. In a conspicuous omission, Caro mentioned the Robert Moses Power Dams at Niagara and Massena only in passing, feats of biographical importance which deserved more attention in place of harping on how the bankers supported Moses because they held financial stakes in the success of his projects, how the unions supported Moses because his construction created union jobs, and so on. On a matter of style, Caro's nonlinear and at times repetitive narration burdens any reader trying to piece together a chronology of events.
The Power Broker is, however, a masterful novelization of the historical record in which Caro's ingenious framing of major characters as foils to Moses animates the story throughout. Governor Al Smith, the political antihero who taught Moses the way to power but stayed steadfast in his lifelong commitment to the impoverished from whose ranks he rose. Franklin Delano Roosevelt, who shared Moses' background and his personality and many of the same ideals but became his sworn enemy. Fiorello La Guardia, the ferocious Mayor of New York City who was as passionate as Moses about building public works and kindred spirits in how to get things done. ("In this administration, I am the majority.") Robert F. Wagner Jr., a well-intentioned reformist Mayor who weathered a thousand battles with Moses at the height of his power. And Nelson Rockefeller, the cold-blooded banker who outmaneuvered Moses' fiery crusades and created the perfect conditions to evict Moses from the empire he had ruled.
In addition to his characterization, Caro's writing style makes The Power Broker an enjoyable read. His trademark one-sentence paragraphs, often used in conjunction with paragraph-length sentences, established a rhythm which conveyed the inexorable inertia of power. Another technique Caro employed frequently was to ask his readers to reconstruct the narrative for themselves based on sparse evidence, highlighting the inscrutable nature of power. In one passage, the reader must infer Moses’ shadowy machinations to solve a puzzling sequence of events concerning the East Tremont section of the Cross-Bronx Expressway which demonstrated how common sense and the public good bowed to the political considerations and corruption of Moses' relentless political machine. A different excerpt, describing a newspaper column by Eleanor Roosevelt, exemplified Caro’s ability to dramatize ordinary events:
A single, small paragraph, on a subject she would not raise again. But revealing nonetheless, as the smallest ripple in a pond’s still water reveals the hidden trout below. For that paragraph was the ripple, the only ripple, that revealed that far below the surface of the public controversy over Robert Moses’ huge bridge, down in the quiet, murky depths, impenetrable to the public gaze, in which real power lurks, private passions were beginning to roil the water—and Robert Moses’ great enemy was beginning to move against him.
I strongly recommend Robert Caro's The Power Broker, an entertaining, devastating, and eye-opening account of the forces which move human endeavor. Despite its Faustian qualities, power-seeking is not inherently evil or unjustified; having the ability to control one's own fate is preferable to being controlled by others. Nonetheless, Robert Moses' life is a cautionary tale on the importance of having good reason for making the necessary sacrifices for power.
Discuss
Nectome 2026 retrospective (and our final pre-sale!)
Earlier this year we ran a discount card sale from our post on LessWrong, and we were delighted by the outpouring of support, critique, and analysis we received. We ended up selling much more than we expected, and after the sale deadline ended people reached out to us saying that they missed the deadline for various reasons. We wanted to offer them (and the rest of the rationalist community) one more chance to get in on the deal before we officially launch in January next year because of all the support and encouragement we have gotten from this community in our last decade of research and development. So we have decided to run one last discount card sale, and this is the last sale we’re ever going to offer to individuals.[1] We want to make sure everyone in the extended rationalist community has a chance to get in on it.
The short version:
- Nectome offers whole body, end-of-life preservation for future revival.
- We believe that we can and should preserve people such that preservation straightforwardly works according to current neuroscience.
- Good quality requires real constraints: preservation must start within 12 minutes after medical-aid-in-dying.
- In March 2026 we announced preservation and our initial pre-sales in Less Dead.
- From March to April 2026 we sold ~fifty pre-sales which together were worth ~$1.3M, mostly to LessWrongers.
- We’ve raised substantial investment funding, including from Lightcone Commons.
- We've doubled our team size and moved into a larger corporate office. We also doubled our lab space.
- We've secured a beautiful location in Oregon where people can be preserved.
- We successfully preserved our first client dog.
- We're planning to open for human preservation mid-January.
- Lots of people missed the last pre-sale, so today we're announcing our final discount card sale ever. You can get a ~10x discounted preservation if you buy before January 15th.
The rest of this post details what we've done so far in 2026 and some of our plans for 2027.
ResearchAt Nectome our guiding principle is that we only offer preservations that work. That's why we have so many constraints with our current protocol - it's the only way we know how to preserve the whole connectome. That said, we keenly feel our limitations. What do you do when someone dies unexpectedly? Is there any way to accommodate hospice deaths, instead of exclusively MAiD deaths? Under what circumstances, if any, can we extend the twelve-minute perfusion window?
That's why we approached Lightcone Commons with a proposal to research more flexible preservations. They judged our project to have enough potential that they invested ~$3M. In 2027 we will use this funding to investigate the limits of preservation and make preservation more flexible without compromising on quality.
Team GrowthIn his excellent post "Nectome: All That I Know", Max Harms was optimistic about Nectome, and hoped for a bigger Nectome team. We've indeed seen a lot of growth over the last year: we hired an in-house general counsel, a second ops person, a chemist, a mechanical engineer, and several contractors including a fractional CFO. We anticipate adding a second preservation team[2] before the end of 2026.
All together, we've just about doubled the size of our team, moved into a bigger office, and expanded our lab! This is a growth rate I'm happy with; we've been able to maintain company culture and find excellent team members, while expanding our capacity as an organization. I expect we will double in size again in 2027, with an emphasis on the sales and marketing team.
Preservation LocationOne of our big projects in 2026 has been finding a preservation location. We wanted somewhere comfortable and private for clients to spend a few days with their families before being preserved; our lab is very effective, but it isn't where anyone wants to say goodbye to their loved ones.
In the interest of maintaining the privacy of the location, we won't be publishing the address, but I'm happy to say that we now have somewhere we'll be proud to bring our clients. It's currently undergoing renovations to make it wheelchair-accessible, since I expect that many of our clients will have limited mobility. Construction should be finished in time for our official opening in January.
Equipment UpgradesOur engineering team has been hard at work building version 3.0 of the mobile preservation lab.[3] Our in-house counsel isn't letting me post pictures, due to a high concentration of trade secrets, but early reviews describe it as "like stepping onto the bridge of the starship Enterprise" and "much less janky than the last version". In addition to looking really cool, the new design automates a lot of work that was previously manually controlled by the preservation team (like the cryoprotectant ramp), and introduces a number of backup and failsafe features.
We've also built a compact, highly-automated rat preservation machine, which sits comfortably on a stainless steel cart and can run two rat preservations in parallel. This will be a force multiplier for our science team in terms of the ease of running experiments and we're excited to produce several new papers in 2027.
Pet PreservationWhile we aren't officially open for business until January, we completed our first paid preservation earlier this year. The client was a well-loved family dog with a terminal cancer diagnosis, who traveled to Oregon along with her owners. Our team was able to preserve her, and she's currently being kept safe in cold long-term storage.
This was a big milestone for us, and we're grateful for the opportunity to train our team on some of the real-world messiness that comes with real preservations: outside medical (well, veterinary) professionals, a grieving family, an infirm client, and dignified long-term storage. I expect we'll be seeing a lot more of all these over the coming year.
Sale detailsAll-in-all it's been a pivotal year for Nectome and I think we're set up for success in 2027. For those who are interested, here's the details of our preservation discount card:
- You pay us $29k, once, and get the card (and a beautiful welcome packet, including a cool metal wallet card).
- The card gives you a discount to our current preservation price depending on how long you've had the card. After ten years, the preservation is free.
- Whenever you want, you can redeem the card for a discounted (or free, if it’s been at least ten years) preservation on anyone eligible.
- To be eligible, they must be terminally ill, capable of traveling to a preservation location (currently only in Portland, Oregon), and able to consent to MAiD at the time of the procedure.
- The person to be preserved signs paperwork with us detailing all their preferences, including revival restrictions, chain-of-custody, etc.
- You buy a preservation, you get a preservation: if for some reason the person you designate can't be preserved, you can designate someone else.[4]
If a prudent person purchases a preservation discount card today and then uses their free preservation in ten years, that preservation will cost us more to perform than we made from the card (unless we have managed to substantially lower costs in our process by then.) Therefore, we're limiting our sale to 100 total discount cards, at a price of $29k each.[5] This is the last time we’ll offer an individual sale discount from our full price (currently $250,000).
Towards a brighter future, togetherOne of my dream jobs is to be the person on the other end of a preservation, who helps people rejoin the world. I want to see people's faces light up with the excitement that comes from new technologies, new stories, new places to go. I want to show people the endings to their favorite webcomics and glowfic threads they never got to finish before they were preserved. I want to plan "welcome home" parties where revivees meet their relatives and great-great grandchildren. I want to help people grieve their losses and live with the survivor's guilt they've suffered from living in a world before it learned how to take care of itself. I think I'll be good at that part, because I'll know that pain personally. I want to see the projects that people start when they know they will be alive for as long as they want.
I don't know if I'll even be qualified for such an important job in the future. But I do hope to be preserved myself, one day, and later wake up in the company of some of the people who I helped preserve along the way. I hope as preservation becomes more popular we can all learn to think longer-term, safe in the knowledge that we have the time to build the future right, starting today, and still be part of it ourselves.
Let's go to the future, together, with love.
Aurelia Song
- ^
We don’t want to rule out, for instance, preservation as an employment benefit for companies, or having it be part of a medical health plan. We’d love to be available through Medicare eventually! But this is the last time we’re going to offer an individually discounted price.
- ^
Currently our preservation team is Borys, Anna L, and myself working together with support from everyone else at Nectome. I anticipate in 2027 a second dedicated team makes sense, so I'm hiring preservationists right now. I want two good people who can become senior team leads as we grow. If you think you want to be a preservationist, email me at a@nectome.com and let's talk. You have a chance to be a leader in a new and rapidly expanding profession that will bring hope to millions!
- ^
I've built around 11 different perfusion machine versions over the years, designed to perfuse mice, rats, rabbits, pigs, sheep, and humans. At one point, I had a version that could fit inside the trunk of a car and be deployed at a funeral home in ~5 hours. The current mobile truck design is the best version we've made so far, giving us substantial flexibility for preserving people, data logging, etc. At some point, I might write a “history of perfusion machines” post if people are interested.
- ^
If you die before designating someone, your estate gets the discount card, like any other property you own.
- ^
You may ask, given this, why are we offering the sale at all? A few reasons:
- Money now is more valuable to us than money later as we build capacity.
- We want to give rationalists who wouldn't be able to afford the full price a chance to get in early.
- We expect some cards may be used earlier than 10 years.
- A discount at this point feels appropriate given that clients are sharing some of our startup risk.
Discuss
Страницы
- « первая
- ‹ предыдущая
- …
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- …
- следующая ›
- последняя »