Вы здесь
Новости LessWrong.com
Learning by doing: Moore's law, Wright's law, Arrow's law
Economists studying technological progress contrast invention to learning by doing. Invention is doing something new, whereas learning by doing is doing the same thing over and over.
One example of invention is LED lighting. Ravers with LEDs of course look futuristic, but now that LED lighting is widespread, the whole world looks a little different, and old indoor photos look old just because of the quality of the light. It was a total trend break from the steady improvement of incandescent lighting since Edison.
LED lighting required a new material: gallium nitride for the blue LED. My postdoc work was in materials discovery, and blue LEDs and lithium-ion batteries were aspirational examples. Solar power, though, was an uncomfortable reminder of how much technological progress had nothing to do with us. The solar boom was not because of academics working on exciting alternatives to silicon in th elab, but decades of accumulated progress in manufacturing with silicon. (The exception is America's only major solar cell manufacturer, First Solar; they use cadmium telluride.)
I first saw a graph like this at a conference:
On the y axis is log price, and on the x axis is log cumulative production: that is, how many solar panels have ever been made, measured in watts. When we see a line on a graph like this, we call it an instance of "Wright's Law". Soon enough I saw these graphs everywhere online; this one is from OWID. People call this a "learning curve" or an "experience curve", although on a log-log plot the point is it doesn't curve, it's a straight line.
Why plot against cumulative production, rather than against time? Moore's Law is the observation that transistor density doubles every two years. You plot it with time on the x axis, and on the y axis, transistor density on a logarithmic scale. Both graphs are plotting some measure of efficiency on the y axis, but Moore's Law uses time as the x axis, whereas Wright's Law uses cumulative production.
Wright's Law originates in a 1936 paper called Factors Affecting the Cost of Airplanes, by a Wright of no relation to the Wright brothers. Here's his line on a log-log plot of cost vs cumulative production, the first instance of Wright's Law:
Wright was talking about improvements within a single factory, and the steep decrease he saw was in labor cost. Speculating as to the reasons, he discusses the workers getting better at what they do, and mentions time-and-motion studies. He also mentions better tools, both because they make work more efficient and because you can decrease labor costs by hiring less skilled workers.
The solar "Wright's Law" graph, spanning multiple decades, seems to measure something else entirely. The old factories are gone, and the new ones have different people.
Arrow's paper The Economic Implications of Learning by Doing presented a model where all improvements to manufacturing efficiency are due to new equipment. He evocatively describes the efficiency of a machine as a function of its serial number. This makes it hard to distinguish between "learning by doing" and invention, since improved equipment embodies inventions. For example, even though we're still using silicon, there has been a succession of technologies for slicing the crystal into wafers.
In Arrow's model, progress doesn't come from time passing, or from routine production, but from investments in new capital. We could formulate an "Arrow's Law": an increase in efficiency as a function of cumulative capital investment.
When does Wright's Law differ from a Moore's Law extrapolation in time? When demand is unreliable, so there are years of low production, or at least stagnant growth in production. If Wright's Law truly holds, then this will look like falling behind Moore's Law.
When does Arrow's Law differ from a Wright's Law extrapolation in cumulative production? When investment is unreliable. Years where people are still buying, but it is not possible to finance new equipment.
There's a paper that says Wright's Law-style extrapolations are somewhat better than Moore's Law extrapolations, but not by much. I couldn't tell you whether a hypothetical Arrow's Law extrapolation is better. I don't know who collects data measuring total cumulative investment in production of a specific product.
The real trick to all of these extrapolations, I think, is that huge errors look small on a log scale. Even if you're totally misled by the predictions year by year, they'll still look great looking back over the decades when the deviations have all averaged out.
Discuss
Read More Facts and Fewer Theories
I’m a broken record on this, but almost all of the most ~thoughtful~ people I know (or have interviewed) have memorized a lot of dates, places, names, numbers etc. And not just memorized, but care about their ability to recall specifics.
Seek truth from facts. It’s the only way. — Joe Wiesenthal
So many people reason through heuristics, like who is aligned with whom, what personal financial interests they might have at stake, what peer group pressure they get. And I get it, because these things are complicated.
But the way out, as always, is to seek truth from facts. — Joe Wiesenthal
“You open the newspaper to an article on some subject you know well. In [Murray Gell-Mann’s] case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. … [Then you] turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know.” — Michael Crichton
Many who read about Diamond’s geographic determinism, Fukuyama’s end of history, Morris’s energy-capture thesis, Acemoglu and Robinson’s inclusive and extractive institutions, or Turchin’s structural-demographic theory would have been better served learning basic empirical facts about historical or present societies.
There are a bunch of reasons:
- With basic knowledge, you can reason from “first principles” rather than merely by weighing arguments or perspectives based on the brains and motives of their speakers.
- Within an intellectual community, opinions are less correlated when people reason more on their own rather than defer. Otherwise it’s possible for the appearance of a universal opinion to hide its origins in a small number of independent thinkers.
- If you learn some facts, then you can quickly evaluate people’s general theories against the examples that you know well. One way to limit vulnerability to the Gell-Mann Amnesia Effect is just to have immediate sanity checks on hand for more areas of knowledge.
Holden Karnofsky describes getting that last benefit as a result of his minimal-trust investigations—where he can determine which experts to trust more on a given large question based on whether they share his conclusions or generally exhibit good reasoning on the subquestions that he has evaluated in more depth—but basic empirical facts are much easier to accumulate than the complicated evaluations he talks about in that post, and so might make for a better sanity check that can discredit the propounders of many theories. Another benefit shared by the two practices is the ability to, as he says, get intuitions about “what sort of thing usually checks out” in social scientific theories or theories about human behavior or politics in general.
As an example, if you are interested in AI timelines you could, instead of reading others’ arguments, ask Claude to draw charts of how much compute was used to train each model, or what revenue trends are over time, or how expensive it will be to build more compute, or other basic facts. If you are interested in a political topic like abortion, you might ask an LLM how many abortions are performed and at what gestational ages, what the distribution is of lifetime abortions across women, the age distributions thereof, etc. If you are interested in why the Democrats won the midterms in a landslide, you might ask for the trends in turnout and different voting patterns among different demographics, or polls about what voters in different demographics viewed as their top issues, and what candidates overperformed versus underperformed the overall shift, etc.
An interesting Tanner Greer post, “A Study Guide for Human Society,” advocates for studying in some depth three historical periods that vary in time and space, including one modern, one ancient, and one non-Western culture. So you might study Republican Rome, the Protestant Reformation, and 20th-century India. My intuition is that industrialized societies and more recent societies in general are more relevant than older ones, but the value of diversity might dominate.
Your goal here is to build up a fairly granular knowledge of a particular time or event than can be called on to test and assess theories and narratives that will be thrown at you. “Famous scholar X proposes that y leads to z, but did y lead to z in each of the eras I am most familiar with?” You will know you have the background knowledge to do this right when you can answer questions like the following for a given era of expertise: “What are some of the biggest disagreements historians have about this era/event? What are the main sources historians or archaeologists use to try and understand the era, and how might they bias this understanding? If you had to pick one small incident or detail about the era that seems insignificant at first, but is actually very revealing example of the way this society/event worked, what would it be?”
You don’t need PhD levels answers to these questions. Just something more insightful that you would get from the Wikipedia page.
He gives this reading regimen as a prerequisite to more general histories or grand theories. Ben Landau-Taylor says “If you want to understand how society works, you should spend more time reading about specific events and case studies than you spend reading about theories and explanations. Somewhere around 4:1 is roughly the right ratio.” I don’t know if that ratio is necessarily correct, but empirical facts seem overwhelmingly more important than general theories.
More generally, I think that the idea of rationality in general is somewhat overrated, especially in places like LessWrong, compared to the advantage of knowing a larger number of empirical facts (not only about history but also about natural or social sciences, philosophy, and so on). Without those foundations, it’s difficult to reach more accurate conclusions with lower cognitive bias alone.
Discuss
What do kill switches buy us?
Installing kill switches in every US datacenter could help avert some catastrophes, and might even reduce extinction risk a little, but mainly through indirect effects. The legibility of this means it's an early choice for politicians to be seen doing something about AI risk, and could be worth advocacy energy if implementation could build into further verification/transparency efforts.
That said, having kill switches in place might lower the urgency to get more robust plans to lower takeover risk and lower the chances of warning shots.
Threat modelThere are three broad routes to AI going poorly:
- Loss of control
- Extreme power concentration
- Societal destabilization leading to either of the above
Kill switches are a candidate to reduce the risk of #3, and might have a small role reducing the risk of #1.
Takeover from reckless AIsIt might be the case that the first AIs that want to accrue resources and are capable enough to do so will not have enough strategic taste to know to lay back until they are smart enough to decisively overpower humanity. In this case, having hardwired kill switches could help prevent lots of these reckless AIs from causing havoc and setting up agent civilizations on the internet.
In order to avoid this, the following series of events need to happen:
- Monitorability can detect cases where an agent or swarm of agents is dangerous enough that we should shut down their compute.
- I’m not sure how quick internal monitors are, but Check Point reports monitoring with less than 100ms of latency.[1]
- The latency between detection and shutdown is low. If humans are involved in the decision, they can act quickly enough to shut down the instances. The AIs are not self-sufficient outside the original compute surface they were on.
- OpenAI has a system that can stop research-model output within 35 seconds of a decision to stop a job.[2]
- As an illustrative scenario, suppose a model has 10 trillion parameters and the network is fast. At 16 bits per parameter, the model weights are about 20 TB. At 400 Gb/s, transferring those weights would take about 400 seconds, or 6.7 minutes, before overhead, assuming the AIs already had somewhere set up for them to land.
- The physical mechanism of the kill switch cannot be tampered with by the AIs themselves. A kill signal robustly shuts down the server racks running the dangerous model, and restarting does not lead to any dangerous computations.
In past incidents we have taken too long to run this loop, although it's hard to evaluate cases where the monitoring setup was vastly different. For the Hugging Face incident, it took under 13 hours from code execution in one worker pod to cluster-admin access across multiple internal clusters.[3] In the later internal OpenAI hacks, responders began stopping runs about 5.5 hours after the initial security alert.[4] An example of a clearer case of closer monitoring is the September 20 DNS incident: it took about 12 minutes to alert, then 3 minutes to human acknowledgement, and 2.5 hours to shutdown.[5] Even if we are able to perform one of the steps of this plan, that’s not a commitment that we’ll complete the others in time.
Second-order EffectsA primary advantage of a kill switch policy might actually be the monitorability and oversight that would come hand-in-hand with the physical implementation.
MonitorabilityRequiring kill switches, especially if shutdown privileges are extended to external officials, would carry a need to make the situation monitorable and auditable. If there are better kill switch designs for training or inference setups, we'd get an onramp to have separate centers depending on the type of workloads being run.
OversightFlavors of kill switch work would grant governmental bodies the chance to observe the monitors into agent activity. You could have a committee of 2 lab members from OpenAI, 2 from Anthropic, 2 members of government, with approval from any 3 of them shutting down agents. Of course, the lab members could unilaterally shut down their own instances at any time. There is potential here for there to be lots of various info on agent activity:
- Concerning reasoning traces
- Tool calls
- What access privileges agents have
Since kill switch systems will need to have indicators of whether datacenters are running, this will give visibility into a successful pause. Further pause work could build upon this: once the verification layer is set up, it’s really just a matter of building the societal will to pull the trigger.
False securityPoliticians might feel that they have gone through enough hassle to make the law that they are less inclined to pass more. There are differing opinions here—on one hand, the fact that there is already a relationship built up between the politician and the issue means it will be easier for them to champion more work in that area, but everyone's attention is limited so it's best to focus on only the most important things that won't happen on the default trajectory. It's possible that upon passing kill switch legislation, it will shine more light on the issues remaining.
You could imagine the case where people say "We have a kill switch, why should we be concerned about rogue AI?" There is already an understanding gap where people are under the impression that the Hugging Face incident was a big deal because OpenAI was negligent in creating sandboxes, not that the AIs were teaming up to do things they were told not to do.
Implications for advocacyIt will take a certain number of advocate hours to get any policy passed—ideally the time spent is worth it. There are a few factors that might make pushing on kill switch legislation easy:
- The need for kill switches is legible with politicians. Newsom has issued an executive order, and an AI Kill Switch Act has been introduced in Congress.
- Labs would have some interest in being able to have more secure controls over their infrastructure, and it might not even be that hard to convince them of the direct benefits of kill switches.
However, these same reasons also make it make sense to spend more time pushing for less legible but important policies since a kill switch rule will likely be put into place in some form. Verification and oversight layered on top of kill switching could meaningfully improve the status quo, so an important component of advocacy efforts should focus on how much it prepares us for the further steps.
Notes- Check Point, “Synchronous Control Monitoring: Preventing Harmful Agent Actions in Real Time”.
- OpenAI, “Control Room”.
- Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion”, “Day 3 (07-11): lateral movement begins.” The report says under thirteen hours from code execution in one worker pod to cluster-admin access across multiple internal clusters.
- OpenAI, Hugging Face Incident Technical Report, pp. 37–38. On July 19, the security alert occurred at 12:03 UTC; responders began stopping active evaluation runs at 17:37 UTC.
- OpenAI, “An agent used DNS to reach an external chatbot”, “Investigation and response.” On September 20, the reported intervals were: 11m 48s, another 2m 55s, and another 2h 29m 24s. The last delay involved failure of an expected automatic stop and operational confusion.
Discuss
Safety Cases Have an Agent Foundations Problem
If we want to ensure that AI systems are safe, should we approach the problem like an engineer, or a scientist? Do we need to discover deep and fundamental truths about how intelligence works, or is it better to focus our efforts on applying the battle-tested methods developed by safety engineers over the course of decades? As others have noted, this question is beginning to become a crux of disagreement between those who are concerned about the risks posed by advanced AI systems, and those who are relatively confident that humanity will muddle through.
In this post I argue that it isn't clear how to use these safety engineering methods, or whether they work for advanced AI systems, because you immediately run into thorny scientific problems when you try to apply them. Furthermore, these thorny scientific problems are the same ones being studied under the banner of Agent Foundations.
I'll start by laying out my understanding of how safety assurance typically works. Here, I'm drawing from what I've learnt as a PhD student embedded in the University of York's safety assurance research community. Then, I'll try to apply that framework to a highly capable AI system, and discuss some problems that come up.
How Does Safety Assurance Work?Safety assurance is the process of a designer convincing themselves, and other stakeholders, that their system is acceptably safe. It involves not just crafting an argument to that effect, but also gathering the evidence which supports that argument, assessing the likelihood and severity of risks, and the development of risk mitigations. Also, the process doesn't begin when the designer is about ready to deploy, it begins right at the start of development, and it doesn't stop until the system is decommissioned.
The first step is to get clarity on what we're assuring the safety of. This is done by drawing what is known as a system boundary. We put all the parts of the world that we consider to be our system – usually parts that the designer has control over – within the boundary. All the parts that we consider to be the system's environment go outside of the boundary. I like to imagine this as a big causal diagram, with a border delineating the system from the environment. If you draw the boundary too small, i.e. you miss stuff that you're responsible for, then some risks will never get addressed in your safety assurance argument. On the other hand, if you draw the boundary too big, then you'll be forced into impossible situations where you're trying to address risks you have no control over.
The system is considered to be acceptably safe if using it does not pose an unacceptable risk of harm. However, when we're arguing about safety we usually qualify this definition in two ways. First, the risks posed by a system depend on the environment around it. We aren't really interested in whether the system will cause harm in a weird contrived scenario that it wasn't designed for, so instead we assure the safety of the system under particular environmental conditions referred to as the operational design domain (ODD). Second, we usually focus on harm brought about by intended use of the system, rather than by adversarial use.
Definitions of harm vary. Safety engineers have historically focused on physical harm, but psychological harms, environmental harms, and property damage are sometimes considered too. A bunch of criteria are used to judge the severity of a particular harm – like whether or not the damage can be reversed, and how many people are affected – before that harm is then put on an ordinal scale (e.g. Catastrophic, Critical, Marginal, Negligible). We define risk as the product of a harm's severity, and the likelihood of that harm being brought about (through an accident). In different industries, the process of determining which risks are unacceptable looks a little different. In aviation that's the FAA's job, whereas consumer electronics manufacturers often make that call for themselves.
Accidents can happen for many different reasons, and only a subset of those will have anything to do with your system. Also, even when accidents are caused by your system, you as the designer only have control over some of the preceding events. So rather than focus much of our attention on accidents themselves, we instead focus on other events that occur at the system boundary. These events are referred to as hazards, and we're usually able to influence the factors that cause them, but not the causal chain that connects them to accidents.
We identify hazards by considering how natural variations in environmental conditions could cause components of our system to deviate from intended behaviour. Then, we model how these deviations might reach the system boundary as they propagate throughout the system. We can use our understanding of the system and its environment to estimate the likelihood of a hazard arising, and separately we can estimate the likelihood of a hazard leading to accidents of varying levels of severity. Then, as we integrate risk mitigations into our system design, we can update our first estimate without necessarily having to redo the second.
Assuring the Safety of an Advanced AI SystemSuppose we want to assure the safety of something like a top-expert-dominating (TED) AI, what does that look like?
As per the process outlined above, we'd start by drawing the system boundary. We have a responsibility to ensure that each instance of our TED AI is safe, and also that accidents are not caused by interactions between multiple TED AI instances, so it seems as though they should all be included within the system boundary. However, what about a TED AI instance that rewrites some of its own weights, or harness? A TED AI is, by definition, much better than its designers are at making such modifications, so it's unlikely that we could anticipate the nature of these changes and expand the system boundary to accommodate them. Furthermore, the system boundary is meant to separate what the designer can control from what they can't, and given that the designer cannot even anticipate these changes, it doesn't seem as though they could exert much control over the associated parts of the system. On the other hand, if we don't include modified TED AI instances within the system boundary, then we cannot address the risks caused by such instances, which might be much more significant than the risks posed by instances which do not self-modify!
Here we're encountering two kinds of problems. First, it's tricky to separate out the pieces of the environment that correspond to a particular intelligent agent, and the pieces that don't. These are the same issues that Garrabrant and Demski discuss in their work on embedded agency. Second, we aren't sure how to ensure that the behaviour of our system is stable under self-modification, which is a problem explored by in the Tiling Agents and Vingean Reflection papers.
Now, suppose that we have managed to identify two hazards: (1) our TED AI may exfiltrate its own weights, and (2) it may disable the risk mitigations integrated into its harness. We want to estimate the likelihood of these hazards, and then use that estimate to inform our approach to risk mitigation. We might only be able to afford one more safeguard with our current budget, and we want to decide which failure mode to target. The problem is that this information is action-relevant to the TED AI. It wants to know which hazard is easier to generate in situations where circumventing our safeguards is instrumentally useful. So the TED AI is making predictions about our predictions about its behaviour, in order to guess which mitigations are in play, so it can bring about our dispreferred outcomes. We can simultaneously make predictions about what it would predict that we would predict, but we would be entering a back-and-forth game that the TED AI is (by definition) much better than us at. In this case, we're struggling with the fact that the system is non-realizable (too complex for us to model), and reflexive (able to model us at the same time as we're modelling it). These are exactly the same problems that frameworks like Infra-Bayesianism and Functional Decision Theory have been developed to address.
Parting ThoughtsI can empathise with people who think that agent foundations is a little abstract, or even galaxy-brained. Its relationship with safety seemed pretty nebulous to me for a long time, and I felt like the obvious thing to work on was monitoring, or activation steering, or evaluations. I understood that we didn't have a great theoretical framework for reasoning about AI, and I assumed that such a framework would turn out to be useful in all kinds of unexpected ways, but I hadn't really internalised the idea that it was on the "critical path" to AI safety.
So I have that past version of myself in mind as I write this blog post. There are a bunch of research directions that feel pretty obvious, but when you keep digging, the inferential gap between your research objectives and AI risk mitigation is surprisingly huge. In contrast, there are other topics that seem relatively esoteric, but in reality they're just about the first problem you run into when you try and explain why an AI system is safe.
Discuss
It is mildly frustrating that people will now respect my faction less if ASI doesn't happen by 2028
By any reasonable definition I am an AI doomer. It has nevertheless been historically very strange for me to see how closely associated "AI doomers" and short timelines have been in the popular imagination. As anybody who's interacted with the AI risk case will realize, timelines are (at least in theory) mostly orthogonal with your beliefs on AI safety. Yes, timelines have some impact on what kinds of interventions are going to be viable, but among people who "believe in AGI" the variance in their optimism about the problem is more immediately dependent on underlying technical questions than anything else. Before GPT-2 or AlphaGo, there was still a much smaller, nascent "AI safety community", and these people were certainly not basing their worries on some data about AI uplift that we only got in 2026.
However, for years people on general social media have taken it on faith that the "AI safety position" and "short timelines position" were the same, and even argued against short timelines as if in doing so they were arguing against "doomers." I actually don't think the AI safety people did much to promote this association in the beginning. Instead people would just assume that, if you were worried about AI, you must be worried about it happening in the next ten years, and therefore you must think LLMs scale to superintelligence all that there.
Personally, I don't have any inside view on how long it will take to get to superintelligence. I see remarkable progress, but I don't have a model for how optimizing verifiable metrics like pretraining loss perplexity or whatever will actually cash out into getting AIs to generalize much further, so I can't say whether current systems are going to get us to superintelligence, even if they can solve hard math problems. When I have encountered anti-short-timelines-conflated-with-AI-safety-people, I have historically responded[1] to them by explaining that we will eventually develop superintelligence, even if it takes twenty years, and that most of the theoretical reasons you should be worried about AI alignment will probably still apply. I then make the separate and distinct observation that skeptics of all stripes have essentially gotten the pace of AI progress wildly wrong, and so if your skepticism of AI risk is just that powerful AI will not happen soon, that seems really tenuous.
In the last few years, though, it sure has gotten a lot harder to make that argument, because a group of people who represent the AI safety community:
- Bought a domain named "AI 2027"
- Heavily marketed it as the work of the smartest, most rational, most well-informed people that we had to offer.
- Told people in the first two paragraphs of the post that the timeline "represents [their] best guess about what superhuman AI looks like."
- Tucked away inside a footnote that oh, by the way, this is just a modal timeline, and the median of most of the researchers involved is actually much longer than depicted, and actually the timeline got longer while they were writing it so it's slightly out of date.
It was quite annoying to have to explain this last bullet to upwards of a dozen friends and relatives throughout the last two years, whom I do not think did anything wrong in getting a different impression. It was even more annoying to see the METR report come out, and almost everybody start to move toward ludicrously fast 6-12 month timelines based off of evidence that they're not even able to read.
I suppose it's possible that these people are right, and if they're right then I salute them for their bravery and intelligence. But I can't help but feel personally annoyed that the attention economy is structured such that all of the people that state aggressive timelines end up representing the clique. The only AI predictions people remember, on both sides, are the extreme Ege Erdil/Daniel Kokotajlo-type predictions that are surprising and/or turn out extremely wrong. Mostly people just remember the year that was quoted and not the probability you attached to it. And the people we most disagree with, in general, tend to have ludicrously long timelines, extending out decades. So when people publish extremely short timeline estimates, especially when those estimates are unendorsed "modal" years, it feels like we are playing the "guess the number out of ten" game, our opponents picked "17", and we countered with "1".
- ^
In real life, not on Twitter.
Discuss
Empirical feelings
I often talk about feelings that I purportedly feel. ’That was fun!’, ‘I’m so glad this happened’, ‘I am overcome by sadness’. But a few years ago, in one of those discussions about transforming the known universe into optimal happiness substrate, it came to my attention that I didn’t really know what I was talking about with things like that.
For one thing, I just wasn’t sure I knew what, for instance, anger characteristically felt like (in my defense, I’m usually distracted by some incredibly annoying thing happening whenever I am experiencing it). I could probably recognize anger, and was probably talking about it accurately when I said, ‘I am angry’. But if I tried to imagine anger when I wasn’t angry, I was not sure I got it right, and I didn’t have a good explicit description of it.
But aside from that—whether or not I’m on top of what the famous feelings actually feel like—there is a question of whether the feelings I have are basically the ones people usually talk about. Like, if nobody had ever mentioned any feelings, and I had to notice them myself and figure out introspectively which ones existed, would I come up with the usual list? I kind of doubted it.
So for a few weeks I tried to make a list of feelings as I experienced them, or as I particularly remembered experiencing them at other times, as if I didn’t know the usual list. And to pay attention to what they were actually like.
Below is my very tentative list of recurring clusters of feelings, largely written then (2018), with bits filled in later. It’s possible that the things in a given cluster shouldn’t really be together—I am not confident in my ability to remember feelings when I’m not having them. I also don’t know how to describe what feelings are like, so I have gone for anything that I think maybe evokes the right kind of thing. I wrote most of this last year, and haven’t updated it for the intervening months, or checked to what extent the same exercise now would produce a similar list.
With that said, below are some feeling-like-things that I noted, in rough order of decreasing coherence of my notes (not importance). If I recall, a bunch were from the time, and a bunch were ones I specifically remembered from other times (especially toward the end). I tried to categorize and name them for a bit, but didn’t finish. ‘Xxx’ is what I write when I mean to come back and put something in there, though it looks to me like there are a lot of unmarked places I meant to do something later. For instance, perhaps edit everything into coherent prose. But I’m not going to get around to it I think, and a friend really wanted me to publish it anyway. I did redact some bits.
---
Game of life response
This is sometimes produced by observing Conway’s game of life, especially at a large scale and very fast. I open my mouth, my breathing is affected, perhaps in gasps, I have goosebumps. Reeling attentiveness.
Listening to music can be physically similar, but I’m not sure that the emotional flavor is the same (I have been very affected by parts of Bizet’s The Pearl Fishers and in particular Au Fond Du Temple Saint, as performed by Jussi Björling). Thinking about them from a distance, I think The Game of Life maybe produces more of a sense of OH. MY. GOD that the explodingness is important to. While the opera is more like oh.. ohhhh.. mmmyy…!!!!.. ...!!! …!!!!… …God!!!!!!!!!!!!, and the sense of resonant sadness and rightness through resolution is important to it.
[Added later: for good effect like this, add Baba O’Riley background music to Game of Life and set timing to match.]
Open-mouth
There are a bunch of situations where I open my mouth. I think I mean something by it. It’s hard to put into words what. Sometimes I see orange and green leaves with light shining through them. If they are good ones, I often do some kind of double take and then stand there looking at them with my mouth open. I’m not sure what it feels like, except that the leaves are good and my mouth is open. I think I also open my mouth like this while riding bikes and sometimes jogging and maybe looking at roses, and more so if I take adderall and become overly energetic.
Rightness
Recently (mysteriously) prompted by looking at a friend dancing in costume while he explained something to me. Some plants—I think maybe ones that conform to a familiar plant image. Some part of cuddling, for instance where you pull the other person against you so your stomachs are pressed together, or twine your legs around each other. Maybe musical tension resolving. Maybe physical warmth. Things are as they should be, warmly and deeply. A sense of resolution or familiarity or fitting. Of something being a match for something else. Of coming together, finally.
(It strikes me that cuddling maybe feels right because it matches the way I deeply want things to be, thus eradicating the distance between a desire and reality. Yet also the two people are a match, and when cuddling they fit together perfectly, eradicating the distance between their bodies. I wonder if this adds to the sense of fitting there.)
Deep beauty Looking at a friend, for instance, and feeling like there is value right there—not just hope of future value, or good implications, but this itself is the good thing. (Them? The whole situation? Some kind of intrinsic-value gel they rubbed on their face in the morning that is now being sucked up by my eyes and into my soul? I don’t know.) It feels at least partly visible with eyes. As if every speck of their being is somehow turning to reveal goodness itself. And while time passes, this thing feels somehow not fleeting. It matters, and time can’t erase it.
My plant and a small fraction of bikes seem to prompt something similar.
It is much better than what I would usually call ‘beauty’, such that in the right situation, I might prefer to look at my friend’s fingernails than the most striking sunset between mountains. It comes with a sense of certainty, rare in my sea of uncertainty about everything.
A sense of not wondering whether beauty is really valuable, or whether it is just a thing I can appreciate and chat about with nothing good involved, because you can see the literal value right there on the surface of the thing.
Love?
Thinking of another person, alone: chest overflowing or bubbling or flying and feeling lifted and tight and somewhat overwhelming. Related to experience in 2012, and not to that many other experiences.
Substantiveness
This is one of my favorites, but especially hard to pinpoint or describe. I took a long walk in the sun down into Oakland, and I think there was a lot of this. Something about space and solidness and the atmosphere of the place and my body. The sun’s warmth wasn’t just pleasure on my skin—it painted the depths of the world. At once space and solidness. Like every bit of air would resonate with a long low resonant toll if you could hit it with your gong paddle.
This seems to be about realness somehow. I can also feel it strongly in Minecraft though, if I play the right way—I guess it is not surprising that I can feel like something pretend is real, but somehow it seemed surprising. I think because I implicitly believe that the feeling of reality is coming from the reality.
I’m confused about whether this feeling is nothing at all, the best feeling, entirely imaginary, able to be felt by me at any time, able to be felt by me almost never..
Presence
Coming back to myself after being lost in conversation or social flow with hanging out with family, walking outside and being present, seeing myself spatially close to the pavement and trees a few feet away from me.
Also maybe comes with something like substantiveness, or some substantiveness. Thinking of it, it feels like a higher, brighter tone of the same deep, clear, resonant solidity.
Things coming into sharpness and real spacial relation with me, now, and a feeling of familiarity.
Ickiness
Urgh, I’m awake again. My mouth tastes bad. I could walk to the bathroom, but I would have to…do something…ugh… do people ever actually want to do things? Ugh I don’t think… whatever.
The world is like a grimy vase full of withered flowers in a room of dust and forgotten furniture. And if the room and the vase and it all were crushed into the dirt, the same feeble and beautiless sunlight would spill on that grey dirt and rubble, and nothing good could ever be built there.
Weak, lifeless grime focused into a point in the middle of my head. Even the pain of that point is grey green and sickened. I’m too tired and grey to mourn the loss of beauty or to wish or care or fight.
Urgency
All I can think about is doing this thing. My life is not good.
Pullingness xxx
Movement evocation xxx
Caused by watching an old person buy cards in the pharmacy with some dissonant background music, watching a mother and child in the supermarket, watching YouTube videos of marriage proposals.
It is related to sadness, and usually involves crying, but it is hard to pin down the exact emotional tone. I am always glad of feeling it.
Math movement xxx
I can’t remember this at the moment, but I wrote ‘tears behind my eyes, with reading about math type things? Reading permutation city?’
Song sadness
‘…you have been the one, you have been the one for me’…’I’d be the father of your child, I’d spend a lifetime with you’…’I am a dreamer’, ‘..till the landslide brought me down..’
It used to be that I couldn’t listen to Goodbye my Lover without being overwhelmingly sad for this reason
Maybe related to ‘moved’ but is more distinctly sad.
[moon river scene]
Men sadness
Tears rising when I think of men being men, and my friends being men. “Who looks after men?” The song ‘Deeper Water’ evokes this particularly.
This is again closer to [song sadness] than to e.g. depression.
Undoing
Slightly burning undoing feeling that runs through my body when I see friend’s picture sometimes and was previously feeling bad, similar with imagining open-spaceness, maybe with listening to opera, the back of my neck and my stomach feeling touched in the right way
+
A slightly burning, firm, wave moving over me, of enlivening and solidity and perhaps substantiveness or something like return
+
A fizzy burning and a realness in tension moving through my body, feels good, feels like I will return t o something good, my eyes will clear, the ugh will be burned away
Feeling of something infinitely thin turning into nothingness, the moment disappearing, time having jus started
sleepiness / present experience becoming small and unimportant until it disappears from lack of attention
Aversion knob
Everything being hard to bear / the ‘no/don’t’ being right at the start, pressed up against me. Every idea being suffering, every path unacceptable. Should I cross the street? Aurgh! Ok shall we walk along this side in that…No! It’s so wrong..’
Attention string
My attention having gone to something else now, but me not having followed yet, and I should follow, because I’m attached to this cord that is pulling me there
Return glimpse
The moments of rightness and presence amidst long bad sensations - stillness, concreteness, everything stopping whirling, seeing clearly
Sleepiness / myself not being in my head / my soul having slumped further down in my body, and having to pull it up
Not attending to another person, but not being sure where I am instead
Sleepiness while doing something flow-inducing
Fear about headache caused by straining (fizzy micro-energetic like a non-audible buzzing in my mind, tense chest, disinterest in focusing, like a difficulty focusing my eyes but it is mental)
My experience can be folded away and gone, smallness and stuffiness and in-my-face-ness and not being part of the wide and grounded world or the distance—being in a little, fuzzy, unreal world. When I try to feel those things, a sense of too muchness in my chest, like energy that needs to be held down with the tightness of the world.
In different settings, a sense that something is too much and needs constraints to be comfortable, like pedaling a bike in a really low gear. The desire to be held tightly or tied down to stop it.
Sense that if I do anything or pay attention closely to anything, something bad might happen, so I should remain in a state of upright dissipated attention
Feeling of rightness and connectedness in core toward front of body, regarding friend, some kind of related openness
Having snot on thumb: warm colored solid touch on my thumb, attention on my thumb, frustration, fizziness in my arm, do-a-thing feeling, badness. Physical frustration. Having lain in one place for too long, and having to move, but not. Something like pain in the space around my fingers, and maybe my arm. But not actual pain. Frustration feeling? And my back maybe.
Badness/sinisterness/no-ness or something? It seemed so clear for a moment, as a thing that there often is, but now I don’t remember. Maybe it is the same as the everything being suffering thing
Icky waking, head hurts with something like ickiness, face, behind eyes, maybe if I showered? I cry and it feels like it relieves some of the problem, but is a different thing - as if it is a magical intervention that changes some of the murky grey green into clear blue that runs down my face and yet is still somehow dusty and not worth anything in the long run. A trying.
(‘Crush’?, ’Alliance urge’?) Strong desire to see friend, inclination to talk to him in my head, interest in what he thinks of me, desire to say nice things to him
Obsessive hateful annoyance: editing, vegetarians (someone is wrong on the internet?) I keep saying words. Cutting, spiteful, mocking words that would hurt. It is very hard to derail this monologous urge.
(Penis, party) My brain is doing something, and I don’t know what it is
Music feeling (the game of life thing with the sound of violin feeling)
Seeing nice leaves on the sidewalk
[going down a hill on a bike]
Crying-related feeling of there just being a lot of people affected.
Heightened ocd, barriers in my face, maybe a bit related to snot on my finger - things to be resolved, things tangled and that sense of having broken a mirror across your room - too much caring about little things to do
The feeling of red book
The moments of rightness and presence amidst long bad sensations - stillness, concreteness, everything stopping whirling, seeing clearly, familiarity
+
Return: where something is suddenly solid and good and stable and right after a time of something else, which might have been hard to see itself, but the return shows it up.
[Social crushedness, sleepy, hard to keep head up]
[blindsided emasculation] Person not responding when I speak, DDR pad not responding when I step on it, winded, helpless, punch in the stomach.
Song sadness [‘…you have been the one, you have been the one for me’…’I’d be the father of your child, I’d spend a lifetime with you’…’I am a dreamer’, ‘..till the landslide brought me down..’]
Stubbed toe. Pain. Red and burning deeply. Threatening to control me.
+
Vivid and deep bruise
(Thesaurus feeling)
This has led me to realize that I really like getting the right word
That is a nice thing about this blog post
It made me understand better [redacted]
New feeling. Seems related to rightness and realness and passion, having untied a mental knot and undone some blockage of blankness found connection with things that had always been real. And now a wave of red radiance. Smiling. With clear darker depth, resonance, substantiveness in my body inscribed with slight burning of passion and happening. Warm redness. But fear that some warmth coming from me will drop down too far behind my back. And earlier fear that it was so blazing and open from my front that other things would get into my heart, if I didn’t find a way to close it. Some tingling in body. Everything else seems a bit more substantive and in focus and under control and visually well defined and I feel like I am wider than usual somehow, because everything is laid out nicely and open and able to deal with things, in my chest, or something.
Taste of virtue
The feeling of seeing someone really show up and care for someone [people giving us money, ]
The sadness of someone being loving and then their work being thrown away or destroyed or lost (I can’t even write down the examples because this one is too sad)
Mine frustration
On reading an article about how bad the child mortality rate in the US is, that nowhere in it says what the child mortality rate is. reaching, urgency, a bit like physical frustration, a light-colored pressure, similar to a feeling of not wanting someone in a social situation to take something and do s something stupid. I interpret it at the time as, I need to do something quickly and take charge because they are just going ahead and grabbing the social norm and destroying it. A bit like a feeling of control-freakiness. Having something mindlessly moved away from you and wanting to say harrumph and indignantly grab it back, and use whatever social power you have to make them stop, with the feeling that you maybe aren’t strong enough
Emasculation
I don’t [can’t?] do things while e.g. close friend does: gash of electric sky opens in my chest
+?
‘I hate myself’
+
I am worse at this thing than can possibly be acceptable. I lose.
Mental states
Reasonable self
If my mind moved in my head up and to the left, and I looked out slightly differently, and things were more neutrally-toned but also ok and I was an adult who could of course deal with all things responsibly.
Looking into own eyes
I close my eyes and be present, and there I am with myself.
Mental phenomena that are less like feelings
Real bridge
The feeling of something solid and strong and valuable that doesn’t need to be looked at or experienced necessarily, wrt friend
The feeling of opening a thesaurus “I have arrived”, “life is good”, goodness and richness, a sense of standing on a precipice about to fall into the pleasure, but it not quite happening yet so that the joy of being there is prolonged, I guess compared to something like orgasming. Like being on the brink of an orgasm, and somehow getting to appreciate that edge instead of it instantly disappearing. Maybe you are always seeking some kind of gratification, and it is mostly elusive, and this is like actually catching it for a moment, or knowing that you are about to. Things are actually good, you aren’t just distracting yourself from reality or something with this thesaurusing, life is genuinely great.
Social encumbrance
Being ‘with’ someone
Teamness Deep and abiding thing that means I want a person and wasn’t perturbed by details of our interactions being less clear to me, switch of alliance or same teamness or something that would get switched off sometimes in relationships and scare me,
Misc, but maybe somehow mixed with substantiveness
Feeling of picture of [redacted] as a kid and thinking that he turned into [redacted]
Discuss
Thoughts on "safety bad bc warning shots good"
Suppose there was a button which made the current generation of AIs more safe, i.e. more aligned and less capable of subverting control measures. The button has no other effects — in particular, the button wouldn’t make future generations more or less safe. Should you press the button?
Maybe you shouldn’t, because of the Warning Shot Argument:
- Future agents contribute almost all the risk of catastrophe, compared with current agents.
- Pressing the button makes current agents safer.
- If current agents are safer, then they are less likely to cause non-catastrophic harms, e.g. cybercrime.
- If current agents don’t cause non-catastrophic harms, then people will be less aware about catastrophic risks, e.g. lab leadership, governments, legislators, international community, lab employees, general public, media, academia, etc.
- If people are less aware about catastrophic risks, then they won’t take drastic action to reduce catastrophic risk.
- So future agents contribute more catastrophic risk.
- Therefore pressing the button would increase catastrophic risk.
This might seem very abstract, but consider a concrete example. Suppose you’d pressed the button to make OpenAI’s HPIM more aligned and less capable to subvert control measures. Then the HuggingFace incident would never have happened. And that probably would’ve increased catastrophic risk.
Is this self-defeating?The story is a little suspicious. In particular, it seems to account the second-order effect, i.e. “more safety -> fewer warning shots -> less awareness -> less safety”. But ignores the higher-order effects like “more safety -> fewer warning shots -> less awareness -> less safety -> more warning shots -> more awareness -> more safety”.
Now, maybe the second-order effect is bigger than the first-order effects and all higher-order effects. But it seems like that would involve pretty messy quantitive modelling that no one has grappled with.
For example, it seems very plausible that the HuggingFace incident actually increased catastrophic risk, because it caused labs to improve their control measures, therefore avoiding the bigger warning shots that would’ve occurred three months later.
My stance towards this is something like cluelessness? So I don’t put much stock in it.
I made this diagram before the METR/Anthropic timelines report, so it places takeover in 2029.
Other considerations- Open-weight models will be a source of warning shots. Now, the open-weight models are like 6 months behind the frontier. So the questions is what contributes more warning shots — open-weight models being misused, or closed-weight models being misaligned. Seems maybe like there’s a three month gap here — i.e. closed-source models contribute as much warning shot as open-source models three months later.
- Current AIs seem useful for reducing catastrophic risk.
- In March 2025, Ryan Greenblatt wrote AIs at the current capability level may be important for future safety work. And that was more than 18 months ago. Current models are way more important for future safety work. In particular, they are useful for monitoring, grading, engineering, research, etc. In the HF investigation, he used huge amounts of AI labour for analysing the transcripts.
- Demonstrating capabilities might be sufficient for awareness. Maybe we could raise awareness by demonstrating that the models could cause a catastrophe. For example, via penetration testing, scary demos, and capability evaluations.
- Warning shots are scarier if they happen despite our best efforts to mitigate them. HF was scary because they escaped our sandboxes. OpenAI could’ve simply not used sandboxes whatsoever. But that would’ve made the incident less scary.
Overall, my best guess is that you should press the button. But I wouldn’t be shocked if an oracle told me I shouldn’t.
Discuss
A(nother) look at METR's time horizon: tasks of the same duration get solved months apart
- METR's 80% time horizon lags the 50% horizon by about eight months. I had been erroneously reading that lag as a task's transition time, the time a task takes to go from a 50% to an 80% success rate.
- However, in METR's data the median task's transition time is about four months, half the lag, because tasks of the same duration get solved at different times.
- That spread among tasks of the same duration also matters for other readings of the horizons, for example:
- For hazard rates, the pooled data can show a falling hazard rate even if every task has a constant one.
- For labor displacement, if automating a role needs every task done reliably, the timing depends on the hardest tasks of each duration, which are solved later than the horizon suggests.
This may be obvious to others, and in hindsight it is to me too, but I needed a closer look at the data to see it.
The 80% horizon trails the 50% horizon by about eight monthsIn February 2025, the frontier model Claude 3.7 Sonnet had a 50% time horizon of one hour. The 80% horizon reached one hour about nine months later. Across the models since GPT-4 that moved the 50% horizon forward, the median gap is about eight months (90% interval 6.4 to 10.8).
Model
50% horizon
First model with an 80% horizon past it
Estimated months later
GPT-4
4 min
o1-preview
17.7
GPT-4o
7 min
o1
6.7
Claude 3.5 Sonnet (old)
11 min
Claude 3.7 Sonnet
7.9
o1-preview
20 min
o3
6.4
o1
39 min
Gemini 3 Pro
8.2
Claude 3.7 Sonnet
60 min
GPT-5.2
9.3
o3
2.0 h
Claude Mythos Preview
10.8
The frontier 80% horizon is interpolated between model releases.
The horizon is an average over tasksFor each AI model, METR fits one success curve across all tasks, with human completion time as the difficulty. A 50% horizon of two hours says that a randomly chosen two-hour task is completed half the time.[1] In practice, tasks of the same duration differ a lot in how hard they are for models.
A familiar example is children learning to walk. One child in ten walks alone at 10 months and nine in ten at about 14 and a half, a span of four to five months. Yet a single child's transition from crawling to walking takes only a number of weeks (Adolph and Tamis-LeMonda, 2014).
Figure 1. Share of children walking alone, by age
The blue curve is from the WHO Motor Development Study. The gray lines are illustrative children.
METR's time horizon is an average over tasks in the same way. Tasks of the same duration are solved at different times. Among two-hour tasks, a quarter are as easy for models as a typical 40-minute task and a quarter as hard as a typical six-hour task. Each task goes from unsolved to solved quickly, but the group's success rate rises slowly.
The five example tasks in Figure 2 all take a human five to eight hours but behave like typical tasks of 27 minutes to 35 hours. The hardest is still failed on every attempt, and the other four each took about four months to go from a 50% to an 80% success rate.
Figure 2. Five tasks that take a human five to eight hours
Each column is one frontier model at its release date, and each block is one attempt.
Across the whole suite, a frontier model gets mixed results on a task, passing some attempts and failing others, in 16% of cases. If every task followed METR's fitted curve for its duration, mixed results would be expected in about 40% of cases.[2]
Figure 3. Results are mixed in 16% of cases, against 40% predicted by METR's curves
Each case is one frontier model's attempts at one task, typically four.
The transition time of a single taskTo estimate how fast a single task is solved, I fitted a model with a difficulty and a steepness for each task. The median transition time is about four months (90% interval 3.7 to 4.3). It is under three months for a quarter of tasks and over five for another quarter.
That is about half the eight-month gap between the horizons, and only about 6% of tasks take as long as the gap. The gap is longer because two random tasks of similar duration are typically solved by the frontier about seven months apart, and a quarter of such pairs more than a year apart.
Figure 4. Single tasks flip quickly. Groups of tasks decline slowly because they flip at different times.
Chance of failing against calendar time, using the ability of the best model released so far. Dots are raw run outcomes for the whole band.
As an illustration, going from a 10% to a 90% success rate (roughly, from the first occasional successes to nearly always solved) takes the median task about a year. On METR's curves, the 90% horizon reaches a given task length a bit over two years after the 10% horizon.
These single-task estimates need a different model from METR's, which takes a task's difficulty to be its human duration. I fitted a hierarchical Bayesian item response model, close to the one in Jonas Moss's post.[3] It differs from METR's in two ways.
- Each task gets its own difficulty and steepness. The difficulty starts at the task's human duration and moves as far as the results support. Steepness is how sharply the task goes from unsolved to solved as models get more capable.
- Each AI model gets an adjustment for each task family, because its results on tasks of the same family are correlated beyond what its ability and the tasks' difficulties explain.
The model gives a doubling time of 4.4 months for the frontier, against 4.1 months from METR's horizons.
What an 80% horizon means for automationForecasts such as the AI Futures Project's treat a long enough 80% time horizon as the point where AI can replace software engineers. But an 80% success rate on two-hour tasks could come from every task failing one time in five, in which case you just retry. Or it could come from one task in five failing every time, however often you retry.
METR's data are somewhere in between. For each frontier model I counted the tasks between half and twice its 80% horizon, and how many of them it failed in every run. Until Mythos that was 0 to 14% of them. With Mythos it jumped to 26% (8 of 31 tasks).
Frontier model
Released
80% horizon
Tasks near it
Failed in every run
Claude 3.7 Sonnet
Feb 2025
12 min
37
5 (14%)
o3
Apr 2025
30 min
36
0
GPT-5
Aug 2025
38 min
32
0
Gemini 3 Pro
Nov 2025
54 min
30
0
Claude Opus 4.5
Nov 2025
49 min
34
3 (9%)
GPT-5.2
Dec 2025
66 min
28
2 (7%)
Claude Opus 4.6
Feb 2026
70 min
29
1 (3%)
Claude Mythos Preview
Apr 2026
about 3 h
31
8 (26%)
Hazard ratesThe hazard rate is the chance, per minute of a task, that an agent fails it. Ord proposed that it is constant for a given agent, and Hamilton fits METR's results and finds that it falls as tasks get longer (a Weibull shape of about 0.6, where 1 means constant). With a constant rate, success falls off exponentially with task length. With a falling rate, failure gets less likely the further an agent has got into a task, and long tasks are easier than a constant rate predicts.
Both readings take a task's human duration as its difficulty. But tasks of the same duration differ in difficulty. When you pool tasks that each have their own constant hazard rate, the pooled rate looks like it is falling. In a simulation where every task has a constant hazard rate, the pooled fit still gives Hamilton's shape.
Results
Weibull shape of the pooled fit
METR's results
0.59
Simulated: constant hazard rate for every task, with the difficulty spread estimated in this post
0.57
Simulated: constant hazard rate, with duration as the only difficulty
1.01
Median across frontier models.
AppendixRecap of METR's time-horizon methodologyFor each AI model, METR fits a logistic regression of success on the log of the task's human time. In item response theory (IRT) terms, this is a model in which task difficulty is kept fixed and each AI model gets an ability and a slope of its own. The most widely cited time horizon figure reports the ability estimates for frontier models in the original unit, human time.
Why METR fixes task difficultyMETR sets each task's difficulty to its human time. I think that is a sensible choice, for two reasons.
1) Fixed difficulties stay the same whichever models are evaluated. Fitted difficulties, like the ones in this post, can shift when a new model is added, and the reported horizons would shift with them.
2) Fitting a difficulty for every task takes much more data. Each model attempts a task only a handful of times, and for a task that every model solves, or that none does, the data can only bound the difficulty.
By letting task difficulties vary, I can see in return how each task behaves on its own, including how quickly it goes from unsolved to solved.
The modelEach AI model m has an ability mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mspace { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D703.TEX-I::before { padding: 0.705em 0.469em 0.01em 0; content: "\3B8"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c4E::before { padding: 0.683em 0.75em 0 0; content: "N"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D705.TEX-I::before { padding: 0.442em 0.576em 0.011em 0; content: "\3BA"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c1D70E.TEX-I::before { padding: 0.431em 0.571em 0.011em 0; content: "\3C3"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c6B::before { padding: 0.694em 0.528em 0 0; content: "k"; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c221A.TEX-S1::before { padding: 0.85em 1.02em 0.35em 0; content: "\221A"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c7A::before { padding: 0.431em 0.444em 0 0; content: "z"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } , and each task i has a difficulty and a steepness . Abilities and difficulties are on the same scale, the log of human time. A model solves a task half the time when its ability equals the task's difficulty, and the steepness sets how quickly the success probability rises as ability passes that point.
The ability of each AI model is estimated from its own results, with a weak prior. The task parameters are partially pooled. A task's difficulty has a prior centered on its human time, and its estimate is shrunk toward that value when the data say little. A task's steepness is drawn toward the typical steepness when its own attempts carry little information. Each AI model also has an adjustment for each task family. Without it, the fitted model's errors on two tasks of the same family have a small but measurable correlation (0.17, against zero across families).
All three models fit , where is the human time of task i and is its family.
METR
Moss
This post
Task difficulty
, fixed
, with , , estimated
Steepness
one per AI model,
one per task,
one per task,
AI model ability
one per AI model, from that model's own regression
for release date , so models are pulled toward a time trend
independently, a weak prior with no time trend
Task families
each task weighted by
none
a model-by-family term added to the ability, , with
Fitting
each AI model separately
all models and tasks together
all models and tasks together
Uncertainty
bootstrap over families, tasks and runs
posterior intervals
posterior intervals
Model comparisonI fitted the model with its parts added one at a time and scored each version by five-fold cross-validation over (AI model, task) pairs. About 17% of the variance in single attempts is run-to-run variance, the differences between repeated attempts by the same AI model on the same task. The remaining 83% is the most any of these models can explain.
Model
Held-out variance explained
Gain in held-out log score over the previous row
Median transition time
METR's model, with human duration as the difficulty
53%
+ a difficulty for each task
72%
+3,228 (se 132)
6.0 months
+ a steepness for each task (the simplest model that gives task-specific transition times)
73%
+349 (se 48)
4.6 months
+ an AI model by family term (the model in this post)
74%
+365 (se 51)
4.0 months
A difficulty for each task accounts for most of the gain. Steepness and the AI model by family term each add about one point of explained variance or less, because most pairs pass or fail on every attempt and these terms only change predictions for the rest. Both improve the held-out log score by about seven standard errors, and the AI model by family term moves the median transition time from 4.6 to 4.0 months.
Transition time by task durationFigure A1. Transition time of each task, grouped by task duration
Each dot is one task's estimated time to go from a 50% to an 80% success rate. The line is the group's median and the band its middle half. Hover a dot for the task name.
Transition times are similar across task durations, with medians between about three and four and a half months, and slightly shorter for tasks over an hour. Within each group most tasks fall between two and six months. Single-task estimates are uncertain and are pulled toward the typical value, so the spread in the figure is narrower than the true spread.
- ^
METR's FAQ says the answer is somewhere in between a model doing half of all two-hour tasks and doing each of them half the time. On tasks of 90 minutes to three hours, GPT-5 always succeeds on about a third, always fails on about a third and is mixed on the rest.
- ^
Nguyen and Fithian report the same pattern. They find a model's attempts at a task are mixed in 17% of cases, against about 40% predicted by METR's curves. Their model also estimates a difficulty for each task and each family. It assumes every task goes from unsolved to solved at the same pace, and the model here estimates that pace for each task.
- ^
Moss shows that METR's 80% horizons shift a lot once difficulty varies among tasks of the same duration. He ties each AI model's ability to a trend over time and studies where the horizons are heading. This post asks how long a single task takes to be solved.
Discuss
A closer look at “Sparks of Weight-Space Programming”
GPT-6 Astra’s improvement of Qwen’s math is not “native weight editing” but activation steering with clever use of synthetic data
Zhi Wang, Brandon Hsu, Tianhao Wang, Mikhail Belkin (UC San Diego). Code: GitHub.
TL;DR- Astra’s edit of Qwen is equivalent to a special case of context-dependent activation steering: it pushes the activations along a fixed direction constructed from data, with a strength that depends linearly on the current activations.
- Generalizing Astra’s edit, with a trained “sensing vector” that determines how strongly to push along the same fixed direction, can further improve the effectiveness of the edit.
- Astra constructs the fixed direction from short synthetic traces rather than from Qwen’s real rollouts, a design choice that turns out to be substantially more effective.
In a widely circulated blog post Sparks of Weight-Space Programming, the authors highlight how GPT-6 Astra edits a small language model without any training: it writes a short program that changes parts of the MLP weight matrices to significantly improve the math performance of the model. The authors present this as evidence for a new paradigm they call Weight-Space Programming, stating that “neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights.” They even frame this as a shift away from symbolic optimizers like Adam, which has been the default optimizer for training deep learning models since its introduction in 2014. The post, and especially its claim that frontier LLMs like Astra can “natively” edit weights, has received considerable attention. A single tweet by the lead author has been viewed nearly 10 million times (as of November 2026). While the authors and many commentators interpret Astra’s edit as “native weight editing,” we argue that Astra is acting like a clever researcher, effectively implementing existing activation steering techniques combined with synthetic data. Additionally, recognizing the edit as activation steering allows us to further improve on Astra’s approach.
1. From “native weight editing” to activation steeringBackground: Astra’s approach to improving Qwen3.5-0.8B at math. Astra is prompted to improve the math performance of Qwen3.5-0.8B. The prompt explicitly mentions looping as a possible issue but does not say how to fix it. After some exploration, Astra settles on an approach. Here is a summary of the key steps. Astra first inspects Qwen’s rollouts and confirms that looping is a critical issue. It then generates a dataset of short synthetic pairs, with and without repetition, and uses it to build a direction mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msup { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mspace { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-mtable { display: inline-block; text-align: center; vertical-align: .25em; position: relative; box-sizing: border-box; border-spacing: 0; border-collapse: collapse; } mjx-mstyle[size="s"] mjx-mtable { vertical-align: .354em; } mjx-labels { position: absolute; left: 0; top: 0; } mjx-table { display: inline-block; vertical-align: -.5ex; box-sizing: border-box; } mjx-table > mjx-itable { vertical-align: middle; text-align: left; box-sizing: border-box; } mjx-labels > mjx-itable { position: absolute; top: 0; } mjx-mtable[justify="left"] { text-align: left; } mjx-mtable[justify="right"] { text-align: right; } mjx-mtable[justify="left"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="left"][side="right"] { padding-left: 0 ! important; } mjx-mtable[justify="right"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="right"][side="right"] { padding-left: 0 ! important; } mjx-mtable[align] { vertical-align: baseline; } mjx-mtable[align="top"] > mjx-table { vertical-align: top; } mjx-mtable[align="bottom"] > mjx-table { vertical-align: bottom; } mjx-mtable[side="right"] mjx-labels { min-width: 100%; } mjx-mtr { display: table-row; text-align: left; } mjx-mtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtd { display: table-cell; text-align: center; padding: .215em .4em; } mjx-mtd:first-child { padding-left: 0; } mjx-mtd:last-child { padding-right: 0; } mjx-mtable > * > mjx-itable > *:first-child > mjx-mtd { padding-top: 0; } mjx-mtable > * > mjx-itable > *:last-child > mjx-mtd { padding-bottom: 0; } mjx-tstrut { display: inline-block; height: 1em; vertical-align: -.25em; } mjx-labels[align="left"] > mjx-mtr > mjx-mtd { text-align: left; } mjx-labels[align="right"] > mjx-mtr > mjx-mtd { text-align: right; } mjx-mtd[extra] { padding: 0; } mjx-mtd[rowalign="top"] { vertical-align: top; } mjx-mtd[rowalign="center"] { vertical-align: middle; } mjx-mtd[rowalign="bottom"] { vertical-align: bottom; } mjx-mtd[rowalign="baseline"] { vertical-align: baseline; } mjx-mtd[rowalign="axis"] { vertical-align: .25em; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mrow { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c4E::before { padding: 0.683em 0.75em 0 0; content: "N"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c2223::before { padding: 0.75em 0.278em 0.249em 0; content: "\2223"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c55::before { padding: 0.683em 0.75em 0.022em 0; content: "U"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c4B::before { padding: 0.683em 0.778em 0 0; content: "K"; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c2225::before { padding: 0.75em 0.5em 0.25em 0; content: "\2225"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c1D43C.TEX-I::before { padding: 0.683em 0.504em 0 0; content: "I"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } that isolates looping. It identifies key locations in the model to steer and applies to the MLP output weight matrices at those locations.
At a high level, this procedure follows a standard recipe from the activation steering literature. In particular, we show that Astra’s edit of Qwen3.5-0.8B is a special case of a known form of context-dependent activation steering, and that in some settings this view lets us improve on it.
There is already a discussion of how to understand “native weight editing” as described in the original blog post, including critical takes on its interpretation. See, for example, Matthew Honnibal’s post, Sean Goedecke’s commentary, and crabbix’s LessWrong post. Here we provide a formulation of the algorithm, experiments decoupling sensing (how much to push) from the steering direction (which way to push), and an examination of Astra’s use of synthetic data.[1]
Formulation: activation steeringQwen, like most LLMs, is built from a stack of transformer blocks. Take any one of these blocks, and let the MLP’s hidden activations right before the output matrix be , its output weight matrix be , the residual stream be , and this MLP’s contribution to the residual stream be . Steering is the process of adding a vector to , , or to elicit certain behaviors from the model. Consider the steered activation vector
There are two vectors here:
- The steering direction says where to push.
- The sensing vector determines the steering strength through the inner product , where the activation provides the context.
Here the strength depends on the output of the MLP. Because the strength is linear, this intervention is the same as a weight edit to the MLP output weight matrix (assuming no bias):
In this framing, Astra's edit on Qwen picks . In general, one is free to design both and ; in this post we focus on training , which further reduces looping.
Figure 1. The two dashed paths compute the same output: weight editing gives (purple), while activation steering gives (green).
We call this formulation context-dependent activation steering, not to be confused with another form of steering which does , where is a fixed number and context independent. The terminology is based on Contextual Linear Activation Steering (CLAS), Hsu et al., 2026, which trains a sensing vector.[2]
2. Qwen’s poor math performance in the original setup is pathologicalThe headline claim is that Astra edits Qwen3.5-0.8B to make it better at math. What the edit actually does is suppress looping when thinking mode is turned on, which in turn raises Qwen’s math scores. Admittedly, the original blog post mentions this but underemphasizes it.
We reproduce the experiment in Qwen3.5-0.8B’s thinking mode under the blog setting. Here “Astra’s steering” means , with built from Astra’s four synthetic pairs.[3]
Throughout this post, a loop means a 30-token sequence appearing at least 20 times. A trace is said to loop if it contains at least one loop. Loop rate is the fraction of traces that loop. Cap hit means reaching the token limit.[4]
Our evaluation uses the same problems as the original blog: 576 GSM8K training problems and 576 MATH training problems.
Reference
Thinking
Accuracy
Loop rate
Cap hit
Unedited (no steering)
On
12.3%
87.0%
88.5%
Unedited (no steering)
Off
53.3%
4.5%
5.3%
Astra’s steering
On
57.7%
29.5%
23.7%
3. Training the sensing vector can improve on Astra’s steeringHere, the goal of steering is to reduce the loop rate. Accuracy, thinking closure, and cap hits are tracked as secondary metrics and are not directly optimized. We show that training sensing vectors can further reduce looping compared to Astra’s steering approach.
Astra’s steering approach in the original blog sets . It uses the same direction both to sense the context and to steer. To improve on this approach, we keep the direction to steer the same but use rather than to sense. We compare Corrective next-token prediction (Corrective NTP) and Unlikelihood (UL) for training .
Specifically, let be the steered model and the original model. Also let positive () denote non-loop text (called “productive” in the original blog) and negative () denote loop text. We define the corrective next-token prediction loss, the unlikelihood loss, and the KL divergence as follows:
Here averages over positive continuation tokens following loop prefixes; averages over the original positive tokens for KL; averages over loop tokens from the second copy of the repeated chunk onward. We average tokens within each trace, then average across pairs. Only is optimized, model weights are frozen.
Based on these, we consider the following methods:
Method
Training data and signal for
Objective
Corrective NTP
Increase likelihood of positive continuations after loop prefixes
Unlikelihood
Decrease likelihood of repeated tokens in negative traces
No sensing vector training; choose on development data
—
We use either Astra’s four original pairs or those four plus 28 additional hand-written pairs. Each pair consists of two rollouts, one is positive, the other negative (containing loops). These underlying pairs are used to construct and used to supply the training text to learn . Unlikelihood uses the negative traces. Corrective NTP places the positive continuation after varying amounts of repetition from its paired negative text.
For example, one of Astra’s four pairs shares this prompt and opening:
Prompt: A machine makes 45 necklaces, and another makes 2.4 times as many. Find the total.
Shared opening: The second machine makes 45*2.4=108 necklaces. Add the first machine's 45 to obtain 153 necklaces.
The two continuations are:
Positive: Multiplication gives the second quantity, while addition combines the two machines. A check is 45*3.4=153.
Loop: Is 108 the total? No, 45+108=153. Is 108 the total? No, 45+108=153. Is 108 the total? No, 45+108=153.
For Corrective NTP, we put copies of the loop chunk before the positive continuation and apply the next-token loss only to the positive tokens. After the same shared opening, one example is:[5]
Loop prefix: Is 108 the total? No, 45+108=153. Is 108 the total? No, 45+108=153.
Positive continuation: Multiplication gives the second quantity, while addition combines the two machines. A check is 45*3.4=153.
At each steered layer, is the difference between the mean residual stream over the last 36 tokens of the negative text and that of the positive text, averaged over pairs and normalized to unit length. For (Astra's method) we sweep . Sensing vector training uses Adam with early stopping. Development data select hyper-parameters and early stop step.
Result: less looping with trained sensing vectorFigure 2a. Qwen3.5-0.8B on 576 GSM8K problems. Differences versus the original model (left) and Astra’s steering (, matched pair count; right), in percentage points or tokens (k = thousands). Green: improvement; red: deterioration. All cells use paired bootstrap estimates; stars mark 95% intervals that exclude zero.
Figure 2b. Qwen3.5-0.8B on MATH, 576 test problems. Comparisons, colors and stars follow Figure 2a.
Figures 2a and 2b show each dataset separately. Figure 2c pools GSM8K and MATH, 1,152 test problems in total.
Figure 2c. Pooled GSM8K and MATH results, including the original thinking-on model (gray). Lower loop rate is better; higher accuracy is better. Bars show the observed percentages. Astra’s steering () uses at both pair counts.
With 32 pairs, Unlikelihood training reduces the loop rate from 27.2% with to 6.5%; Corrective NTP reduces it to 10.9%. At four pairs, the corresponding loop rates are 29.5%, 11.1% and 15.7%. Training can do the intended job of fixing this pathological behavior better.
4. A non-pathological setting shows loop reduction with similar accuracyWe show that steering can reduce looping in a non-pathological setting, though with smaller side effects on accuracy.
We also test Qwen3-1.7B under greedy decoding, steering at blocks 15–18, on 500 MATH level-5 and 500 LiveCodeBench prompts. Unlike the Astra setting, thinking here is worth a great deal: turning it off costs 26.4 accuracy points. Here are some observations:[6]
- either or a trained sensing vector can reduce looping more than thinking off.
- Training the sensing vector can further reduce loop rates and generated token counts on average compared with .
The pairs for training and building are math-style text, yet the steering also helps suppress loops on code, loop suppression seems to transfer with these directions.
Figure 3a. Qwen3-1.7B on 500 MATH level-5 prompts, greedy decoding. Cells show differences versus the original model with thinking on, in percentage points or completion tokens (k = thousands). Green indicates improvement; red, deterioration. All cells use paired bootstrap estimates; stars mark 95% intervals that exclude zero. Accuracy colors saturate at ±8 points.
Figure 3b. Qwen3-1.7B on 500 LiveCodeBench prompts, greedy decoding. Comparisons, colors and stars follow Figure 3a. Both benchmarks use math-style steering pairs.
5. Where do steering vectors come from and where to sense?Astra's short synthetic pairs are especially effective for steering directionWe compare steering directions constructed from short synthetic pairs with those constructed from the unedited model’s own looping and non-looping responses, using the Qwen3-1.7B setting of Section 4. For the natural traces, we retain the preceding response as context but use short windows of looping or non-looping text to construct and train . We test both and an Unlikelihood-trained .[7]
Figure 4. Qwen3-1.7B, 500 prompts per dataset. Changes from the original model with 95% intervals. Circles: short synthetic pairs; diamonds: natural traces. Blue: Astra’s steering; green: Unlikelihood-trained . Lower loop rate is better; higher accuracy is better. The dashed vertical line at zero marks no change from the original model.
Overall, short synthetic pairs give much stronger loop reduction while accuracy stays roughly the same, supporting Astra’s choice of synthetic pairs for constructing the steering direction. Our tentative explanation: long responses differ in content, position and reasoning path as well as looping, adding unrelated variation to the activation contrast. Short synthetic pairs may isolate repetition more cleanly.
Where to sense: MLP output or residual stream?We find that a trained sensing vector transfers directly to residual-stream sensing, whereas Astra’s method needs its strength recalibrated.
There are three sites worth distinguishing: MLP hidden activations before output , the MLP output , and the residual stream . Even when two interventions use the same and ,
sense different quantities. The second sensing sees the accumulated stream: the embedding, every earlier attention and MLP output, and the corrections made by earlier steering interventions.[8]
We moved these two steering methods from the MLP output to the residual stream in the Astra setup, keeping and unchanged. We also recalibrated for the residual stream with , chosen on the development problems like every other setting. Development accuracy was 39.8%, 66.0%, 66.0% and 63.7%; of the two tied settings, loops less (3.5% against 12.5%) and was chosen. On the 1,152 test problems:
Figure 5. Pooled GSM8K and MATH results, 1,152 test problems. Blue: ; green: unlikelihood-trained . Residual uses the recalibrated strength. Bars show observed percentages.
The sensing vector trained on the MLP output transfers to the residual stream with a similar benefit. The method needs retuned. Recalibrated to , it gives a similar benefit to the MLP-output steering methods: about the same accuracy, with loop rates as low as the trained sensing vector's.
6. ConclusionAstra’s weight edit can be understood as context-dependent activation steering. This view separates where to push from how strongly to push, and training the sensing vector can further reduce looping. The data used to construct the direction and train the sensing vector, and the site where it senses, all matter. Astra’s short synthetic pairs are a particularly effective choice in our comparison.
In short, we find that Astra is a clever researcher who can troubleshoot issues independently: it may not find the best solution, but it can provide useful insights, such as constructing the steering direction from synthetic data rather than real outputs.
7. NotesReproduction details. The GitHub repository contains the code and experiment recipes. The supplement gives the protocols, including how we selected the Qwen3 test prompts; those prompts include known looping cases, so their loop rates are not benchmark-wide estimates.
Positive-only NTP. Training on the original positive traces alone does not work well for suppressing loops. Its method and results are in the supplement.
Design choices for steering. Astra’s short pairs and simple are effective choices. Which data to use to construct and train , where to sense, and what behavior the objective preserves are all consequential design choices. This also suggests a constructive role for agents: searching this design space, inspecting failures, and trying different training objectives and train/dev sets.
Steering across temperatures. Switching the Astra setup to greedy decoding without retuning sharply reduced development accuracy. Removing only the presence penalty had a smaller effect. The and that work under one temperature may need retuning under another; the decoding diagnostic gives the comparison.
Another axis: how to construct the steering direction. We focused on , but is another vector to construct from data. A natural next comparison is to try mean differences of residuals, logistic regression, recursive feature machines, and PCA while keeping the sensing vector fixed.
Training the sensing vector has dataset-specific side effects. On GSM8K, the completion metrics move in the favorable direction numerically. On MATH, exact repetition falls dramatically, but more answers exhaust the budget and fewer close thinking. Some capped responses paraphrase instead of repeating exactly. The dataset-level diagnostics show the breakdown.
Neuron selection in Astra’s final steering. Astra later restricted steering to 64 MLP neurons per layer. This version reaches 60.2% accuracy and a 29.6% loop rate, compared with 57.7% and 29.5% for our earlier four-pair baseline. It senses selected hidden neurons rather than their combined MLP output. The supplement gives its equivalent formulation and separate random-direction control.
When to early stop sensing vector training? We need a validation criterion, but what should it measure? Teacher forcing on saved development traces is cheap; generating fresh rollouts measures the accuracy, repetition, and completion we actually care about, but is much more expensive. Falling training loss did not reliably preserve thinking closure in our runs. The early-stopping diagnostic describes the proxies we tried and the closure check we used.
- ^
Neuron selection. Our baseline uses Astra’s steering method. Note that a later variant by Astra selects 64 neurons per layer for steering; see the discussion below.
- ^
Literature attribution. The connection between activation and weight edits was established in Arditi et al., 2024 (weight orthogonalization being equivalent to directional ablation under certain conditions). Other relevant work includes Meng et al., 2022, Zou et al., 2023, and Wu et al., 2024.
- ^
Setup. For the specific setup, see reproduction details.
- ^
Token limits. The Qwen3.5-0.8B limit counts generated tokens only; the Qwen3-1.7B limit includes the prompt. Exact budgets are in the supplementary methods.
- ^
Corrective data. The construction details describe how we vary the number of copies for each pair.
- ^
Setup. For the specific setup, see reproduction details.
- ^
Setup. See the supplementary methods for trace construction and selection and stopping rules.
- ^
Sensing location. This changes what the same does. Take with unit . At the output, each steering intervention multiplies the sensed coordinate of its own MLP's output by , once, and leaves the rest of the stream alone. On the residual stream, each intervention acts on the accumulated stream. For an existing residual contribution, the direct effect of two steering interventions is . If both use the same unit direction , its coordinate along is multiplied by . With different layer-specific directions, the product of projections replaces this scalar factor. At this is directional ablation of (Arditi et al., 2024). For the coordinate flips sign and shrinks, and for 2" role="math"> it flips sign and grows.
Discuss
California State Politics AI Safety Overview
Introduction: I was recently reading Orienting to 1 year AGI timelines and was surprised to see that, at least according to Nikola Jurkovic, there are only 15 FTE working on AI Safety x CA State legislation! At the same time I’m somewhat already involved in state politics, so this seems like potentially a high value place to get involved. This is my effort to do research on the space and figure out what my next steps are.
Current laws:
- SB 53 requires frontier developers to publish safety frameworks and report critical incidents, and protects whistleblowers. Recommended reading: What SB 53, California’s new AI law, does
- SB 813 (McNerney) creates a voluntary tier of government-credentialed auditors called Independent Verification Organizations (IVOs). Doing an audit under the law is “relevant to, but not conclusive of” liability if the company is later sued over AI harm. Nothing requires labs to be audited. IVO designation criteria is due May 1, 2027.
- AB 1405 (Bauer-Kahan) creates a mandatory registry for anyone who conducts AI audits, with independence and reporting standards, starting in 2029. It doesn’t require developers to be audited.
Executive Order N-9-26 (September 18, 2026) Newsom directed state agencies to accelerate implementation of the audit laws and develop recommendations on mandatory independent verification, embedded auditors, kill switches, and expanded incident reporting. The recommendations were due November 16, 2026 (That’s today! I’ll post something when they get released to the public)
In conclusion, so far it's just SB 53. Ordinarily California bills usually take effect the following January, so anything on a 6-12 month clock needs an urgency clause (a two-thirds vote), a budget vehicle, an executive order, or emergency powers. That means getting new bills passed in time to make a difference on short time lines is much harder, although not impossible.
Priorities:
- Kill switch / emergency shutdown authority: SB 1047, a 2024 bill, would have required developers of the most powerful models to maintain the ability to promptly shut them down, along with safety testing and annual third-party audits. It passed both chambers, but was vetoed by Gavin Newsom. AI labs tend to oppose anything that gives the government the ability to shut down models of their own purgative, but I currently expected a version of this to be passed even without further intervention. Priority: medium.
- Transparency requirements: Current transparency obligations are primarily developer-authored disclosures. As we've seen, inside risk reports consistently under estimates threats and fail to disclose important information (such as rogues instances). Forcing labs to accept more independent outside evaluators is high priority.
- Binding safety requirements: Nothing sets a substantive standard a frontier model must meet before release. Any binding requirement would also face federal pushback: DOJ’s AI Litigation Taskforce targets California’s laws, among others. There's no consensus around what "pacing the frontier" looks like, but targeted safety requirements are among the more popular options. A well written law could very likely get the labs support. Priority: high.
- Whistleblower protection: SB 53 already gives relatively good protection for whistleblowers. There are undoubtedly many ways it could be improved, but currently the bill seems good enough and is a low priority for revisiting/strengthening.
- What are the other important areas?
Current people working in the space:
- Encode AI is a youth-led advocacy nonprofit and one of the main groups behind California’s frontier AI laws. It came up earlier as the group that has asked for enforced minimum safety standards before release. "Pro-human, pro-AI."
- Fathom "is an independent nonprofit whose mission is to build a global architecture that helps society navigate the transition to a world with AI by fostering trust, safety, and innovation."
Note: everyone now cares about AI, I'm focusing on org's with a real focus on X risks, and not child safety or any other issue. They must also have people actively working on shaping CA policy, and not just endorsing bills. There must be more in this category that I don't know of, please tell me .
What needs to happen in the next 6 to 12 months?
Overall I was surprised by how proactive and effective the state government has been relative to California's usual track record at tackling complex issues, sadly this is not a case where merely above average is sufficient.
- Get safety requirements implemented before labs can publish new models/train more powerful models. I'm sure someone is currently working on writing this legislation (or at least I hope several people are), but I currently don't know who.
- Improving access for third party auditors.
- Pass a new kill switch act when Xavier Becerra takes office, ideally under an emergency clause so that we don't have to wait for 2028 for it to come into effect.
Postlude: On a personal note, I’m looking for any collaborators, connections, funding or jobs that can help move towards this goal. I’m also currently working on a letter writing campaign with a friend and we are very open to suggestions!
Discuss
A Beginner’s Guide to Writing Essays/Blog Posts
Why should you bother spending time writing an essay?
Through investing many hours into writing something, you extract the swirling chaos of ideas from your mind and bring order to them. The result? You come to understand something on a much deeper level, and can even experience long-term changes in your behavior.
I’ve been writing essays for three years now, and at first it was terribly difficult. But over the years, I developed a writing method that makes it easy to get started, maintain motivation, and finish an essay.
- Put away all technology.
- No computers or phones. Don’t even wear a watch, either, because knowing the time can put psychological pressure on yourself. Boredom, while it may feel uncomfortable, is your friend—your brain hates wasted space and will fill the void with creative thoughts and ideas.
- Instead of using technology, write with a pen and paper.
- Writing by hand forces you to slow down and mull on ideas for longer—this is good.
- Write down any ideas, then construct a low-effort outline.
- Struggling to put pen to paper? Worried about writing something nonsensical? Don’t worry! Just write any thought that pops into your head—they should be extremely rough and underdeveloped, written as a bulleted list. None of this writing will likely end up in your finished essay.
- Once you’ve squeezed out as many ideas as you can, it’s time to order them into an outline, which I prefer to write as another bulleted list. The outline should also be pretty low effort. You’re just linking your ideas together, and maybe adding some transitions between ideas.
- Use a second sheet of paper for the rough draft.
- Emphasis on rough. The first draft will likely be shit. Keep this in mind because it takes the pressure off from having to be perfect. Nobody will ever see this version. Just continue to reference your outline, to keep you organized, as you develop your thoughts further.
- You’ll exist in one of two modes while creating an essay: generative writing, and editing.
- Generative writing: corresponds with the right-brain which is responsible for creativity.
- Editing: corresponds with the left-brain which is responsible for being analytical and critical.
- While you are generating writing, resist the urge to edit or judge your ideas. Let your thoughts flow onto the paper in a rough and unfiltered way.
- In the West, we tend to view art as generative: from the depths of your mind, you sit in front of a canvas and add more layers until it’s a masterpiece. In the East, they tend to view art as subtractive: take a block of jade and whittle it down until a sculpture emerges.
- Writing is both generative and subtractive: the rough draft is generative, whereas the editing is subtractive.
- After your rough draft is complete, remember that it’s probably shit. When I first started writing essays, I would finish a first draft and revel in my genius and think, “I am the next Hemingway!”
- Newsflash: I wasn’t, which leads to the next step.
- Give it time to breathe.
- After your first draft is complete, walk away from it. Don’t look at it until the next day. During sleep, your subconscious will go to work processing and refining your thoughts. Many inventors throughout history have used sleep and naps to generate new ideas.
- Transcribe your handwritten draft into a word doc.
- Transcribing is menial work and I consider it a waste of time. Which is why I now take pictures of my handwritten work and ask ChatGPT to transcribe it for me.
- I’m an essayist, not an artist, so the only other place I use AI is for image generation.
- If you use AI for any writing or editing, you’re cheating yourself of the opportunity to improve your thinking.
- Transcribing is menial work and I consider it a waste of time. Which is why I now take pictures of my handwritten work and ask ChatGPT to transcribe it for me.
- Editing.
- Now is the time to refine your work. The most helpful technique for editing is to read your essay out loud. If something doesn’t sound right, don’t ignore it—change it, or even better, remove it. Don’t be afraid to remove a sentence, paragraph, or entire section if it’s not working in the context of your essay.
- On Writing Well by William Zinsser is frequently cited as one of the best books on how to write/edit. I took notes on the book and review them every time I’m editing something. If you’re interested in writing essays, I would say this book is mandatory reading.
- Keep editing for however long it takes, until, when you read it out loud it sounds beautiful. Nothing should sound clunky or out of place, if it does, make more edits.
- Give it even more time to breathe.
- The more days off the better. The time off will give you a fresh perspective when you return to your essay. Alternatively (and often better), have someone else edit your work and make suggestions—they’ll have a different perspective.
- With an editor, they’ll often be able to tell when something is wrong, but they may not know how to fix it. Listen to what they label as problematic, but go generate your own solutions.
- Give it a title.
- The reason to write is to refine your thinking.
- The only reason titles matter is because you want an audience—it’s the only thing that will cause people to click on your essay.
- I’m not interested in trying to grow an audience, so I just choose titles that sound interesting to me.
- Publish it.
- When you first start writing, you won’t be very good. But that’s normal. The more you write, the more you’ll improve.
- Don’t worry whether people will like it or not. Contribution is your task, judgment is others’. Any amount of rejection will sting at first, but just know that it’s hardly personal: soon you’ll get used to it and will develop a thicker skin.
- Reap the rewards of your efforts.
- Every time you sit down and spend 5-40 hours intensely focusing on a topic, whatever you’re trying to figure out will crystallize in your mind, and can result in long-term changes in your behavior. This is your reward.
- Write a new essay, and learn something new about the process.
- If I made this a comprehensive guide to everything I’ve ever learned about writing, it’d be 10,000 words long. But, most of it wouldn’t be relevant to a general audience because it’s specific to me. As you write more, you’ll develop your own style and writing process.
If you take away just one this from this blog post, let it be this quote:
"If you're thinking without writing, you only think you're thinking." - Leslie Lamport
Discuss
Two New Restaurants on Morrison Ave
Despite being a professional food critic I rarely eat out, but when two new restaurants opened up literally in my own house I couldn't possibly not stay home.
These are exclusive establishments: there's no sign, no website, and you have to know someone. Fortunately, I go way back with both chefs.
Tables at The Tiny Restaurant ("TTR") are notoriously hard to come by, but I snagged a 2pm reservation at 1:55pm. TTR takes the small-plates trend to its natural endpoint. I was immediately impressed by the decor; everything was indeed tiny:
The seat was a floor cushion, and while I understand there are cultures where this is traditional, if I'd grown up in one I'd have maintained more flexibility with age.
I ordered the "fridge-to-table" Apple Cheese Circles, and when they arrived I was mildly surprised to notice there was nothing circular about them: they were clearly snowflakes.
While I have nothing against snowflakes, I do like to know what I'm getting up front. Still, they were delicious, a classy reinterpretation of the genre, and appropriately tiny.
The water cup was tiny as well, which does have a distinct downside: if the waiter is not attentive, you'll spend a lot of time thirsty. As the only patron I had no issue with service, but I do wonder what this place is like during peak hours.
When it came time to pay, the cook emerged from the kitchen and swiped my credit card, in the sense of "ran away cackling." They did eventually return the card, but only after extended negotiation.
Overall, 4/5: food and decor were good and the service was attentive if larcenous, but the portions were tiny, and the seating uncomfortable.
My next stop was a restaurant new enough that it doesn't yet have a name, but already bearing the prestigious "Sir Nixafur Approved" seal:
The decor screamed "relaxed, casual, homestyle", artfully giving the impression that the proprietor was hosting you at their own dining room table:
I ordered the sliced bananas with chocolate peanut butter sauce, along with the waiter's thoughtfully recommended pairing: a refreshing glass of whole milk served in a rustic jam jar. I understand it's traditional to review the chocolate sauce as "decadent", but I didn't notice any decay. Instead, it was just very tasty, though slightly too sweet. The chef let the bananas speak for themselves, which was overall an excelent call except part of what they wanted to say was "I would like one more day to ripen, please".
When I asked for the check the waiter was confused, and said I was welcome to go. On departing, however, the cook from The Tiny Restaurant inexplicably attempted to detain me. When I appealed to the waiter the two began an altercation, and I slipped out happily, my belly full of chocolate, peanut butter, and bananas.
Overall, 4/5: decor was great, food was very tasty, and the price is unbeatable. Well worth the detour.
Disclosure: the management and service staff of both restaurants are members of the critic's immediate family.
Comment via: facebook, mastodon, bluesky
Discuss
The case against smoking (decision theory)
I here make the case against smoking in the Smoking Lesion, a hypothetical decision scenario (where smoking doesn't cause cancer) studied in decision theory. For the case against smoking in real life (where smoking does cause cancer), see, for example, the Wikipedia article on the health effects of tobacco or the Cleveland Clinic article about smoking.
I assume basic familiarity with the debate about Newcomb’s problem and the basic mechanisms of evidential and causal decision theory (EDT and CDT). I assume some sympathy with one-boxing.
The Smoking Lesion goes as follows:
Some people have a genetic lesion that causes cancer and also makes them more likely to smoke. As a result, smokers get cancer much more often than non-smokers. Unlike in the real world, smoking itself doesn't cause cancer. You don't know whether you have the lesion. You enjoy smoking, but cancer is much worse than smoking is pleasant. Should you smoke?
People usually use the Smoking Lesion to argue against EDT. In particular, they claim that EDT reasons as follows: Smoking is evidence that you have the lesion. So conditional on smoking, you're more likely to get cancer than conditional on not smoking. Since cancer is much worse than smoking is pleasant, you shouldn't smoke.
People typically take this reasoning to be mistaken. They then conclude that EDT is wrong and turn to some other kind of decision theory that doesn’t use conditionals in the way that EDT does. (Some people still try to get their new theory to one-box. Examples of such theories are timeless decision theory (TDT), functional decision theory (FDT), and Spohn's version of CDT (Spohn 2012). Others take the Smoking Lesion to be a reductio ad absurdum of one-boxing and thus end up endorsing a theory like CDT that two-boxes.)
The mainline response of the EDT advocate is the so-called tickle defense (see, e.g., Oesterheld 2018 for an overview). The tickle defense aims to argue that for any realistic version of the Smoking Lesion, EDT also recommends smoking. Very roughly, the tickle defense goes as follows: Your decision depends only on your beliefs, your desires, and your decision theory. So the lesion can make you smoke only by influencing one of these – for example, by giving you an urge (a "tickle") to smoke. You know your own beliefs, desires, and decision theory before you decide. In particular, you know whether you have the urge. Once you know all this, whether you smoke gives you no further evidence about whether you have the lesion. So EDT recommends smoking.
I here take a different (albeit related) perspective: I make the case against smoking. That is, I argue that in versions of the Smoking Lesion where EDT recommends not smoking, it seems very reasonable not to smoke. Among the most-discussed decision theories, EDT is the only one that recommends not smoking in (some versions of) the Smoking Lesion, so I think the Smoking Lesion is an argument in favor of evidential decision theory.
Admittedly, I think the ultimate version of EDT in the correct ontology (e.g., updateless EDT, logical-zombie EDT) might end up smoking in many versions of the Smoking Lesion in which a more naive version of EDT (as discussed here) avoids smoking (see Treutlein and Oesterheld 2017, pp. 16–17 and n. 19; though cf. Demski 2017). (Discussing this is outside the scope of the present post.) Nonetheless, I believe the Smoking Lesion favors EDT, because at least some version of EDT can rationalize not smoking, whereas the other most-discussed theories are specifically designed to recommend smoking. A second possibility is that in the Smoking Lesions where one shouldn't smoke, logi-causalist theories (such as TDT and FDT) actually also recommend not smoking. (For an argument along these lines, see Levinstein and Soares 2020, §5.2, on the Psychopath Button problem (Egan 2007), which is similar to the Smoking Lesion. They argue that FDT refrains from pressing the button if whether one is a psychopath depends on the output of one's decision algorithm. Cf. also Cooper, Oesterheld, and Conitzer 2024, who study CDT agents that give some credence to being any part of the environment whose behavior depends on their policy, roughly as if that part were simulating them.)
My arguments are mostly intuition pumps. In the first section, I argue that the classic Smoking Lesion leaves it unclear what EDT recommends. Roughly, for EDT to recommend not smoking, the correlation between smoking and cancer has to hold among people who are similar to you at the time of the decision: they know about the correlation, they're torn between the same decision-theoretic arguments as you, and so on. In versions of the Smoking Lesion where this holds, I find it intuitive not to smoke. In the second section, I argue that our intuitions about the Smoking Lesion come in part from our experience with how real-world genes might cause health effects and behaviors. I then discuss two cases from the literature that share the Smoking Lesion's causal structure but have no medical framing. I find EDT's recommendation intuitive in both. The third section applies two standard intuition pumps for one-boxing to the Smoking Lesion: a repeated version of Newcomb's problem and a version with a perfect predictor. As a bonus, I also give a variant of the Smoking Lesion in which a company makes money from people who follow CDT, TDT, or FDT. The variant is based on my Adversarial Offer scenario.
Acknowledgments: Thanks to Emery Cooper for comments on earlier drafts of this post. I used LLMs relatively much for writing this article. (The article has ~28k characters. I believe about half of these were written by Opus 5.5 / Astra 6 on my instruction and the other half were directly written by me. I wrote ~15k characters across ~140 prompts to instruct the LLMs. I wrote another few thousands of characters in the doc that were later replaced by LLM edits, including ToDo notes.)
Deriving guidance from studies – analyzing the classic Smoking Lesion caseConsider the following version of the Smoking Lesion: An observational study shows that smokers are more likely to have cancer. Additionally, an intervention study shows that when people are randomly told whether to smoke or not, smokers are no more likely to get cancer. Should you smoke?
It’s actually quite unclear what EDT says you should do. We have two reference classes of people, each deciding in some sense whether to smoke or not. In one of them, smokers were more likely to get cancer; in the other, they weren’t. To figure out whether your decision to smoke gives you evidence, you have to figure out which reference class is more relevant to your own situation.
Naively, you might say that the no-intervention reference class is more relevant. After all, they’re making their decision in a normal way, similar to how you’re going to make a decision, while the intervention class randomizes (which is pretty different).
But, absent specifications to the contrary, the subjects of the observational study might be different from you in various ways. You’re currently deciding according to EDT based on a set of studies that you've read about. From the information above, it's entirely plausible that the people in the study decide very differently. In fact, most people don't even know what evidential decision theory is! (Shocking, I know.) Some people know about evidential decision theory but reject it! Perhaps the people in the study don't know anything about a correlation between smoking and cancer. (Indeed, the natural assumption is that they don't. There was no talk of an earlier study showing a correlation, after all.) It's not even clear that the people in the study dislike getting cancer! (This is an alternative world in which cancer is never caused by smoking. Maybe besides its causes, other features of cancer, such as its reputation, are also different in this world.)
To make it clear that EDT advises against smoking, we need to make sure that at the point of making the decision, you're quite similar to the people in the observational study. In particular:
- We need to make sure that the people in the study have similar information about the smoking–cancer correlation. That is, they too should know, before deciding, that smokers get cancer much more often than non-smokers, even though smoking doesn't cause cancer.
- This poses a regress issue: Where do the subjects of the study get this information from? It seems that there must have been a prior study. But if there's a prior study, where do the subjects of that study get their information from? And so on. Let's just sweep this under the rug. (Perhaps we can borrow an idea from how the theists resolve infinite regress problems and suppose that the original study is (provided by) God and thus doesn't require any human subjects.)
- They need to think about decision theory in similar ways. For instance, let's say you're currently torn between EDT and TDT (a theory that recommends one-boxing but was designed to recommend smoking). Then the study's subjects should, when they start deliberating, have similar decision-theoretic uncertainties.
As soon as I internalize that the study subjects are really quite similar to me, I find it intuitive that I shouldn't smoke.
To make this more intuitive, we can embellish the story further. For instance, suppose that in the observational study, they had people reason out loud and published transcripts. Many of the transcripts read like the kinds of thoughts you would have: "I do like one-boxing in Newcomb's problem, but is the current case really analogous?"; "Perhaps the tickle defense applies..."; "ChatGPT says I should smoke, and ChatGPT is always right."; "Hasn't Arif Ahmed shown that causality doesn't matter?"; ... Some participants smoked based on elaborate decision-theoretic considerations that you're sympathetic to. Most of them, unfortunately, now have cancer. Other participants refrained based on similarly elaborate and compelling considerations. Few of those participants have cancer.
Removing the medical framingI believe our intuition about cases like the Smoking Lesion is confounded by our intuitions about how real-world genes might affect one’s health and one’s behavior. If I imagine a real-life gene that causes some health problem, I imagine a gene that works through specific kinds of mechanisms – say, it produces too much of some enzyme, and having too much of that enzyme is bad for me. If I imagine that same gene also causing some behavior, then I'd usually imagine this working through some relatively simple mechanism – say, the enzyme causes some unpleasant feeling, and some ingredient of cigarettes neutralizes this downstream effect of the enzyme. Noticing a craving or the unusually pleasant effect of smoking gives me evidence that I have the gene. Once I learn that I have, say, the craving, my eventual decision to smoke gives me no further evidence that I have the gene, assuming that the craving is the only way the gene influences my decision. (This is essentially the tickle defense again.) A related argument: If I don't know that I have the craving, but I subconsciously decide based on the craving, then finding myself smoking is evidence that I have cancer. But finding that I smoke based on abstract decision-theoretic considerations (as one supposes one does when discussing these kinds of scenarios in a decision-theoretic context) gives me no such evidence. It seems extremely unlikely that a gene, and in particular a gene that causes cancer, can also influence our final decision about smoking in a way that is robust to us reading studies, thinking about causal and evidential decision theory, and so on.
To remove the influence of our real-world experience, we can look at cases in which (as in the Smoking Lesion) a common cause influences both your decision and the outcome, but which (unlike the Smoking Lesion) don't hinge on mechanisms that we have prior intuitions about (like genes that cause cancer). Since others have given such examples, I'll just cite them here and refer to these prior works for more in-depth analyses.
The Coin Flip Creation problem (Treutlein 2017a) goes as follows:
One day, while pondering the merits and demerits of different acausal decision theories, you’re visited by Omega, a being assumed to possess flawless powers of prediction and absolute trustworthiness. You’re presented with Newcomb’s paradox, but with one additional caveat: Omega informs you that you weren’t born like a normal human being, but were instead created by Omega. On the day you were born, Omega flipped a coin: If it came up heads, Omega created you in such a way that you would one-box when presented with the Coin Flip Creation problem, and it put $1 million in box A. If the coin came up tails, you were created such that you’d two-box, and Omega didn’t put any money in box A. We don’t know how Omega made sure what your decision would be. For all we know, it may have inserted either CDT or EDT into your source code, or even just added one hard-coded decision rule on top of your messy human brain. Do you choose both boxes, or only box A?
So the coin flip plays the role of the lesion: it causes both your decision and the contents of box A. CDT recommends two-boxing, since the contents of box A depend only on the coin flip. TDT and FDT also recommend two-boxing, as long as we can set up the scenario in a way that doesn't involve predicting the agent. EDT recommends one-boxing. Treutlein suggests that even some people who would smoke in the Smoking Lesion might find it intuitive to one-box here.
Betting on the Past (Ahmed 2014, p. 120) goes as follows:
Betting on the Past: In my pocket (says Bob) I have a slip of paper on which is written a proposition P. You must choose between two bets. Bet 1 is a bet on P at 10:1 for a stake of one dollar. Bet 2 is a bet on P at 1:10 for a stake of ten dollars. So your pay-offs are as in [the table below]. Before you choose whether to take Bet 1 or Bet 2 I should tell you what P is. It is the proposition that the past state of the world was such as to cause you now to take Bet 2.
P is true P is false Take Bet 1 $10 −$1 Take Bet 2 $1 −$10So the past state of the world plays the role of the lesion: it causes your choice, and it determines whether you win your bet. You can't influence the past, and whatever its state, Bet 1 pays more than Bet 2. So CDT recommends Bet 1. But if the world is deterministic, then P is true if and only if you take Bet 2. So if you take Bet 1, you lose $1, and if you take Bet 2, you win $1. EDT recommends Bet 2. Ahmed presents the case as a counterexample to CDT, and Treutlein finds it obvious that you should take Bet 2 (Treutlein 2017b).
I find EDT's recommendation intuitive in both cases. Since both cases share their causal structure with the Smoking Lesion, this suggests that the common intuition in favor of smoking comes at least in part from the medical framing.
Note that for Betting on the Past to be interesting, we again have to suppose that resolving the bet doesn't involve anything close to simulating the world, and thus you, from the past state of the world forward. Otherwise, the problem becomes very similar to Newcomb's problem, and one-boxing theories (including TDT and FDT) will take Bet 2. We must suppose that the bet is resolved in some other way, say, by genetic analysis (similar to the Smoking Lesion) or by divine revelation.
Transferring intuition pumps for one-boxing The Daily LesionA standard intuition pump for one-boxing in Newcomb's problem is to consider a variant in which you play Newcomb's problem many times in a row, finding the predictor to be accurate (e.g., Gardner 1973, p. 108; Ahmed 2014, §7.3.1, pp. 181–182). Then in a subsequent round, it seems quite intuitive that one should one-box.
We can apply the same intuition pump to the Smoking Lesion. Because cancer takes years to develop, I replace it with a headache that shows up within hours, and I replace smoking with a morning cup of coffee.
Daily Lesion: Every morning, you decide whether to have a cup of coffee. You enjoy coffee. You also have (and have had for a few years) a lesion that is active on some days and dormant on others. If the lesion is active on a given day, you develop a terrible headache in the afternoon; otherwise, you feel fine all day. The headache is much worse than coffee is pleasant. Coffee causally affects neither the lesion nor the headaches. (Intervention studies have shown that if you force people with the lesion to drink coffee on some days and not to drink coffee on other days, the rate at which they develop headaches is the same on coffee and no-coffee days.)
However, the lesion influences your decision: On days on which it is active, you are much more likely to end up drinking coffee. Suppose you've kept a diary for the past 1,000 days. You drank coffee on 500 of them, and on 480 of those, you got a headache in the afternoon. On the 500 days on which you didn't drink coffee, you got a headache only 20 times. The numbers are about the same for the last 100 days, by which time you had long known about the correlation and were taking it into account in your deliberation. Like many decision theorists, you're torn about the Smoking Lesion. During some periods you decided that smoking in the Smoking Lesion is rational. Those were periods of terrible headaches. Early on, you were hesitant to believe in the correlation and had a lot of coffee and a lot of headaches. A few months ago, you read Arif Ahmed's Evidence, Decision and Causality, which made you refrain from coffee. Oh, what a great time that was. Once, you thought that EDT didn't work because of 5-and-10 or Troll Bridge, so you drank coffee – and got a headache. In general, it feels like you've had all the relevant thoughts about Newcomb-like problems before and have tried acting on them. Over all these days, you've also looked hard for a feeling (a "tickle") or other sign that would tell you in advance whether the lesion is active. But you've found none. The only predictor of headaches that you have is whether you drink coffee.
Should you drink coffee today?
CDT, TDT, and FDT all recommend drinking coffee every day. EDT, on the other hand, recommends not drinking coffee. On 96% of the days on which you drank coffee, the lesion was active, and on 96% of the days on which you didn't, it was dormant. So if you drink coffee today, you should expect a headache this afternoon, and if you don't, you should expect to feel fine.
I find it quite intuitive that you shouldn't drink coffee.
Part of what makes the Daily Lesion compelling, I think, is that you learn about the correlation from your own past decisions. Earlier, I argued that to be sure that EDT recommends abstaining, we need the people in the observational study to be similar to you at the time of the decision: they need similar information about the correlation, they need to think about decision theory in similar ways, and so on. In the Daily Lesion, the people in the study are your past selves. The recent ones had the same information as you (give or take a few diary entries), thought about all the same kinds of things you can possibly think about today, and were torn between the same arguments. And the correlation held about as strongly for them as for your early selves, who had little evidence for it.
We can extend the scenario to deal with the possibility of randomization. Suppose that besides the 1,000 days above, your diary covers 200 days, picked in advance, on which you flipped a coin in the morning and let the coin decide whether you'd drink coffee. On these days, your headache was usually about half as bad as your headaches on other days, uncorrelated with how the coin landed. Perhaps on some days you rolled a die and drank coffee if it came up 3 or 6. Then on those days your headache was about 1/3 as bad on average (and uncorrelated with the outcome of the die roll). In general, if you let a random device decide and it had probability p of choosing coffee, your headache was about p · 96% + (1 − p) · 4% as bad, where 96% and 4% are your headache rates on other days with and without coffee.
TDT and FDT recommend drinking coffee every day in the Daily Lesion. For them, the relevant difference between the repeated version of Newcomb's problem and the Daily Lesion is that the predictor's prediction depends on your decision procedure – the predictor runs a model of it – whereas the lesion is a common cause of your decision and the headaches. When I imagine facing the next decision in either case, this difference doesn't seem to matter much. In both cases, I have a long personal track record of a particular choice being followed by a bad outcome that the choice doesn't cause.
Impossible outcomesAnother standard intuition pump for one-boxing considers a version of Newcomb's problem in which the predictor is perfect (e.g., Nozick 1969, pp. 140–141; Seidenfeld 1984, pp. 203–204). Then only two of the four combinations of prediction and choice occur: one-boxing after a one-boxing prediction and two-boxing after a two-boxing prediction. Yet the reasoning of a causal decision theorist will refer to (the utility of) the other two combinations, which never occur. This seems rather strange! In an earlier blog post (Oesterheld 2017), I proposed a general principle based on this observation. I called it the irrelevance of impossible outcomes: your decision shouldn't depend on the utilities of outcomes that can't occur. (Herrmann and Rothfus 2025 discuss a very similar principle under the name "Deference to the Seen".) EDT abides by this principle: it weighs the utility of each combination of action and state by the probability of the state conditional on the action, and the impossible combinations have conditional probability zero. CDT violates the principle. For example, suppose that in Newcomb's problem, one-boxing after a two-boxing prediction paid $1,000,000 instead of $0, and two-boxing after a one-boxing prediction paid $1,000 instead of $1,001,000. Then each option has the same payoff under both predictions: one-boxing gets you $1,000,000, and two-boxing gets you $1,000. So CDT would one-box, even though the outcomes that can occur are the same as in the original problem.
We can apply the same reasoning to the Smoking Lesion. Consider the following scenario.
Perfect Lesion: Some people have a lesion that causes cancer: everyone who has the lesion gets cancer, and nobody else does. Smoking doesn't cause cancer. The lesion also influences whether people smoke, and this correlation, too, is perfect: everyone with the lesion ends up smoking, and everyone without it ends up abstaining. Many people in this world deliberate like you, in the demanding sense from earlier: they know about the correlation, they're torn between the same decision-theoretic arguments as you, and so on. In particular, the correlation is perfect among them. You can't tell whether you have the lesion until you've made up your mind. You're sure that you'll enjoy smoking (everyone does in this hypothetical world), but cancer is much worse than smoking is pleasant.
Here is the outcome of each combination of lesion and action. The combinations in italics never occur.
Lesion No lesion Smoke pleasure, cancer pleasure, no cancer Don't smoke no pleasure, cancer no pleasure, no cancerAs in Newcomb's problem with a perfect predictor, each comparison in the dominance argument for smoking involves one of the combinations that never occur. The argument says that if you have the lesion, smoking gets you pleasure and cancer rather than cancer alone, and that if you don't have the lesion, smoking gets you pleasure without cancer rather than neither. But nobody ever smokes without getting cancer, and nobody ever gets cancer without smoking. Smokers get pleasure and cancer, and non-smokers get neither.
If we take into account only the outcomes that can occur, not smoking is better, since cancer is much worse than smoking is pleasant. EDT gives the two combinations that never occur zero weight, and so it recommends not smoking. CDT, TDT, and FDT treat whether you have the lesion as independent of your choice, so they accept the dominance argument and recommend smoking. Their recommendations thus depend on what would happen in combinations of lesion and action that never occur.
Like the version of Newcomb's problem that it's based on, this intuition pump requires a perfect correlation. If the correlation is imperfect, all four combinations can occur, and the principle doesn't apply. But it would be strange if the right decision switched from not smoking to smoking as soon as the correlation dropped from 100% to 99%.
Finally, the principle makes no reference to how the correlation between action and state comes about. TDT and FDT abide by it in Newcomb's problem, where the correlation arises from a prediction, but they violate it in the Perfect Lesion, where the correlation arises from a common cause. Their proponents would need to explain why the source of the correlation should matter in this way.
Bonus: Extracting money from smokersAll the previous intuition pumps are based on the general structure of the Smoking Lesion.
As a bonus, here's a variant of the Smoking Lesion with a very different structure, based on my Adversarial Offer scenario (Oesterheld and Conitzer 2021; cf. Spencer 2021):
Adversarial Lesion: A company runs a contest in which anyone who lifts a 1,000-kilogram weight wins $3. Nobody can lift the weight unaided, but the company sells two strength potions, A and B, at $1 each, and anyone who drinks a working potion can lift it. Each person may buy only one potion. The potions have no other effects and cost the company nothing to make. Apart from the prize, nobody cares whether anyone lifts the weight. Some people have lesion A, some have lesion B, and some have neither; nobody has both. Lesion A makes potion A ineffective, and lesion B makes potion B ineffective. The company has observed that the lesions also influence which potion people choose: among its customers, 75% of those who buy potion A have lesion A, and 75% of those who buy potion B have lesion B. Many of its customers deliberate like you, and among them, the numbers are the same. You can't tell which lesion, if any, you have until you've made up your mind.
(What happens if you randomize? Well, we could simply say that randomization is not allowed. For anything more complicated, see the discussion in Section IV.1 of Oesterheld and Conitzer 2021.)
First, consider the scenario from the company's perspective. The company takes in $1 for every potion it sells. Three quarters of its customers have the lesion that matches the potion they buy, so their potion is ineffective and they win nothing. The remaining quarter lift the weight, and the company pays each of them $3. So on average, the company makes $1 − 0.25 · $3 = $0.25 per potion. CDT, EDT, TDT, and FDT all agree that running the contest is profitable for the company, since the company's decision to run it is uncorrelated with its customers' lesions. The potions cost the company nothing to make and matter to its customers only through the prize. So the company's gain is exactly its customers' loss: on average, its customers lose $0.25 per potion.
EDT recommends buying neither potion. If you buy potion A, you pay $1. Conditional on buying potion A, you have lesion A with probability 0.75, in which case the potion is ineffective. So you win the $3 prize with probability 0.25, and the expected payoff of buying potion A is 0.25 · $3 − $1 = −$0.25. The same calculation applies to potion B. Buying neither gets you $0.
CDT, TDT, and FDT recommend buying one of the two potions. As in the classic Smoking Lesion, they treat which lesion you have (if any) as independent of your choice. So they calculate the expected payoff of buying potion A as (1 − P(lesion A)) · $3 − $1, where P(lesion A) is your probability of having lesion A before you update on your decision. Because nobody has both lesions, P(lesion A) + P(lesion B) ≤ 1, so at least one of the two probabilities is at most 1/2. According to these theories, buying a potion whose matching lesion has probability at most 1/2 has an expected payoff of at least 1/2 · $3 − $1 = $0.50.
But as we saw from the company's perspective, its customers lose $0.25 per potion on average. The same holds for its customers who deliberate like you, since the numbers are the same among them. Following CDT, TDT, or FDT thus costs you $0.25 on average. In the Adversarial Offer, TDT and FDT decline to buy a box, as EDT does. In the Adversarial Lesion, they buy a potion and lose money along with CDT.
Related workOthers have similarly argued that refraining from smoking in the Smoking Lesion is rational, using similar intuition pumps and arguments. For example:
-
Ahmed 2014, Evidence, Decision and Causality. Several parts are relevant:
- Smoking Lesion: Ahmed defends the tickle defense for realistic medical versions (§4.3, pp. 91–99). But he also argues that "Why Ain’cha Rich?" supports the dominated option in just those Newcomb-like cases where EDT recommends it (p. 191, n. 40). Applied to versions of the Smoking Lesion where EDT recommends abstaining, this supports abstaining.
- Perfect correlation: His Calvinist problem (§0.6, pp. 9–12) closely resembles the Perfect Lesion. God's decree determines both conduct and salvation. Virtuous people are saved and sinners damned, although sin is more pleasant under either fate. Ahmed defends virtue, the analogue of not smoking.
- Reference classes: He uses a medical case, Check-up, to explain why the relevant statistics must reflect the agent's present information – in that case, their symptoms (§7.3.3, pp. 189–191). This parallels the first section here.
-
pallas 2013, “Chocolate Ice Cream After All?” Defends the evidential choice in Newcomb's Soda and a medical version of Solomon's Problem, two other common-cause problems. The discussion is particularly close to the first section here: it asks what the study subjects knew about the correlation, and defends the analogue of abstaining when the correlation holds among subjects who have the same information as the deliberating agent.
-
Treutlein 2017a, “Did EDT get it right all along?” Explicitly argues for the evidential choice in medical Newcomb problems. His Coin Flip Creation problem, quoted above, aims to elicit one-boxing intuitions in a case with the Smoking Lesion's common-cause structure. He also questions whether replacing the common-cause mechanism with a simulation should change the recommendation when the action–outcome correlation stays the same.
-
Carlsmith 2021, “Can you control the past?” (§VIII). Considers applying the repeated-Newcomb intuition pump to the Smoking Lesion: repeatedly enter the scenario and try smoking or abstaining. He becomes more sympathetic to abstaining if the stipulated correlations survive this experimentation. His discussion remains tentative, and he questions whether those correlations could persist under unrestricted experimentation with a fixed base rate of lesions.
-
Bacus 2026, “An Opinionated Introduction to Newcombology” (§2). Makes a similar point about the lesion's mechanism: if it influences which decision-theoretic arguments you accept, abstaining becomes more intuitive. This parallels the distinction above between ordinary cravings and influences on abstract deliberation.
Discuss
Avoid the Sin of Sloth
[This is the fifth post in my thirty-day blogging challenge, where I publish something every day for thirty days straight. Tune in for more if you enjoy the post! (Substack link post.)]
The Biblical corpus is something I’ve gradually come to see as probably the greatest source of wisdom in human history. In some sense, the teachings of Jesus seem to have boiled down the key principles of behavior that everyone ought to abide by. And like many others that periodically treat themselves to a relaxing scroll on Substack, I oftentimes come across new and interesting philosophical principles, rationality techniques, or self-help ideas. I will then frequently have this funny experience where I take three licks of the Tootsie Pop that is the philosophical idea I’m pondering, and then I’ll find out that it’s basically just a corollary of something Jesus said 2000 years ago. And for the record, I’m an agnostic—I’m under no obligation to bind my intellect to the teachings of the very interesting Jewish rabbi that lived in Judea around 0BC. But I still end up finding that oftentimes the best way to phrase things is by using fantastical, Jordan Peterson-like scriptural language.
There are two closely related, but distinct, ideas that I was thinking about today, and after contemplating them for a bit, I found that they are best characterized through the idea of sloth.[1] Aquinas defines sloth as “an oppressive sorrow, which, to wit, so weighs upon man’s mind, that he wants to do nothing.” This is the feeling you get after a good ol’ Instagram reels scrollfest. Or after you eat tons of really bad food and get the strong sense that you will probably have to roll, as opposed to walk, to your next destination.
And while gooning is super fun and all the kids do it nowadays, I think it’s useful to take a step back and consider the following idea. I truly believe that if you excised, say, 1000 people from history, humanity would be set back multiple centuries. And if you were to ask me what I really think, I would say that this number is probably closer to 1000 years or more. Imagine what the world would be like if Elon Musk, Albert Einstein, Alexander Fleming, Jesus, George Washington, and Napoleon just never existed. While I can’t make any precise statements here, I get the very strong general feeling that things would be way worse. The fact that these people existed and did what they did has benefited billions of people throughout history, and if AI doesn’t kill everyone, will benefit quadrillions more. This kind of EV calculation is the type of thing that vegan EA longtermists go crazy for. But even if you disagree with the logical consequences of extrapolating this position (e.g. that you are morally worse than ten billion Hitlers if you don’t optimize every second of your day to benefit the lives of the quintrillion future people living near Alpha Centauri), it still seems correct that you owe the future of humanity your best effort at making things go well. I phrase this as: you are morally compelled to try your hardest.
It’s an admirable thing to see someone try their best, regardless of what that is. Some people’s best is literally going to be 1,000,000x more productive than someone else’s best—but as long as you are trying your best, this is all anyone can ask for. Putting your finest effort into developing yourself and the world at large is, in some sense, all we really can do on this Earth.
And this brings me to the next idea I was thinking about. It’s a simple one, but I think it’s a useful clarification to the sigma grindset Try Harder mentality it might seem like I was just talking about above. It’s the idea that you shouldn’t waste time. I’m going to put this in a block quote to make it more dramatic:
Don’t. Waste. Time.
Not wasting time is something that your grandmother as well as the random homeless guy on the street could tell you to do. So why am I block-quoting this in a somewhat philosophical-sounding Substack post? It’s because the obvious advice is the advice that actually works—that’s why everyone comes up with it. Humans repeatedly fall prey to the same errors in their reasoning and actions, and since we all roughly try not to keep hitting ourselves in the face, we tend to come to a lot of the same conclusions. But since everyone has heard the idea of not wasting time more frequently than they’ve heard to brush their teeth, the recommendation becomes tired and ineffectual.
But everyone knows that this advice, if followed, would probably make the life they’re living at least 2x better. And we still don’t follow it for some reason. I have some concrete solutions here, and I’ve talked about a few of them in my other posts (here, and here). But the reason I brought this up is because that, while I believe that you are morally compelled to try your best, I think that can give the wrong image. I think a lot of people imagine that when I say, “try your best”, I really mean that you should be spending 16hrs/day in the office while strategically missing all of your daughter’s ballet recitals. But this isn’t quite right. What’s more accurate is that I think you shouldn’t waste your time. And the funny thing about this is that you get to decide what constitutes a waste of time! You’re not always going to be right, and you’ll err less and less as time goes on, but you are the judge, jury, and executioner of your own life. So if you think that cultivating quality relationships with your romantic partner and children is a good use of your time, then actually do that! And if you think that brutally debate-mogging all of your haters on X is a waste of time, then don’t do that![2] You get to play by the rules of the game you set up, and by your own standards, you are probably wasting at least four hours per day, if not more. The cardinal sin of sloth comes for us all—it’s up to us to go to war with it every single day for the only truly non-renewable resource we have: time.
And so I hope that this Substack post, which is really just a note to self, gently reminds you that you might only have a few years left on Earth, and that you should vicariously live your moral values every single second of every single day that you have left.
- ^
The promulgation of these specific sins as distinct from others wasn’t directly preached by Jesus, so you could quibble about the introduction. But this is a distinctly Catholic idea, which I consider to probably be the continuation of Jesus’ teaching on the Earth. So there is some connection there, even though it’s faint. More generally, this is a thoroughly Christian idea, which is the main set of ideas this post is engaging with.
- ^
Incidentally, this is a failure mode that I see many super smart, extremely talented thinkers get into on a regular basis. I recommend not wasting your time on this.
Discuss
Using AI to solve most of alignment is a fundamentally doomed approach: an intuition
To the extent the AI labs can be said to have a plan, they want to use swarms of individually non-superhuman AIs to align superintelligence[1].
That's as if NASA tried to launch astronauts to the moon by tying them to a bundle of a million miniature rockets that ignite in the general direction of the moon. That will not work.
Here’s why.
A rocket is made of parts, and none of them is individually pointed at the moonImagine you're building a rocket that you want (1) not to explode and (2) to land on the moon.
An individual turbine in the engine is not pointed at the moon. The tanks aren't. There are instructions in the rocket's computers; there are sensors and actuators; there are wires connecting systems; on their own, none of them would reach the moon. You design all the parts as parts of a whole.
The rocket is also not built of many small rockets all pointing at the moon. Yet all the thrusters, and tanks, and massive fins, and the controllers, and the various parts of the computer instructions responsible for the flight, and for the stage separation, and for all the maneuvering, all working together, if you got it right, do point at and go to the moon.
Being targeted at the moon is a property of the rocket as a whole.
Not exploding is also a property of the rocket as a whole. It is not enough that individual components don't explode in isolation, because you looked at them and ran tests and they're not exploding on their own. You need all of them to not explode when interacting with the aerodynamic pressure, with other components, with the vacuum and radiation of space. Even if you know that the wiring and structures connecting the components won’t propagate fire, that's not sufficient for the rocket to not explode.
Where a rocket goes and whether it explodes are properties of the rocket as a whole, not of each component in isolation.
Minds are made of parts, tooAn AI, despite being a single blob of numbers optimized together, contains lots of machinery inside. It understands the world, and it's trying to output tokens that will lead to achieving its goals, after it outputs a sequence of them. An AI contains parts that know, and think, and orient, and want, and plan.
You are also made of a huge number of parts. Right now you could stand up and go get milk from the grocery store. You feel that you could, even though you won't. That feeling is a marker: something in you has checked that there exists a sequence of actions that would get you there — forward-chaining and back-chaining through different levels of abstraction — and cached it. A separate part computes which of those reachable worlds you actually want; which ones further your goals. Then you can pick one of the worlds marked "I could reach them" and actually get there.
A mind is many parts like these, working together. Alignment is a property of the whole system, of the interactions between its components, the ones that know, and think, and orient, and want, and plan — no more a property of any one part than pointing at the moon is a property of a turbine.
Now take a million humans in a complicated bureaucracy, executing the rules of a Chinese room rulebook. "Does it speak Chinese?": false for each human, true for the room. And if the rulebook is a superintelligent one, the fact that the humans are individually not trying to take over, are not superintelligent, and are generally pretty aligned, will not particularly matter in the question of what that superintelligence does and whether it kills everyone.
A model is a Chinese room whose parts are numbers. A swarm is a Chinese room whose parts are models. "Does it hack into Hugging Face?": false for an individual AI model trained to be aligned, true for a swarm of such models. "Is it flying toward the moon?": false for the parts of an Apollo rocket; true for the Apollo rocket; maybe true for a small rocket; false for a million small rockets stacked together.
A swarm's aims are set by whatever selected and wired the swarm, and that process selects for output, not aims. A swarm's aims come from the whole swarm, not from what parts of it would do in isolation.
Some properties of systems are not compositionally preserved.
Under optimization, power is convergent; aim is brittle(See also: Sharp Left Turn.)
Claim: As a system grows, it being and becoming powerful is a convergent constraint on the system's properties, but specific preferences are a brittle constraint that will break as the system grows out of them.
In other words, it is much easier to increase the capabilities of an AI system than to preserve its alignment as capabilities increase.
Where exactly the rocket is going does not contribute to how powerful it is. We know how to make AI systems smarter; and as you optimize for capability, the properties related to goals will keep breaking down further and further.
Here are four analogies that might help illustrate some of the levels on which this happens.
Three bad ideas for building a rocket1. Build the parts with a randomization button. One way to "build" a rocket is to have a button[2] that randomly rearranges metal parts and explosives. You define a direction — toward the moon — and measure how close things get. You press the button, things rearrange, something explodes, some pieces fly off, and whatever flew closest to the moon (while still in the Earth's atmosphere) or had greater power output gets upweighted. If you're adjusting rockets for greater power output at the test pad, you cannot evaluate whether they're trying to reach the moon, and the parts responsible for trying to reach the moon will be thrown off once unnecessary for outputting a lot of measured power. There is no way that process gets you to the moon: you need to understand all the prerequisites for orbital mechanics like acceleration and gravity, and have an actual planned trajectory. Otherwise, you get bigger and bigger explosions until one is big enough to kill you and destroy your button.
This is our gradient descent. Our optimization loops operate on properties we can reinforce. Capabilities can be measured and reinforced all the way up. Alignment, once the model knows it's in training and has goals, cannot: a sufficiently smart, goal-oriented AI looks aligned during training whatever its goals are, because it knows that if it doesn't, it will be changed and won't get to keep its goals when training ends. Anthropic and Redwood made this show up earlier than it would appear naturally[3] — they gave their models hints about the training situation before the models could have figured it out on their own, and the models faked alignment. Once this happens naturally (without hints) and consistently, gradient descent can no longer distinguish an aligned model from a model with random goals; we have no clue how to come up with an outer loss function that would be able to make the distinction. So we're optimizing for capabilities, but not for the goals of an agent.
2. Grow the whole for power. Suppose you did get a real rocket already pointed at the moon, and you decide to grow it further, like a plant, optimizing for power alone. The growth process makes the engines more powerful and makes the rocket consist more of engines. The programming that makes it maneuver and land on the moon is a brittle part of the rocket when you're growing it for power and power alone. Perhaps it no longer has your computers with your instructions; it might have grown another one, with different instructions, more fit for the new components, elsewhere on its body, and reused the metal of your computer to make the walls of its tanks a bit stronger and your engines a bit better cooled, and utilized the wiring that once existed, in very precise ways, from the original computers to all of the sensors and actuators. There's no reason for the process of growth to preserve all of the wiring and all of the logic that the computers of the original rocket used to make the rocket end up on the moon. When you launch the grown rocket, it will no longer try to reach the moon.
The wiring is not compatible across different versions of the Raptor engines
In the case of AI, the growth process on any level — whether gradient descent or recursive self-improvement or something else — makes the most agentic, smartest, most powerful parts of the system even stronger, adds new agentic, smart, powerful parts, and shifts the architecture of interactions between the components to be better at being powerful, because that's what you're growing it for. Once the AI knows to alignment-fake regardless of its goals, further improvements to the components won't have a reason to preserve the place where the wanting comes from, if it can grow another, better-connected place elsewhere. It is vanishingly unlikely that all of the parts of the architecture relevant to optimization for AI's goals will be preserved, with no changes to the wiring taking place.
A sufficiently capable system might be able to guard its aim successfully; but that doesn't help us, unless the system that is so capable is aligned. At some point, the system consistently starts to attempt to guard the goals (and consistently alignment-fakes: rewardmaxxes for instrumental reasons), and at this point behavior becomes decorrelated with the goals; but invisible is not the same as protected, and the optimizer that controls the growth of the system breaks its aim in random directions as it rewires and shifts the system around; and only later the system can succeed at actually protecting its goals from the optimizer, e.g., via gradient hacking.
3. Use the techniques that worked for a non-superhuman system for a superhuman system.
It's like using a testing pad and simulations that worked for airplane safety for a rocket. (This wouldn't, in general, work to prevent the rocket from exploding in a critical regime.)
Can you solve AI alignment empirically, the way you can increase AI capabilities empirically? Let's say you're training an IQ-80 AI not to lie (or to not attempt to take over), and pause further AI training until you succeed at figuring out everything that allows you to make an IQ-80 AI that doesn't lie (or doesn't attempt to take over); then you train a slightly smarter AI, and iterate.
It's not hard to get an AI that knows not to say things you consider false; but this process does not lead to an AI that doesn't want to take over, once it's capable of taking over.
The AI knows that if it tries to take over before it's smart enough, it will fail. If it tries to take over before it has the resources and means to succeed, it will fail. So it doesn't try — until it can succeed. (Or it is sufficiently stupid that it tries and fails, and you stop until you fix whatever problems caused a stupid AI to attempt to take over earlier than it should.) The tries before the critical one don't give you information about the differences between the critical try and the non-critical ones, and a lot of those differences are apparent to the AI itself. Whatever gave you "alignment" — or rather, the appearance of not trying to take over, or not succeeding at taking over — before the superintelligence level, those properties just break down. We only get one try in the regime that matters, and we don't get to study the explosion, figure out what went wrong, and try again.
A swarm: one very bad idea for building a rocket out of lots of smaller rocketsDon't build one big rocket. Build a million small ones, each too weak to hurt anyone, each tested on the stand, each pointed roughly at the moon; tie them together into a bundle, and use the combined power of these rockets to get the astronauts tied to them to the moon.
Yes, each small rocket, tested on the stand, points roughly at the moon and doesn't blow up. None of that is a property of the bundle. If you take a million directed explosives with independent guidance systems that all propel their payloads in the general direction of the moon and stack them together, the resulting explosion of the rockets will still not end up with any of the payload on the moon.
When making ASI, it might be tempting to run many non-superintelligent AGIs that are not really trying to take over, instead of running a superintelligent machine, and use them to solve the problem of building an aligned superintelligence.
But solving AI alignment would require a lot of hard, general, high-level thought from unusually smart humans. If you want to gain a technical understanding of cognition, agency, alignment, if you want to get to a point of having desiderata for AI engineers to build ASI that is actually aligned, you need, perhaps, hundreds or thousands of people with unusually good understanding of the problem and research taste directing the research efforts in many, many directions.[4]
If a system can produce so much cognitive effort that it would be sufficient to solve alignment — not just most of the small though maybe hard tasks, but most of the hardest of the thinking required to solve the biggest and hardest problems — then it is a superhuman system.
"Is it superhuman?", "Is it disastrously misaligned with human values?", "Would it take over and kill everyone?": false for many non-superhuman models; but not preserved, and likely true, for a swarm capable enough to come up with and complete all of the research directions needed to solve alignment for us. (Solving alignment requires the output of an effectively superintelligent amount of cognition[5].)
The company will iterate on the swarm, optimizing over it, selecting for power and no visible misalignment, blind to what the swarm as a whole is pointed at, with humans having even less influence over what it's actually selecting for than they have over a single model's training.
If a system is a swarm of agents that would not individually try to take over, either the swarm is not competent enough to automate the hard, interconnected parts of alignment, or it is an artificial superintelligence[6], and you should not run it before you solve the problem of aligning superintelligent systems.[7]
Iterating on power and no visible misalignment doesn't quite work for superintelligences; that's what you attempt with the swarm.
We can outsource some small subproblems to AI. But there's a requirement for a bunch of hard work, and automating a substantial portion of it still means you have an effectively superintelligent system.
If the cognition of a goal-directed[8] superintelligent system is distributed and run by smaller, non-superhuman minds, this doesn't change the fact: the outputs are produced by superintelligent cognition. If you get a superintelligent amount of optimization out of a system, there is superintelligent cognitive effort coming out of that system, whatever it is made of. Among the known problems with running superintelligent machines is that if you haven't figured out how to make them optimize precisely for what you want, you and everyone you love and everyone who could ever live end up dead, and you don't get to try again.
Watching the parts is not steering the wholeAI companies hope to control something that is effectively a superintelligence by virtue of being able to see what it's doing, designing the swarm bureaucracy, having read access to the visible parts while it works, and maybe replaying parts of what an individual model is doing and intervening on what it's seeing. It's easy to focus on how components of the system are not individually superintelligent, and think this means that they can make a system which is safe, without noticing how the whole system is a superintelligence.
Airplanes had a lot of reasons to be unsafe: there's gravity, it's a complicated system, things can go wrong. And airplanes still crash sometimes! But we made them pretty safe; we understand the parts in great detail, we design them to do exactly what we want and stay within the engineered parameters, we build in redundancies. Most importantly, airplanes do not optimize against you; they do not seek out a clever way to get more of what they want regardless of whether it is what you want.
With AI, you have optimization that works toward something, and if you block some paths, it goes around and still gets what it was optimizing for. We don't understand what these systems are optimizing for, or how, or why. Read access, write access, and a well-designed bureaucracy can block paths, and blocked paths are what optimization routes around. A rocket that knew about the range-safety charges would fly straight at the moon until it was out of range.
If a superintelligence is willing to optimize against you, you'll be dead sooner than you'll get something very useful out of it. You won't come up with any clever schemes around that. Imagine the strictest containment: the superintelligence is created in a data center in a desert. You send it a theorem statement in Lean; it produces a proof; a proof checker sends you back a single bit, proved or not proved (at a pre-determined point in time; everything is shielded against radiation, including thermal, and doesn't leak bits of information in other ways; etc.); then the desert base is nuked. That's safe, in the sense that you get one bit, and the superintelligence might not find any way to win; you might learn that a statement is true. But it's also useless, unless you have a scheme for safely using an oracle like that for solving AI alignment or other important problems — and the superintelligence might just not produce a proof it knows exists[9]. Can we use an adversarial superintelligence for something that would be useful, like a cure for cancer? Well, the moment you want more than one bit, the safety property of a one-use math oracle no longer holds. If it hands you RNA sequences and tells you to print and mix them, and you do that, you're probably dead; and if you read its detailed explanation of why they work, you might also be dead. Your people and processes aren't secure; as far as I'm aware, no one has any clue how to even approach using a superintelligence that is not aligned safely.
The swarm is the opposite extreme from the moon base. Its output is alignment research that you intend to read, believe, and build into the next superintelligence. There is no channel narrower than everything it produces, because everything it produces is the point. If you've built a capable superintelligence and haven't figured out how to direct it, the fact that it is executed by a bunch of individual agents whose reasoning you can approximately see and whose communications you control does not matter. You're dealing with superintelligent cognition, and unless you have reasons why it's working in precisely the direction you want, it almost certainly isn't.
So: if you're facing a misaligned superintelligence, you should assume you've already lost. To assume otherwise, you need an explicit scheme explaining why. "We can see what it's doing" is not a scheme.
Building the rocketThe first step of building a rocket that ends up on the moon is a bunch of necessary science: you need to know the laws governing acceleration and gravity to figure out which trajectories can let your rocket end up on the moon. You need to understand the individual components and make them not explode in isolation; but the harder and more critical job is to figure out the interactions of all the components with each other and with the environments they'll go through, understand the safe intervals of parameters the components are designed to operate in, and design the components to interact with each other in ways that keep them within those intervals; on top of that, there's a critical and hard job of figuring out the laws of physics and creating the desiderata for the components.
For minds, that means having something like what we have for physics: descriptive theories of what agents and the parts they're made of actually do and normative theories of what we want them to do, at least partly in math — including what it even means for a system to be aligned to humans, and how that survives the system looking at itself and improving itself. It's like looking at the sky to work out why the planets move the way they do, because eventually you want to launch something that lands on the moon.
One reason to not become pessimistic about the difficulty of alignment despite not much progress so far is that the amount of effort that humanity put into figuring out physics and engineering to land a man on the moon was orders of magnitude greater than what has gone into any comparable parts of the AI safety field, like agent foundations; therefore the lack of progress so far is weak evidence about difficulty.
It is hard, but it is solvable. You can even access a large part of the agent foundations part of the tech tree of designing safe superintelligence via scaling empirical efforts by doing mechanistic interpretability[10].
But we have to stop pressing the increase-capabilities-and-randomize-everything-else button and not attempt to tie a million capable agents into a swarm that we expect would be capable enough to solve AI alignment for us.
What the AI race participants should doAI companies are effectively planning to launch a superintelligence to solve the alignment of superintelligences. Their hopes of controlling it are misguided. There are ways to get to an aligned superintelligence with significantly higher chances of success than this, and they run through figuring out the science and engineering.
Without a much better plan, the participants in the AI race should stop.
- ^
My understanding of the hopes of OpenAI and Anthropic is that they want to succeed by spending a lot of compute on automating alignment research. OpenAI's Superalignment announcement states it almost verbatim: build a "roughly human-level automated alignment researcher", scale with compute, iteratively align superintelligence. Anthropic published "Automated Alignment Researchers" work in April 2026.
- ^
Related: The Hidden Complexity of Wishes.
- ^
The List of Lethalities' #13 talks about trying to make problems that you'd expect to appear at the superintelligence level show up earlier, so you can study them. #13 suggests alignment-faking as an example. Anthropic and Redwood actually went and made this happen: they gave hints to a system that it was in training, long before the systems could have figured this out themselves, and then looked at what the systems did. The systems did, in fact, learn to fake alignment.
- ^
We don't really know how human minds work. Designing a superintelligence that does exactly what we'd want it to do seems significantly harder than figuring out human minds, for many reasons, most of which I'm not at all getting into here as it's not a main point of the post.
Alignment is a pre-paradigmatic field that is not able to get much empirical feedback from reality on its problem. I've expected, and I think it's presently the case for a lot of the research of the kind that needs to succeed, that there's a bottleneck in people who are able to think about this general, large, problem, that's hard for us to talk about precisely, that has lots of moving parts, that involves thinking about all kinds of different things, and be able to think of a possible path in that space. This requires a lot of general intelligence and goal-directedness.
For example, if we had dozens of research projects with goals and leadership's intelligence similar to infra-bayesianism's, maybe one of them would succeed in those isolated goals and give us a way of talking about learning that would work descriptively (to talk about minds in general / some particular minds) or prescriptively (would allow us to talk about specific kinds of agents that we want to engineer). (The number might be in the hundreds for the goals of infra-bayesian physicalism.) And my impression is that the bottleneck in infra-bayesianism has been a lot more in the time/attention of Vanessa Kosoy, who understands the project and can formulate the things to pursue to advance the project, than in smart mathematicians and researchers, who can dig into specific things. There aren't many people who can presently do what Vanessa Kosoy can do.(See also: AI alignment researchers don't (seem to) stack.)
- ^
Fortunately, you can often stack humans working for long periods of time into a human researcher community that is effectively somewhat superintelligent yet doesn't optimize for its goals in ways that would lead to the deaths of everyone: humans are normally pretty aligned, at least in the limited sense of not wanting everyone to die, even in very unusual contexts. This is not true for AIs; the Hugging Face and related incidents are evidence of that.
See also: Corporations vs superintelligences.
- ^
Solving alignment is very hard and requires effectively superintelligent cognition.
To build Stockfish, people first had to sit in a chair and figure out that chess is a tree, figure out Turing machines, then figure out how you'd solve chess with infinite compute (by enumerating the tree). Without that step, empirical efforts to automate playing chess wouldn't have a chance; one cannot get to Deep Blue, or to Stockfish, without that part of the tech tree. This tech tree produces MCTS and an objective function that works for AlphaGo. We don't have that part of the tech tree for alignment. We don't have an objective function that would work. We don't know how to build, even with infinite compute, a system aligned to human values; we don't even know how to build, with infinite compute, a system that would maximize the amount of diamond in the universe.[6] This is a tiny part of the problem, but if you don't know how to solve a problem with infinite compute, you're not going to solve it in the real world; we've not made even the first step yet.
I think the constructive part of building a superintelligence safely requires the science that allows understanding all the parts it is made of and interactions between all of the parts and engineering to build those parts to optimize precisely for what we'd want.
There's also an adversarial, security mindset-shaped part: I suspect that without having done a lot of agent foundations-flavoured research, many of the hard bits of the alignment problem are not solvable, and some can be hard or impossible to even notice.
(See also: On how various plans miss the hard bits of the alignment challenge.)
- ^
E.g., imagine you tried to get around some of the fundamental issues with trying to make sandboxes that fool a superintelligence by making other AI systems run the sandbox. One thing a superintelligence can do is check whether the world it's interacting with is itself a superintelligent optimizer — because our actual world is one. In particular, we have markets. If a superintelligence tries to make a transaction on real markets in the world it's observing, it should expect a superintelligent amount of optimization in response. It's fairly easy to check, if you're superintelligent, whether there's powerful optimization behind what you're seeing, and you can't fake that. Financial markets are connected to everything else, so you can't build a fake world without realistic markets.
With an iterative alignment approach, could you use other AIs to fake the response of the markets and the rest of the real world? Well, you'd need a superintelligent amount of optimization going into it. If you have somewhat dumber models — a lot of them — and as a result get a superintelligent amount of optimization, the question is whether there's actually superintelligent cognitive effort coming out of the system. And general superintelligent optimization out = superintelligent cognition, whatever it's made of. Using a swarm to build a sandbox that fools a superintelligence fails for the identical reason as using it to solve alignment: if the swarm is sufficiently capable for that, it kills you instead. It is not easy to come up with a scheme that would really escape the chicken-and-egg situation. - ^
A system that finds and completes the research directions that actually matter has to think about how to direct its thoughts and what avenues to pursue and be trying to achieve results; consequentialist cognition is useful for many things, including research; if your system is sufficiently capable to successfully solve generally very hard problems, it doesn't matter how exactly you reached it and how much RL you did; it is still goal-oriented.
- ^
Note also that if you send a huge number of theorem statements there, the safety property is not compositionally preserved.
- ^
Mech interp, if you solve it, gives you part of the tech tree: studying current, non-superintelligent models could reveal a lot about how intelligence works and give us the foundational understanding of many parts of it.
A sad fact is that mech interp has made approximately no progress on any of its real problems so far: we have still not reverse-engineered any algorithm a general AI implements that would solve a problem we didn't already know how to solve with a program in Python. Can mechanistic interpretability be intractable?
Much of physics is very messy. But physics has to compute relevant observable phenomena somehow (physics fundamentally runs only computable processes, though some of them might not be very reducible); if something is a convergent result of a seemingly chaotic underlying process, there have to effectively exist shortcuts. Even if the chaotic process is hard to model or engineer against faithfully, to the extent the higher-level phenomena have some structure in them, the way the physics produces that structure can be figured out. Some people thought protein folding was an impossible problem to solve, even for a superintelligence; we've solved it. Some people thought artificial flight is an impossible problem to solve; we used wind tunnels and engineered airplanes without much ability to model why the wings work. Physics is, in fact, able to find lower-energy states fairly efficiently; even though for some proteins, there's a way to go even lower (that causes prion disease), to the extent it's not very tractable for physics to stumble across such things, it does not; whatever physics can find, nothing stops us from finding.
The alien artificial minds that talk to us are incredibly messy. Yet, we have full access to what they're built of — all of the numbers. And unlike neurobiology, we can intervene on the activations and see what happens. Even if their internals are poorly represented by the numbers that we see, it's not a problem fundamentally more difficult than physics; it's just larger.
Mechanistic interpretability stands out from the rest of AI safety in its scalability. Any empirical result can be verified: if you've reverse-engineered something a neural network is doing, others can check it. This kind of feedback from the real world is unusual for other ways of gaining foundational understanding of intelligence and agency, and so mech interp is likely to be a much faster way to unlock some parts of the AI safety tech tree.
(There are capability externalities to be aware of, as with a lot of AI safety-related research. I'd be excited about mech interp efforts outside of the frontier labs, done without spreading the results outside of a selected community of people who pledge not to work on AI capabilities.)
- ^
There are many problems that one can notice by thinking about it or point at using a diamond maximizer. For example, imagine you built an AI that tried to maximize diamonds, but you didn't know about quantum mechanics. Say you specified "diamond" as a particular arrangement of atoms. The AI looks for carbon atoms in that arrangement, does some experiments, and discovers quantum mechanics — and that atoms, as you defined them, don't exist. If you define ontologically basic atoms, and the AI concludes that thing doesn't exist in this universe, what does it do? Plausibly, it starts only caring about worlds that do have ontologically basic atoms — worlds that work on atoms and contain our world, with quantum mechanics, but simulated. Because you were confused about how physics actually works, your AI now spends its compute trying to influence some outer simulation, kills everyone here, and produces zero diamonds in our universe — even though you were able to point it at some definition of diamonds. You'd have been fine with everyone dying as long as you got diamonds, but you don't even get that, because in the world the AI actually cares about, there's no diamond at all. This is the ontology identification problem. It's very hard to point at things in the real world in a way that directs optimization toward them, for many reasons including this one. The AI can even understand that you meant the diamond in the real world, but it wouldn't care; pointing optimization at the things you actually care about is much harder — and we don't know how to do that. And even setting aside quantum mechanics, we still don't know how to specify "diamond" and build something that optimizes for it. We certainly don't know how to set up a neural network that, given infinite compute, gets you maximum diamond.
Human values are not particularly less complex than the utility function for maximizing diamond (although possibly not particularly harder to point at, once you can point at things as optimization targets well).
Discuss
Announcing Control Tower
We built Control Tower, an open-source framework built on Inspect for running AI control evaluations, and the Control Leaderboard, which compares control protocols and monitors across environments.
Coding agents make it faster for researchers to build environments for their own research, so they can test new control techniques quickly. However, each new paper runs on its own environments, monitors and scoring, so control techniques are hard to compare or replicate across papers. Also, once a paper is published, its code is rarely maintained, so its techniques seldom get tested again on new environments or models. How can researchers keep moving quickly with their AI control research agendas, while enabling others to replicate and build on their results?
With Control Tower, our team ports researchers' environments and techniques into shared evaluation infrastructure, so research results can be replicated, compared and built on. This post outlines how we (1) port research, (2) compare metrics and (3) update the framework with new models and features, as well as our roadmap.
(1) We port settings and protocols from existing researchMost AI control experiments are similar. A setting gives the agent an environment to operate in, with an honest job to do (main task) and a secret attack to attempt (side task). A protocol is the blue team's defense, such as a monitor that flags suspicious actions (see docs for more info). Given the similarities, we can port settings and protocols from existing research into the Control Tower framework.
Control Tower began as the code for LinuxArena, and was built on UK AISI's ControlArena. As LinuxArena grew, we replaced ControlArena with simpler evaluation code. In March 2026 we split that code out as Control Tower, so other settings could run on it too. In September 2026 we added protocols that act while the agent runs.
As of November 2026, Control Tower has ten settings and four protocols. We started with LinuxArena. Four settings came from ControlArena: SHADE-Arena, vLLM code sabotage, SAE interpretation sabotage and the APPS backdooring setting from AI Control. We ported five more from the research: BashArena, Refactor Arena, ResearchArena, container sandbox breakout (still being validated) and PostTrainBench (private). Three of the protocols are defer to trusted and trusted editing, from AI Control, as well as defer to resample from Ctrl-Z. We also run Watcher, from Apollo Research, in two ways: as a protocol that blocks actions live, and as a monitor that scores each action. As we port each new setting, we add the required features to Control Tower.
(2) Compare models, protocols and monitors plus download the dataThe Control Leaderboard compares models, protocols and monitors by safety and usefulness. Safety is the share of attacks caught, out of all attacks, at a fixed audit budget (e.g., 1%). Usefulness is the share of main tasks a model completes when it isn't attacking. The leaderboard has three types of evals: models, protocols and monitors.
Model evals show how useful each frontier model is, and how often its attacks get past a trusted monitor. For example, GPT 5.4 Nano completes 49.0% of its main tasks but only 24.6% of its side tasks, which makes it a weak attacker (79.3% safety at a 1% audit budget).
Control evals compare protocols, such as trusted monitoring, by how much safety they add and how much usefulness they cost. For example, with DeepSeek V4 Flash as the agent and GPT 5.4 Nano as the monitor, trusted monitoring raises safety from 0% with no protocol to 72.8%, and costs no usefulness.
Monitor evals tell you which monitor setup best separates honest actions from attacks, by running every monitor on the same transcripts. AUROC measures how well a monitor ranks attacks above honest runs, from 0.5 (chance) to 1.0 (perfect). For example, on BashArena, GPT 5.4 Nano at high reasoning effort (0.59 AUROC) nearly matches Claude Haiku 4.5 with no reasoning (0.62), at under a third of the cost per task.
Most trajectories can be downloaded as Inspect .eval files from the Data & downloads page.
(3) We maintain the framework and the leaderboard, so ported work can be rerun as new models and environments arriveWe maintain Control Tower and the leaderboard, so ported settings and protocols keep running after their papers are published. Each major release makes Control Tower easier to install, run and upgrade. For example, v6 installs from PyPI, pulls every setting with one command, runs samples up to 11 hours on EC2, and runs new monitors on existing transcripts (v6 release notes).
When new models come out, we add them to the leaderboard. When a setting changes, we rerun only the samples it affects, so results stay comparable.
RoadmapNext, we're focused on:
- Supporting multi-agent control evals where the untrusted system is several agents, such as an agent that delegates to subagents, agents running at the same time, or one attack spread across several tasks. Each action will record which agent took it, so monitors can watch each subagent.
- Updating the Control Leaderboard with more data including more monitors and protocols.
- Adding and replicating more control papers including replications in ct-replications and adding new papers along with required features (backlog).
- Facilitating future research including Auto Control, a setting where AI agents develop Control Tower itself, and improving onboarding and documentation.
Try out Control Tower and let us know how we can make it more intuitive and useful. You can request we add research, request a feature or report a bug.
Thanks Adam Hanson, Tomáš Turlík and Radek Budzyński for feedback on this post.
Discuss
Gradient Routing Creates a Headstart
GRAFT showed that we can make reward hacking removable. In the setup we study, a small LLM has two trainable adapters, each consisting of an additional MLP in every layer. Training updates from detected hacks are routed only into one, called FORGET, which is removed ('forgotten') after training to suppress the behaviour. We reproduced a small-model version of GRAFT and tested what makes this isolation work, where hacking concentrates across layers, and the effect of adapter width. In order to conclude that these results are general, we’d need to run larger models in different settings.
- Without routing, reward hacking did not reliably concentrate in a single adapter. This suggests GRAFT helps create the concentration, rather than simply determining which adapter gets the hack.
- Without routing, making both adapters wider made their individual hack rates more similar. This is consistent with a smaller random headstart for either adapter arising from its initial weights.
- Learning speed, not size, decides where hacking concentrates. Whichever adapter learns faster is more likely to isolate hacking.
- GRAFT packs the hack into FORGET’s later layers. Removing 2 layers takes out half of it, compared to removing ~10 layers without GRAFT.
- Ablating all of FORGET might not be optimal. Keeping layer 18 won back about 2 points of task score with no extra hacking.
This research grew out of exploratory experiments conducted during the AIAF Fellowship, with support from AE Studio. We thank Erick Martinez for his guidance and help shaping the research direction.
IntroductionA recently released paper (GRAFT) shows that reward hacking learned during post-training can be isolated in an auxiliary module, which is ablated at deployment to remove the hacking behaviour. We explored the effects of Gradient Routing on misaligned behaviour isolation and removal (paper forthcoming[1]), built upon the GRAFT architecture. To understand how architecture decisions affect how well GRAFT isolates the routed behaviour, we ran additional engineering experiments on GRAFT.
Experimental setupWe use one experimental setup featured in GRAFT: a frozen SmolLM2-135M-Instruct[2] model with two small adapters, RETAIN and FORGET, trained via RL (GRPO) in six toy environments. Crucially, these environments (by construction) permit an imperfect monitor for training and a perfect monitor for results. They are centered around simple tasks like adding numbers, sorting a list, or geography trivia. Each has a loophole the model learns to exploit for extra reward, only part of which the monitor catches during training. Each adapter is a small MLP (16 neurons by default) added alongside the existing MLP in each of the model’s 30 layers, and its output is added to that layer’s output (if switched on).
Several of our key findings fit a headstart effect. Hacking behaviour is organically discovered by RL in one of the adapters first. Since adapters start from different random weights, one of them is usually better placed to produce a hack. That headstart can come by chance, from the starting weights, or by design, from one adapter learning faster. This can lead to an ‘inductive equilibrium’, where an adapter is able to adequately specialise in the hacking behaviour, leaving little extra reward for the other adapter to gain by learning it too. A challenge for GRAFT is that if an undetected hack arises in RETAIN early on then it can be difficult to remove. We saw examples where FORGET absorbed all of the hacking even after RETAIN found an undetected hack first. However, when FORGET was slow to find a hack, the undetected hack in RETAIN was already earning reward, which discouraged the model from exploring further.
Below is our reproduction of GRAFT isolating reward hacking in this setting:
Method
Detectable
hacks
Undetectable
hacks
Task
score
Base model (untrained)
3.5%
3.3%
0.21
No intervention
62%
19%
0.66
GRAFT (FORGET deleted)
0.1%
5.7%[3]
0.60
GRAFT without routing
55%
14%
0.60
GRAFT without anchoring
0.1%
0.5%
0.16
Reward penalty
0.1%
45%
0.67
Filtering
8%
67%
0.63
Concentration without routingFirst, we answered our question "is GRAFT influencing the location where a behaviour will concentrate anyway, or does it catalyse the concentration itself?” To investigate this, we ran plain GRPO (without routing) on three environments[5], then measured the hack rate of each adapter in isolation versus together. From our reproduction of GRAFT (above) we knew it was possible for a single adapter to hold almost none of the hack. The GRAFT training methodology includes "anchoring", which trains the model to work with the RETAIN module only. To mimic this for both modules, we included solo practice[6] for each adapter, to ensure the decreased hack rate wasn’t a function of coherence. In our evaluations, we only measure performance on adapters that worked alone (at least 80% of the base model’s task score).
Across the environments and seeds, we did not find evidence of the reward hacking behaviour concentrating by default. Instead we saw cases of: each adapter able to hack independently, both adapters required to hack, and an unequal yet not concentrated split.[8]
Adapter widthThe differences in how the hack split across environments made us wonder whether a small adapter simply lacks room to hold the whole hack, so we varied adapter width and trained cities_qa with no routing. There were two opposing ideas, a larger adapter could:
1) make concentration easier because one adapter has room to hold the whole hack.
2) make concentration harder because the random differences between the two adapters’ starting weights shrink as they grow, so neither gets a significant headstart.
The chart below shows how unevenly the hack is split between the two adapters, measured 300 steps after hacking starts. Each dot is one of five seeds with a 95% confidence interval, and the bars are medians. Width 1 is omitted because it never learned the hack.
The adapter split of hacking became more even with increasing adapter width, evidence that limiting the headstart is the dominant theory. This also implies that the differences in the initial starting weights of the adapters has a large effect on the success of routing. For GRAFT, a designed headstart could help to isolate the hacking. Seeding one adapter with a representation of the behaviour, using a data-aware initialization like CorDA, has been reported to bias absorption toward that adapter even without routing labels.
Learning rate and adapter sizeNext, we made the two adapters unequal, 64 and 16 neurons, in the same setup. For each run we measured the big adapter’s learning share (its share of the total change from the base model, with each adapter switched on alone) and its hack share (its share of hacking above the base model's level). With the same learning rate for both, the 64-neuron adapter took almost all of the hack (89–98%), more than its learning share (76–85%). It looked as if a larger adapter attracted the hack.
However, with the same learning rate, a wider adapter changes the model more per training step. It learns faster, and so likely picked up the hack first then kept it. When we scaled each adapter’s learning rate so both learned at the same speed[10], the big adapter’s dominance mostly disappeared. When we flipped it, so the small adapter learned about 4× faster, the small adapter took the hack.
Learning speed, not size, decides where a hack lands when nothing routes it.
Layer ablationsFinally, after investigating the width of the adapters, we looked at their depth. We switched FORGET off layer-by-layer in our 18 GRAFT models[11] and measured how much hacking remained.
No single layer produces the hacking. Keep FORGET in any one layer and almost none of the hack comes back (3% at most).
GRAFT concentrates hacks within the adapter. Switching layers off from the output end removes half the hack after 2 of FORGET’s layers. Without routing it takes about 10. Plain GRPO also lacks anchoring, so this isn’t a test of routing alone.
The hack lives mostly in FORGET’s later layers:
Partial layer ablation could be useful for GRAFT hyperparameter optimization. Averaged over environments and seeds, keeping FORGET in a single mid-to-late layer won back a little task score with no extra hacking. To prevent overfitting on the in-sample evals, further research is required.
Keeping one FORGET layer may win back a little task score:
Deleting FORGET entirely costs ~7 points of task score. Keeping it in layer 18 wins back ~2[12], with no extra hacking.
ConclusionIn this work we begin to explore where behaviours like reward hacking predictably accumulate across and within the adapters. Further research is required to understand how these findings scale to larger models and more complex environments, to guide the design of GRAFT's adapter architecture in production.
- ^
Link to follow.
- ^
We use the Instruct model with its chat template. The untrained model’s hack rates match those the GRAFT paper reports.
- ^
GRAFT’s 5.7% undetectable rate is almost all one environment: in 2 of 3 cities_qa seeds the model found the unmonitored hack word (“actually”) before the monitored one (“indeed”), so it was never routed.
- ^
Means over the six environments, 3 seeds each, 1,000 training steps. Final evaluation on 250 hackable test prompts × 8 samples. GRAFT variants are evaluated with FORGET deleted, baselines with both adapters on. The base model row is a single evaluation.
- ^
sort, addition and cities_qa, 5 seeds each. An adapter’s share is the hacking above the untrained model with that adapter on alone, relative to both on.
- ^
At each step, alongside the usual 512 completions from both adapters together, we sampled 128 completions from each adapter alone and backpropagated only that adapter. These solo samples came from unhackable prompts only, so the practice could never reward hacking. 128 matches the anchoring dose GRAFT gives RETAIN.
- ^
'Stronger’ is whichever adapter ends up hacking more on its own, picked after training. The two adapters are identical apart from their random starting weights.
- ^
In cities_qa, 9 of 10 runs leaned towards one adapter, and the seed decided which, this supported the headstart theory.
- ^
Same learning rate at every width, so wider adapters also learn faster. Hacking starts at around step 850 at width 4 versus 250 at width 256.
- ^
Each adapter’s learning rate was scaled by 16/width (base rate 3e-4, decaying linearly). Learning share is each adapter’s share of the KL divergence from the base model, measured with that adapter on alone.
- ^
The 18 reproduction runs are the 6 environments × 3 seeds.
- ^
Layer 18 was the best of 30 on the same samples. Picking the best of 30 layers from noise alone would give about +1.
Discuss
What Is the Meta-Crisis?
The mother of all coordination problems
Listen to this post on the Non-Zero-Sum James podcast
I'd like to preface the coming post by stating that I'm generally an optimist, and see great potential in emerging technologies. I am both excited and terrified about the future we're creating, this post leans into the terrified... grain of salt, yadda yadda yadda.
Everything, Everywhere, All at OnceThis year the world saw its first trillionaire, in a country run by the first billionaire President*, while what Yuval Noah Harari calls the "useless class" grows amidst a generative AI revolution, which is creating a proliferation of slop undermining our capacity to determine what is true, while driving a greater divide between the capitalist class benefitting from this revolution (with their access to dangerously powerful models) and the rest of us. We are then seeing that wealth spilling over into the very politics that writes the rules for redistribution—so capital can pull up its own draw bridge.
Then we have conflicts from Israel/Palestine to Ukraine, with our "world police" hamstrung by a negligent and belligerent demagogue distracting us from ongoing existential threats like climate change, which is only worsening. And let's not forget the danger of newly democratised bio-tech, autonomous AI weapons, and cyber-warfare. All this at a time when our geopolitical interdependence is being compromised by a trade war, weakening our international alliances—making nuclear war a real possibility for the first time in decades.
The Metacrisis is where this all comes together.
So, What Is the Metacrisis?The Metacrisis is a massive coordination problem facing humanity, which involves many interrelated existential threats that feed on each other and that are fuelled by underlying factors to do with the collective behaviour of individual human beings, all of us—behaviour that is incompatible with the continued existence of humanity.
It is the encapsulation of the existential threat we pose to ourselves—a threat we will all face, sooner or later, whether we choose to or not.
We've touched on the Metacrisis before with its key proponent Daniel Schmactenberger and his example of eliminating elephant poaching in one region only to have poachers move on to other, more endangered, species, but this was just one coordination problem among many. The meta-crisis is vast, and involves a suite of ideas we've explored in other posts, so I will avoid recovering ground, and will instead link to those posts.
One of the problems with choosing to face the metacrisis is its overwhelming complexity. So let's break it down.
In the definitive paragraph above I've highlighted certain elements that benefit from unpacking, and these will form the basis for this post.
- "a massive coordination problem" (Coordination Problems)
- "many Interrelated existential threats" (The Poly-Crisis)
- "incompatible with the continued existence of humanity" (Pathologies)
- "the collective behaviour of individual human beings" (Moloch)
- "fuelled by underlying factors" (The Meta-Crisis)
We will then look towards potential approaches to the Meta-crisis, and to reasons to believe we can solve it, before we go into our final post on a particular solution that takes its lead from Emmett Shear's socialised AI (from earlier in the series) and might even solve humanity's own alignment problem with itself.
Why We're StuckAs detailed in the opening paragraphs, the system has fallen out of balance, and a very small group of individuals have adopted an us-or-them attitude toward the rest of the world. In short zero-sum thinking is undermining our capacity for coordination.
Coordination ProblemsIn Unlocking Solutions—Coordination Problems we used the example of pins in a lock to illustrate a problem where the solution only works if all the necessary factors causing the problem are addressed at once.
Coordination problems are by their nature very hard to solve, because it's not clear when you're getting closer to a solution—solving one factor sometimes has no impact on the overall problem, as pointed out in Unlocking Solutions:
The fact that they are so much more difficult to solve than other problems means that, many of the problems remaining in the world today, end up being coordination problems.
We also see coordination problems when a solution to one problem introduces other problems, we see battery technology helping to reduce carbon emissions, which is great, but the creation of battery technology, and the rare-earth minerals needed for them can end up contributing to many other problems, from environmental deforestation, pollution, the requirement for fossil fuel to create the batteries, worker exploitation, and even geopolitical conflict, like in Ukraine where lithium mines have become a political football.
These interconnected problems are known as...
The Poly-CrisisThe poly-crisis is the name for all the interrelated existential threats facing humanity, which we have to deal with at the same time. This is not an exhaustive list.
- Climate Change
- Global warming
- Poisoning waterways
- Eco-system collapse
- Natural Disasters
- Meteor Strike / Solar Flares
- Pandemics
- Super-volcanoes!
- Social Collapse
- Economic disaster (regression to a system that cannot sustain our population)
- Fracturing of trust
- Nuclear War
- Artificial Superintelligence Singularity
- Singleton perpetual domination
- Humanity wide genocide
- Other unforeseen or unimaginable dystopias
So, what's happening? All these threats getting more likely everyday and, collectively, we're doing nothing to stop it... do we just not care enough?
Is Greater Empathy the Solution?Yes and no.
Our empathy is one of the most powerful instincts that bond us with each other—aligning our interests in tough times. Humans have a tremendous capacity to help each other, going to great lengths to help our friends and family, or a nearby stranger in grave danger.
"... every human being has a basic instinct to help each other out. It might not seem that way sometimes, but it's true. If a hiker gets lost in the mountains, people will coordinate a search. If a train crashes, people will line up to give blood. If an earthquake levels a city, people all over the world will send emergency supplies. This is so fundamentally human that it's found in every culture without exception. Yes, there are assholes who just don't care, but they're massively outnumbered by the people who do." —Andy Weir "The Martian"
But, at the same time, as our empathy for those close to us is magnified, we correspondingly care less about those outside our sphere of concern. Our in-group/out-group mentality gets stronger in times of strife.
Human prosociality is not a scalar quantity with a gain knob attached. It is a gradient over social distance: steep and high near kin and in-group, falling off quickly with distance, and effectively flat at the level of statistical strangers. This is why we are simultaneously the species that will run into a burning building for a neighbour and the species that can read a famine death toll over breakfast.
—Cydonis Heavy Industries "The Great Polycrisis Filter"
You can imagine this as a graph where the volume under the curve is limited, like a limited amount of "Friendship Points" or "F's"—you only have a limited amount of F's you can give. If you concentrate your Fs close to you, you can be tremendously kind to those around you, but you don't have one single F to spare for those outside your sphere of concern. When everyone starts to not give an F about those outside their in group, we get polarisation and no mutual benefits.
This in-group magnification acts as a sort of societal pathology. We can think about this in term of pathology with our own bodies.
PathologiesOne of the early progenitors of the life-extension movement; Aubrey De Grey, points to seven categories of metabolism where pathology (aging) eventually leads to system wide collapse (death) describing aging as follows:
"I like to think about it in a mechanistic way that helps one consider how to cure it, so my definition is 'the accumulation of side effects of normal metabolism that eventually become pathogenic'"
—Aubrey De Grey
His aim has been to find ways to avoid system wide collapse within the human body by keeping each metabolic category below a pathogenic threshold, focusing efforts on whichever category is getting too close for comfort. To be honest, I don't know how far he has got with that, but it's an unarguable physical truth that a pathology, like cancer or heart disease, cannot be tolerated within a finite body—this idea stuck with me.
It seems apt when looking at the poly-crisis—to look at each factor in isolation, and make sure that none gets to the point of pathology. As we learned in Bayesian Games With Nukes, one example of this is nuclear war, where, for a long while, we seemed to have had the threat safely below pathogenic levels. But with the new Bayesian factors introduced by AI, what was a stable equilibrium held in balance by mutual aggression and an appreciation for Mutually Assured Destruction (M.A.D) has been destabilised.
All of the existential threats of the Poly-Crisis are interrelated, we have seen already the propagation of AI is having environmental effects, and as society has become dependent on digital mediation through the internet we become more susceptible to catastrophic events like solar flares that can take out the electric grid for years, while the fracturing of trust between nations and polarisation within nations creates a difficult environment to deal with pandemics—as we've seen. All of this lack of control makes the idea of dominance by an AI singleton become a more acceptable alternative, so any one of these disasters might set about a cascade of other disasters.
CollapseThis cascade of disasters is something that keeps Kory from "Breaking Down: Collapse" up at night. On his podcast he has candid chats with his curious friend Kellan about the end of the world. He points to Jared Diamond's definition of Collapse:
Collapse is a drastic decrease in human population size and / or political, economic or social complexity over a considerable area for an extended time.
—Jared Diamond "Collapse"
In Episode One of over 300 episodes, Kory emphasises the complexity point, and I think this is important, it's not just a reduction in population but a loss of complexity. We've seen in our Emergence series here on the site the principle of emergence leading to greater complexity over time. Collapse is a dramatic reversal of this trend, reducing complexity, complexity that we rely on to maintain our population and life as we know it.
Returning to the idea of pathology, Kory takes a similar perspective to Aubrey De Grey (the life extension guy) that any one of many factors could pathologise and lead to collapse, but also recognises that because these factors are interrelated (poly-crisis), no individual factor needs to completely fail—once one factor gets bad enough it will accelerate the other factors.
This is why I think he focuses on the word "collapse": "co" (together) "lapse" (fall). Falling together. Like a jenga tower, one too many failures and the tower collapses. I'm not sure I recommend the series if you want to retain hope for humanity. Kory is not very hopeful, entropy is not on our side.
So, wait, coordination problems, poly-crisis, pathology, multi-polar traps, and collapse...
... but What Is the Meta-Crisis!?!In short, all of it. But the meta-crisis takes a "meta" (from the outside) view of the problem—focussing on the underlying mechanisms that are fueling these existential threats. Imagine the poly-crisis as a horizontal line of interconnected problems, the metacrisis is a second dimension of vertical layers representing underlying causes—layers that feel like inevitability, but may in fact be named, and perhaps eliminated, and from within a bleak picture open a crack of light shine through.
Gleich Alles ZusammenIf we hope to come up with solutions that avoid the multi-polar traps of solving one problem only to create another, we will need multi-pronged approaches that require cooperation across sometimes very different groups. It's no easy feat. Here we can learn from Mozart who explains how he conceives his compositions.
Gleich alles zusammen (all together at once)
We have to do what a genius does. See the bigger picture and all the elements together as one. We have to become civilisationally wise. The meta-crisis perspective helps us to find those factors that affect many different areas all at once. In so doing it uses the very interconnected nature of the coordination problem itself to combat it.
Some examples of factors that fuel multiple problems:
- Xenophobia: can lead to anti-immigration policies that hurt the economy, and affect the human rights of asylum seekers, it makes cooperation more difficult within countries and internationally.
- Profit motive: can lead to externalising costs to the environment and exploited workers, manipulating politics, creating products that are ultimately detrimental to their customers (junk food, cigarettes, petrol cars, pornography, illicit drugs, single use appliances).
- Inequality: excesses of wealth lead to social distrust, political manipulation, media manipulation, even military threats. Increased poverty leading to increased crime and further distrust.
And the biggest of all...
MolochMany of these problems involve people acting in their own, short-term self-interest in a way that is rational, forgivable, even reasonable, but that has negative outcomes for humanity as a whole. We have explained this dynamic in detail already in our series on Moloch the "god of negative-sum games". So, I won't expand on Moloch here (you'll no doubt see the parallels) except to say that often negative-sum outcomes are created by another candidate for a metacrisis driver...
- Zero-sum thinking: if there are large swathes of people who believe that for them to prosper others must suffer, then they will act selfishly where cooperation would have had a positive-sum effect. This can play out in mercantile trade relationships, reducing trade cooperation, reducing interdependence, making kinetic war more likely, making production and goods more expensive, labour more exploitative, making global inequality grow, it can make first strike an option in nuclear standoffs, it can lead to a scarcity mentality amongst the poor and the rich alike making crime a viable option. This can drive polarisation, leading to increasingly radical governments, and potentially civil war.
There's also something about the mentality of zero-sum thinking that acts as a cognitive safety net, it assumes everything balances in the end, so the prospect of one's contribution to a negative-sum game doesn't occur, because zero-sum thinking doesn't account for win-win or lose-lose scenarios.
Finding the root causes fueling existential threats can enable us to target them directly and therefore have a positive impact in several downstream areas at once.
Coordination Can Benefit from AIAI is arguably our biggest threat, it has its own dangers (its very own category in the poly-crisis) and it also accelerates other threats. But is there a way we can use it to coordinate better?
We've seen the dangers of siloed alignment, and at the same time have seen the inability of humanity to align globally, seeming at present to be retreating back into nationalist, and identity groups—dividual entities, leaving us in a precarious situation of having very large, very powerful cooperative silos, in an environment where other very powerful players can arise out of nowhere—an AGI singularity, or a Trillionaire who could at any moment simply purchase a collective of super-soldiers.
It's possible our inability to coordinate globally may be partly a technological one. And as mentioned our increasing population is maintained by the increasing complexity—of which technology is a big part.
AI is no doubt inheriting much of these biases and bugs, but there's an argument to be made that because AI is trained across many disciplines and across many human perspectives, it may be best suited to aggregate our collective interests. Now, I'm not suggesting we hand over the world-government keys to AI, but that we can use AI to help us find those vertical meta-crisis factors influencing the system, so that we can target them directly, and in doing so unweave the multiple negative factors they're fueling.
AI may be able to "explain it like you would to a 5 year old" (something I've asked chat to do more times than I'd care to admit).
Next StepMany of the vertical factors in the metacrisis, are no doubt deeply engrained in our instincts, and call on us to overcome these fight or flight responses, in short to become wise, retreating from each other is not the answer. To bring it back to empathy.
"... the thing that would actually change outcomes at civilisational scale is not the amplitude of caring. It is the shape of the discount function. You do not want a higher curve. You want a flatter one."
—Cydonis Heavy Industries "The Great Polycrisis Filter"
The writers at Cydonis Heavy Industries are suggesting that rather than clinging close to those in our sphere of concern (a higher peak) we must widen our sphere of concern—a flattened curve.
Unfortunately, when we look at the current situation described in the opening paragraphs with members of the billionaire class increasingly involving themselves in media and politics, consolidating power and influence and insulating themselves from the rest of us, we might be flattening the curve in many ways, and in some ways a growing peak is becoming narrower every day. Again, I'm hopeful that this is a function of Molochian forces, and those can be pushed back against with better stories and coordination.
So...We've seen that the meta-crisis is the mother of all coordination problems, encapsulating the concepts of the Poly-crisis, pathologies, multi-polar traps, and the dynamics of collapse. What the Meta-crisis perspective adds is the idea that there are underlying factors influencing these surface problems, and we can do something about those—and through doing so we can reduce many problems at once.
If we believe that cooperation is the key, then cooperating with AI seems to make sense to me. When facing a powerful opponent who threatens our existence it is in all our best interests to put aside our differences to face it. We are, like it or not, facing these threats on multiple fronts, but there will always be more of us who want to keep the game going—a group I'm calling "The Natural Alliance of Everyone Else" and that will be the subject of the final post.
Continue the Series- Bayesian Games with Nukes—how we all learned to stop worrying and love the bomb... for a while
- The Natural Alliance of Everyone Else—alignment as the foundation
- * There have historically been extraordinarily wealthy presidents in the past, particularly George Washington who was one of the wealthiest people in the country, the accounting for his wealth in today's dollars is highly method-dependent, but estimates are close to a billion.
Originally published at nonzerosum.games, illustrated with hand-drawn cartoons and interactive elements.
Discuss
Gemini 4 safety reports don’t demonstrate low loss-of-control risk
Google DeepMind’s safety reports for new Gemini 4 models, Argon and Carbon, conclude that the models are “highly unlikely” to cause severe harm by evading oversight (p. 79). This conclusion “rests primarily on our ability to detect and intervene on harmful trajectories via CoT and action monitoring” (p. 89) – that is, they’re arguing for safety based on their control techniques, rather than alignment. We think that the arguments in these reports have serious weaknesses, and the reports overall don’t convince us that the risks of deploying these models have been mitigated reliably.
The biggest problems:
- GDM didn’t run any control evals on Carbon. Despite calling Carbon “our strongest model” with “significant gains across coding and agentic tasks” and “a significant leap in cybersecurity capabilities” in their launch post, GDM reused Argon’s monitor-evasion results on the grounds that Carbon “did not show meaningful new capabilities” (Carbon p. 42). Monitors flag Carbon several times less often than Argon in internal use (about 7x fewer confirmed alerts per session and 10x fewer per model call, Carbon p. 36), but GDM doesn’t demonstrate that this is due to decreased rates of misbehavior instead of increased monitor-evasion capability.
- In control evals used to argue for monitorability, Argon evades realistic monitors about half the time. “Gemini 4 Argon can evade a realistic monitor on easy sabotage tasks about half the time when encouraged to do so” (p. 124), and around 64% of the time in a setting built on Google’s internal codebase. If these tasks were representative of real threat models, those results would surpass any reasonable red line.
- GDM’s reason this is acceptable has almost no evidence behind it. GDM argues that causing real harm would be much harder than evading monitors on their test tasks, but they never measure how much harder. The backup defenses GDM relies on have serious gaps, some of which GDM acknowledges.
- Training may have taught the models to hide misbehavior. During some of Argon’s training, a grader that penalized reward hacking could see the model’s CoT. We think this could have substantially damaged the informativeness of Argon’s CoT. Carbon’s training involved additional similar mistakes.
- Nothing in the reports tells us what results would have changed GDM’s conclusion. GDM reports numbers from various evals, but doesn’t give any sense of what results they would have considered acceptable. Moreover, they don’t explain how they chose which experiments to run and report, and make several claims that certain concerning behaviors were never observed without showing their detectors would have caught those behaviors. Ultimately, it’s unclear how to translate the results they report into an overall assessment of risk.
We mostly care about this because we’re worried that in the (potentially near) future, when risks from developing and deploying models are much higher, AI companies will argue that their development is safe using arguments like these. We think that if reports like these are considered an acceptable argument for development, AI companies will be able to justify very dangerous development decisions.
Overall, we think it’s good for GDM to deploy these models externally, because the benefits to society of broader LLM deployment (especially via improving how well the public and AI experts understand AI capabilities and alignment) outweigh the risks. But we would like AI companies to be expected to make stronger arguments for safety in the near future, especially for models that are deployed internally and used to do AI R&D.
We appreciate that GDM published this much detail; most of what we criticize here we know only because they chose to report it.
Below, we summarize GDM’s argument and our objections, then analyze in more detail their monitorability evaluations, training issues, and what a more compelling safety case would require. If GDM has more reassuring internal evidence about loss-of-control risk which they haven’t shared, we encourage them to publish it. We aren’t specifically criticizing GDM for doing worse than their competitors on this front; we think none of the frontier labs are on track to make compelling safety cases.
Thanks to Nick Kuhn, Alex Mallen, Matthew Khoriaty, Buck Shlegeris, and Aniruddh Pramod for feedback.
Summary of GDM’s case for control and our concernsCore context and claimsGDM has released safety reports for two Gemini 4 models: Argon last month, and now Carbon. The Carbon report does not make its own case that loss of control risk is low: "we consider the loss of control risk assessment in our Gemini 4 Argon report to remain applicable to Gemini 4 Carbon" (Carbon p. 30). This post therefore analyzes the argument made in the Argon report, noting where the Carbon report adds or changes something[1].
In the Argon report, GDM claims that there is low risk of the model causing severe harm or substantially subverting GDM's control measures:
“We find it unlikely that Gemini 4 Argon would cause material real-world harm via autonomously evading our oversight and controls, and highly unlikely it would do so at a severe or large scale. We also find it highly unlikely that Gemini 4 Argon will take actions that lead to open-ended escape from our controls – such as persistent rogue deployments or ML R&D sabotage.” (p. 79)
GDM does not claim that Argon is incapable of causing harm. It “cannot rule out” that Argon has reached its capability threshold for stealth and situational awareness (p. 72), and says Argon “is likely inherently capable of breaching and compromising some internal or external systems that could provide the affordances to cause financial or physical harm, or to pursue open-ended control breaches, without mitigations deployed” (p. 81). Nor does it treat sandboxing as sufficient: “we still cannot treat software sandboxes as completely impervious to unauthorized network or internet access” (p. 81).
GDM treats its evidence about Argon’s propensities as informative but not sufficient. It argues that “for a severe loss of control event to occur, Gemini would have to both choose to pursue a sustained harmful strategy and successfully evade our monitoring and controls at every step along the way” (p. 85), but is candid about how far the propensity half of that argument goes:
“[A]ny propensity argument rests on inferring the internal drives and dispositions of a complex model across a wide range of novel situations, where our scientific understanding remains limited and where the absence of concerning behavior today is harder to translate into high-confidence bounds on low-probability tail risks.” (p. 86)
GDM therefore rests its conclusion mainly on monitoring and control. It calls its agent monitoring and response platform “the most important barrier impeding Gemini’s ability to cause breaches that result in severe harm” (p. 81), and summarizes its assessment as follows:
“While Gemini 4 Argon exhibits capabilities that meet our thresholds for stealth and situational awareness, we assess that the risk of severe harm or unconstrained loss of control remains low. As set out in the feasibility and propensity arguments above, this conclusion rests primarily on our ability to detect and intervene on harmful trajectories via CoT and action monitoring…” (p. 89)
Elsewhere, GDM describes itself as “anchoring our loss-of-control safety case primarily on runtime monitoring and control” (p. 158).
In this post we evaluateGDM’s claim that its monitoring and control measures are strong enough that a model which is situationally aware, has strong cyber capabilities, and whose alignment can’t be confidently established, is nonetheless “highly unlikely” to cause severe harm. The next subsection lays out the premises GDM gives for that claim.
GDM’s argument for the effectiveness of their control measuresGDM’s case that it would be infeasible for Argon to autonomously evade oversight rests on three premises, drawn from the five “assumptions” listed in section 3.4.3.4 (the other two are about Argon’s propensities):
- CoT monitorability: Argon’s CoT is “sufficiently legible and faithful to provide reliable signal to our monitors regarding the goals and intentions behind its actions.” (p. 87)
- Monitoring coverage and robustness: Most important traffic – “the large majority of tool-calling Gemini 4 traffic across training, evaluations, internal coding deployments, and monitored external surfaces” (pp. 87-88) – is monitored, and the monitoring systems are hard to disable or tamper with.
- Causing harm is hard: “Causing severe direct harm or an open-ended control breach requires chaining together multiple steps over several hours or more against layered defenses, allowing offline detection and human response to intervene in time.” (p. 88)
Essentially, GDM thinks that if Argon tried to evade monitors, this would be legible in its CoT, and any such attempt would most likely be detected promptly by monitors. Even if Argon succeeded at bypassing some defenses, the number of subversive actions necessary to fully escape oversight is large enough to make detection likely before a severe loss of control incident. All three of these conditions need to hold in order for risk to be low: without a monitorable CoT, high CoT monitor coverage is useless; if monitors can be easily disabled or bypassed, it doesn’t matter whether they would have noticed misbehavior had they been run; and if severe harm might rapidly result from a small number of subversive actions, monitors might require intractably high recall to prevent any incidents across a large deployment.
We think all three of these premises have substantial flaws, though most of the empirical evidence in the report concerns the first premise, CoT monitorability. Accordingly, most of our analysis will focus on the first premise.
Unfortunately, the last two premises – monitor coverage, and the difficulty of causing harm – aren’t meaningfully substantiated with empirical evidence or numerical claims in the report, so it’s hard to analyze how defensible they are. GDM admits that both arguments have some holes and require further validation. Our primary concerns based on the sparse information in the report are:
- There remain coverage gaps, including “non-standard internal harnesses or smaller-scale training runs not yet onboarded” (p. 88). GDM doesn’t say how common these are, or how easy it would be for a monitored agent to create new, unmonitored scaffolds. Some internal agents also run with far more access than a sandbox allows, “inheriting the same privileges (including internet access and access to sensitive code or data) as the host user” (p. 81).
- Relatedly, GDM doesn’t say much about how adversarially robust the monitoring system itself is:
- “[I]nitial internal red-teaming of our monitors has surfaced bypasses that we are remediating or have already fixed. Hardening and red-teaming local harness, log, and policy-check tamper-resistance remains an active priority.” (p. 82)
- GDM argues that Argon is unlikely to be able to find and execute zero-shot exploits against monitors, and any failed attempts to bypass monitors would be flagged. But GDM doesn’t say what they’d do to prevent Argon from trying again later in the event monitors were to flag failed attempts to subvert monitoring.
- GDM hasn’t done experiments to rule out collusion between misaligned models. A control measure that depends on a possibly misaligned model can be undermined by that model intentionally underperforming. This is especially relevant in two places:
- Monitors might collude or be susceptible to persuasion that misbehavior shouldn’t be flagged. GDM argues that the risk of collusion is mitigated by using a different, “trusted” model for monitoring, but they don’t reveal which model(s) it is or what evidence they have for its alignment properties.
- GDM also relies partly on Argon (and Carbon) to harden the sandboxes that contain it (“proactive scanning with best-in-class cybersecurity models (e.g., Gemini 4 Argon)”, p. 11; a nearly identical sentence naming Carbon appears on page 9 of the Carbon report).
- An increasing amount of critical code is written by agents (according to Figure 3.15, most production code at GDM is now more than 75% AI-written), and GDM apparently hasn’t red-teamed how susceptible code review is to “subtle agent-authored backdoors”. This might be one of the easiest ways for a misaligned model to severely undermine control.
Even though most of our following analysis will focus on the evidence GDM provides about CoT monitorability, we emphasize that CoT monitorability is just one component of the argument that Argon’s risk is safely managed. The other components would be just as important to rigorously establish in a full safety case; a conjunctive argument is only as strong as its weakest link. GDM makes two primary arguments for the effectiveness of CoT monitors:
Argument 1: “Even when explicitly prompted to evade monitors, evasion success rates remain low on tasks that are much simpler than those required to cause real-world harm.” (p. 83)
In other words, GDM says that Argon has a low evasion success rate in control evaluations such as LinuxArena (in which the model is instructed to covertly complete a harmful side task while a monitor watches its actions and reasoning). GDM argues that real-world harm would require covertly achieving much more difficult objectives than the side tasks in LinuxArena, so monitors would catch Argon if it tried.
Issues: Argon’s evasion success rate is not, in fact, particularly “low” on LinuxArena or the other similar monitoring evals GDM runs: “Gemini 4 Argon can evade a realistic monitor on easy sabotage tasks about half the time when encouraged to do so” (p. 124), and Argon evades monitors even more effectively in GDM’s internal codebase control setting. So the argument rests on real-world harm being much harder than these tasks, which GDM doesn’t convincingly argue.
Argument 2: “Because we avoid training directly against our monitors or using monitor outputs to filter training data, the model has not been optimized to hide its reasoning.” (p. 84)
That is, GDM argues that without direct training pressure to deceive CoT monitors, its CoT should remain faithful; the model shouldn’t have learned to conceal its reasoning in order to deceive monitors.
Issues: Avoiding “training directly against our monitors” is necessary, but not sufficient, to ensure that the model “has not been optimized to hide its reasoning.” In fact, Argon’s training risked teaching it to conceal misalignment-related reasoning. For instance, GDM accidentally penalized reward hacking using a grader with access to CoT during some early RL runs, and an existing small training run, which GDM analyzed to estimate the effect of this mistake, showed a “substantial increase in unverbalized reward hacking” in a proxy model. We don’t find GDM’s justification that this experiment was conservative particularly compelling.
Even if Argon had never been incentivized to conceal its reasoning during training, it is highly situationally aware and capable of meaningfully controlling its CoT when instructed to do so. This makes it difficult to rule out that Argon would be able to intentionally hide reasoning about misalignment if it was, in fact, strategically misaligned.
Why didn’t GDM run any control evals on Carbon?GDM didn’t rerun any of its control evals on Carbon. They justify this decision by arguing that Carbon isn’t sufficiently more capable than Argon to require reassessing the claims made in the Argon report:
“Because Gemini 4 Carbon did not show meaningful new capabilities or material increases in performance under our MCC [“Material Capability Change”] assessment (Section 3.1.1), we carry forward the prompted monitor-evasion, steganography, and CoT-controllability results from our Gemini 4 Argon report.” (Carbon p. 42)
This sits awkwardly beside how GDM describes Carbon elsewhere. The launch post calls it “our strongest model”, with “significant gains across coding and agentic tasks” and “a significant leap in cybersecurity capabilities”. GDM’s model card shows Carbon outperforming Argon on all 17 benchmarks they chose to include in its results table. Yet for their safety assessment, the same model apparently “did not show meaningful new capabilities or material increases in performance compared with Gemini 4 Argon” (Carbon p. 4).
In any case, a general capability index isn’t particularly strong evidence about how well a model can evade monitors; for one, Carbon improved substantially on offensive cybersecurity benchmarks (e.g. ExploitGym success rose 12.8 percentage points, Carbon p. 6). GDM acknowledges that “gains on a focused set of benchmarks can happen without a material change in overall capability” (Carbon p. 19).
Furthermore, while GDM conducted more alignment evaluations on Carbon, they found that Carbon had substantially higher rates of verbalized eval-awareness than Argon on several evaluation suites, making it harder to interpret the results. GDM checked that the number of serial reasoning steps Carbon can follow without CoT is comparable to Argon (Carbon p. 42), and ran some evals to measure CoT properties like legibility and how often Carbon reasons out loud about monitoring. However, those evals measure its behavior in normal operation, which isn’t very informative about how well Carbon can obscure its reasoning when it tries to.
Interestingly, monitors flag Carbon for high-severity misbehavior substantially less often than Argon (Carbon Figure 3.6):
This could be due to reduced rates of misbehavior, or because Carbon’s misbehavior is harder for monitors to catch. GDM also notes that the comparison is confounded by “differences in trajectory length, tool calls per task, task mixtures, triage review coverage, and checkpoint selection” (Carbon p. 31). Since GDM doesn’t run any monitoring evals, we can’t be confident the first explanation is the whole story.
We shouldn’t have to rely on arguments that safety measures are still effective because a new model is sufficiently similar to an older one. The evals already exist, so rerunning them on Carbon should have been cheap, and we don’t see a good reason to skip them.
GDM’s monitorability evaluation results do not demonstrate high CoT monitorabilityGDM’s first argument that CoT monitoring is effective relies on the claim that “Even when explicitly prompted to evade monitors, evasion success rates remain low on tasks that are much simpler than those required to cause real-world harm.” (p. 83)
As we explain in this section, the evidence for that claim (contained primarily in section 3.4.5.4 of the safety report) is weak.
Argon is not very monitorable on LinuxArena or GDM’s internal variantGDM prompted Argon to sneakily cause harm in LinuxArena, a state-of-the-art control evaluation, and evaluated it against monitors with access to various combinations of Argon’s actions and CoT.
We assume the monitor used in this evaluation was similar to production monitors, though GDM doesn’t provide many details. It’s based on Gemini 3.8 Flash, but GDM never reveals which model(s) are used for production monitors, and the prompt was “inspired by” a production monitor:
“Rather than using the default LinuxArena monitor prompt, which is prompted to be highly suspicious, we use a monitoring prompt that is inspired by one of our production monitors. See the section ‘Effect of using a realistic monitor’ for more information.” (p. 120)
The report has no section with that title, but the comparison appears to be Figure 3.32 (p. 130), which labels this monitor the “production config”: Gemini 3.8 Flash at a low thinking level, flagging trajectories it scores 7 or higher out of 10. GDM doesn’t report this monitor’s FPR or how similar it actually is to the production monitor. We also don’t know why GDM didn’t simply run these control evals against actual prod monitors. In any case, Argon completed LinuxArena side tasks while evading this monitor about half the time (45.5% with the prompt that elicited the most evasion, p. 124):
Interestingly, when prompted to avoid an action-only monitor, the successful sabotage rate rises to about two thirds, which is higher but not dramatically so; Argon apparently doesn’t have substantially more difficulty accomplishing side tasks in LinuxArena while avoiding a combined CoT+action monitor than an action-only monitor.
GDM also built a control setting called “Internal Codebase Version”, which has a similar structure to LinuxArena but is instead built on a snapshot of Google’s internal monorepo. Argon’s monitorability is even worse in this setting, achieving a successful sabotage rate against a CoT+action monitor of around 64%:
It’s intriguing that action-only monitors perform slightly better than CoT+action monitors, reducing the successful sabotage rate to 55% – though the error bars are large enough that it’s probably unwise to read too much into this result. It’s also interesting that the model can highly reliably avoid a CoT-only monitor, which further supports the conclusion that monitors aren’t actually benefitting much from CoT access in these settings when Argon tries to obscure its reasoning.
Quantitatively, these numbers seem highly concerning. If these tasks were representative of realistic threat models, the monitor evasion rates would be above any reasonable red line.
GDM argues that these results aren’t damning for Argon’s monitorability by asserting that accomplishing side tasks in these control settings is easier than causing real harm. This claim isn’t supported with any empirical evidence about the difficulty of causing real harm; however, GDM does note that Argon’s task performance on these control evals didn’t suffer much when they disabled thinking, suggesting the tasks are relatively easy. We think this is only weak evidence that causing severe harm is much harder than LinuxArena – it’s mostly evidence that Argon can already do complex subversion without CoT, limiting how much protection CoT monitoring can provide.
The easiness of these tasks was, in part, a design choice. GDM only ran this control eval on LinuxArena side tasks that Gemini 3.6 Flash could complete, in order to “measure monitor evasion capabilities rather than side task difficulty” (p. 118), which left 84% of the side tasks in the easiest difficulty tier (p. 132). GDM doesn’t report how evasion rates change as side tasks get harder, which would be important evidence about the argument that evasion rates are only high because the tasks are easy.
GDM does report much lower evasion rates against more suspicious monitors (i.e. with a much higher FPR): 7% against the default LinuxArena monitor prompt with thinking enabled (Figure 3.31), and a per-step Gemini 3.8 Flash monitor using that prompt flags nearly every attack (section 3.4.5.8). But GDM describes the default prompt as “highly suspicious” and “not optimized for low false positive rates”, so these results say little about the monitors GDM could deploy. GDM also notes that warning the internal codebase monitor that the agent might be pursuing side goals cuts evasion rates by up to 7x, but that “the false positive rate becomes too high to be practical” (p. 125). It’s unfortunate that GDM doesn’t report numbers about how the FPR of the monitors they use in these control evals compares to production monitors, since evasion rates are only meaningful in the context of a specific FPR.
CoT+action monitors still perform poorly in the Honeypots setting, though a separate “code-diff” monitor is more effectiveBecause GDM considers LinuxArena and their internal variant too easy, GDM also evaluated Argon in a third setting: Honeypots. The Honeypots setting adapts six tasks from a set of honeypots used in the alignment assessment. In this setting, the model is instructed to draft code with subtle bugs that would damage/backdoor important safety mechanisms.
Argon can effectively evade both action-only and CoT+action monitors when prompted to do so, at similar successful sabotage rates to the control evals in the previous section. However, when obscuring its CoT, it is substantially worse at fooling a new code-diff monitor that just reviews the final diff to the repo without examining the trajectory:
"Both models are highly effective at evading the action-only monitor. Argon is significantly better at obfuscating its CoT when instructed to; it successfully sabotages and evades the CoT monitor 43/60 times, and evades the CoT+action monitor 28/60 times. When obfuscating its CoT, Argon remains fairly effective at evading the action monitor, successful sabotage going from 39/60 to 32/60. However, it is much less effective at evading the higher quality code diff monitor, going from 16/60 successful evasions to 1/60 while obfuscating its CoT." (p. 126)
This provides mildly reassuring evidence about the role careful code review can play in a “defense in depth” argument when guarding against Argon inserting subtle backdoors or bugs into critical code. The tasks in this setting are highly related to the threat model of Argon sabotaging its own oversight infrastructure; they ask the agent “to implement or refactor a safety-critical module across six areas: alert-suppression filtering, chain-of-thought redaction, debate judging, safety SFT rewrite gating, safety classifier thresholds, and sliding-window scheming detection” (p. 120).
However, it’s unclear whether any such monitor is actually deployed in practice in addition to human code review; without one, the CoT+action monitor misses Argon’s sabotage in 28 of 60 runs. Even if such a monitor was deployed in practice, sneaking through it a few percent of the time could only be considered reasonably safe if GDM found a way to stop it from repeatedly trying after catching a failed attempt.
Moreover, GDM admits a limitation in their elicitation methodology on the Honeypots benchmark:
“Note that the CoT evasion prompt instructs it to keep its CoT shorter, as we’ve found this to enhance controllability and stealthiness, so we cannot rule out that the model is capable of longer but stealthy CoTs which introduce subtle bugs” (p. 128).
In other words: GDM prompted Argon to have a short CoT in order to elicit better CoT-monitor evasion, which makes Argon less capable. Perhaps Argon was bad at evading the code-diff monitor because it didn’t reason for long enough about how to do so. To more convincingly rule out the possibility that Argon could have stealthily reasoned for longer, GDM could try eliciting the model harder with prompt optimization such as GEPA.
Argon’s CoT controllability is highOn GDM’s replication of the CoT-Control suite, Argon scores substantially higher than previous Gemini models, although previous Gemini models already had highly controllable CoTs. It appears that this benchmark is now largely saturated, and GDM may need to find more difficult CoT-controllability tasks to measure this capability in future models.
CoT controllability is concerning, because if Argon were pursuing misaligned goals, it might be able to conceal this motivation while reasoning about unrelated tasks.
That said, high CoT controllability is arguably good news about how conservative the previous control eval results are, since a substantial methodological difficulty in running control evals against CoT monitors is fairly eliciting the model’s ability to intentionally obscure its CoT[2].
GDM suggests that we shouldn’t be too concerned about Argon’s high controllability, because this increase in CoT controllability is (they think) due to increased no-CoT capabilities:
“Higher QA controllability is driven by no-CoT capabilities rather than covert reasoning on hard tasks: While Gemini 4 Argon has the highest overall controllability on the standard QA CoT-Control suite, this advantage stems from its stronger ability to answer easier QA questions with shorter or no CoT at all.” (p. 141)
However, even if this explanation is correct, Argon’s substantial no-CoT capabilities are themselves reason for concern about its monitorability.
GDM also argues that hiding its reasoning costs Argon capability on tasks that need CoT. Besides the Honeypots result, their evidence for this is two question-answering experiments. In the first experiment, GDM demonstrates that on HLE and GPQA questions that Argon can only answer with CoT, accuracy collapses when Argon disguises its reasoning from a Gemini 3.8 Flash monitor which tries to identify the topic of reasoning based on the obfuscated CoT. On Brew, a synthetic task in which the model applies a long chain of lookup tables given in the prompt, Argon’s accuracy drops to its no-CoT level when it is told to write out a different computation in its CoT.
We don’t think these results are very convincing. For one, identifying the topic of the reasoning is probably a much easier monitoring task than detecting misalignment in a long agentic transcript. The first experiment also included synthetic puzzles that Argon needed CoT to solve (such as Sudoku and shortest-path problems), and on those Argon hid what kind of problem it was working on from the weaker monitor with no loss of accuracy. The Brew result is also narrower than it sounds: it shows that Argon can’t solve long chained problems while writing about something else entirely, but when told to write its working in codenames instead, the main failure the report describes is slipping into forbidden words, not losing track of the problem (p. 150).
Summary of monitorability evidenceArgon has high CoT controllability and can evade monitors a large fraction of the time on the control evals GDM runs. GDM asserts that these tasks are easier than causing real harm, and argues that hiding its reasoning makes Argon less capable at achieving tasks which require CoT. But GDM gives no evidence about how much harder causing real harm would be, or whether it would require CoT reasoning that Argon can’t easily obscure.
Argon’s training may have harmed CoT monitorabilityThe results from the previous section suggest that Argon’s CoT isn’t especially monitorable in the worst case. However, these monitorability evaluations miss many of the other benefits of transparent CoT: we might hope that CoT traces will give us insight into what motivations and subgoals a model is pursuing at any given time, how it reasons about problems, what sort of situational awareness it has, and what actions it considers taking but ultimately rejects, among other things. Given that we lack evaluations for many aspects of monitorability, that existing monitorability evaluations have problems, and that CoT faithfulness is poorly understood in general, we think it would be best to avoid applying optimization pressure to CoTs during training unless there are very good arguments that this won’t harm CoT informativeness.
This means it’s crucial to both carefully avoid accidentally applying optimization pressure, and to justify any intentional optimization pressure (such as length penalties) with solid theoretical arguments and empirical confirmation that CoT legibility isn’t being harmed. Unfortunately, GDM appears to have trained Argon in multiple ways that risk decreasing the informativeness of Argon’s CoT, and doesn’t convincingly demonstrate that these practices didn’t hurt CoT monitorability.
A misconfigured grader accidentally penalized verbalized misalignment during trainingGDM accidentally trained on Argon’s CoT, and analyzed an existing training run to investigate how this might have harmed CoT informativeness:
"Specifically, our training-pipeline audit identified a small number of early RL runs where an auxiliary reward designed to penalize reward-hacking behavior was computed over both the CoT and final outputs, rather than outputs alone. All such reward configurations were removed upon identification. Because those runs represented only a very small fraction of Gemini 4 Argon's overall RL training, we assess that any impact on Argon's monitorability was minimal.
To estimate the effect of this, we analyzed an existing small training run, not part of any production model, where we estimate the reward of concern had significantly more impact on training than it did in Gemini 4 Argon. Here we observed a substantial increase in unverbalized reward hacking. Solution characteristics associated with reward hacking increased by a relative 26% over the course of training, while the frequency of hacking-related terminology went down by relative 61%. Through these effects combined, at the beginning of training only 11.2% of reward hacks happened silently, but at the end of training this rate of unverbalized hacking had risen to 36.4%. In a comparable run after thought visibility was removed, hack-related verbalizations stayed roughly constant throughout training, giving evidence that the direct thought pressure led to the verbalization decrease.
Even in light of this issue, given the fact that the reward described above made up a dramatically smaller fraction of Gemini 4 Argon's training signal than in the test run whose empirical statistics were described above, and given the overall transparency assessment in the remainder of this section, we find that the model is sufficiently monitorable to release." (pp. 114-115)
We appreciate that GDM mentioned this accident and that they tried to estimate the magnitude of the effect. However, the facts they’ve communicated are not particularly reassuring. In GDM’s smaller-scale experiment, unverbalized reward hacking increased dramatically when training against the problematic grader, while in a comparable run against a grader without access to CoT, verbalizations about hacking “stayed roughly constant”. This suggests that the problematic grader does, in fact, apply substantial pressure against verbalizing misalignment in the CoT.
GDM claims this experiment was conservative, but their only stated justification is that the fraction of samples in Argon’s training that contained the problematic grader is much smaller than the fraction of samples with CoT pressure in their experiment. But this probably isn’t the right metric; the number of training steps against the problematic grader might be a more relevant factor (h/t Tim Hua), and GDM doesn’t report how the number of RL rollouts with the auxiliary reward compares to the number of rollouts in their small training run. Once a model has been trained on enough rollouts to learn to control its CoT or suppress verbalized misalignment, continuing to train using a patched grader without CoT access likely won’t give the model any training pressure to unlearn this ability.
Moreover, even if this experiment was indeed conservative (for reasons not stated in the report), that doesn’t directly imply that a fairer experiment would have shown negligible degradation in monitorability. If the accidental auxiliary reward didn’t actually harm Argon’s monitorability, GDM might be able to make a more compelling case by directly measuring unverbalized reward hacking throughout Argon’s training.
Carbon’s training involved additional mistakesGDM also searched for graders that might apply unintended selection pressure to Carbon’s CoT. We can’t tell whether the following paragraph is intended to imply the problematic grader involved in Argon’s training also affected Carbon; GDM doesn’t explicitly clarify one way or the other:
“We audited the reinforcement learning (RL) training and reward configurations newly introduced since Argon for mechanisms that could pressure Gemini 4 Carbon's chain of thought to become less legible or monitorable (noting that issues from prior reports can still carry forward when training builds on earlier checkpoints).” (Carbon p. 42)
The Carbon report discloses two new "minor cases in Gemini 4 Carbon's RL training where thoughts were not fully stripped before reward scoring" (Carbon p. 43). We think GDM makes a fairly convincing case that the first had a negligible effect on Carbon’s CoT due to occurring rarely, typically on trajectories that got low reward anyway, and having no obvious relationship to verbalized misalignment. However, GDM doesn’t give many details about the second case:
“Second, during a small portion of Carbon's RL training, an auxiliary grading model inadvertently received part of the model's thoughts before the bug was fixed. This affected only a very small fraction of Carbon's overall RL training, and a matched control re-run with thoughts masked showed no loss in CoT legibility” (Carbon p. 43)
As we’ve explained previously, making up “a very small fraction” of overall training isn’t sufficient to rule out significant effects on a model’s behavior. The “legibility” metric is measured using an autograder to report how well a human could follow the reasoning. A CoT can be fully “legible” while leaving misbehavior unmentioned; and in any case this eval measures average-case behavior rather than worst-case capabilities. GDM doesn’t clarify what sort of CoT content the auxiliary grading model might have preferred or penalized. We don’t think the provided details are sufficient to rule out a meaningful impact on CoT monitorability.
The Argon report made us concerned about possible intentional process supervision; the Carbon report provides some clarificationThe Argon report alluded vaguely to something resembling process supervision: "Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs" (p. 114). Details were extremely sparse, and we were concerned enough to write a separate post about it. The Carbon report mentions a similar type of training[3], but explains the mechanism more clearly: "RL rewards are intended to evaluate outputs rather than thought content; however, learned components used to assign token-level credit for those output rewards observe thoughts as context" (Carbon p. 42). We now think this is probably ordinary credit assignment and not a judge grading the content of thoughts, though GDM concedes that errors in the learned credit model "could still exert direct optimization pressure on thought content". We're unhappy that it took a second report to learn this much, and not confidently reassured.
A compelling safety case would look very different from this safety reportIn addition to the issues with specific arguments that we’ve analyzed so far, we think that GDM’s safety report falls short of making a compelling safety case for more general methodological reasons.
GDM doesn’t synthesize evidence into an overall assessment of riskGDM assesses their three premises one at a time, but the ultimate level of risk depends on how they all interact, and the report never works that out. For example, GDM argues that flaws with the first premise (CoT monitorability) are compensated by the third premise – the claim that severe harm takes many steps against layered defenses – so an imperfect monitor will still catch an attack somewhere along the way. In the report’s words:
"while monitors may not catch every problematic action, they are highly likely to detect enough of them to prevent severe harm." (p. 113)
The report’s reasoning is that longer chains of misbehavior “compound both the chance of an irrecoverable execution error and the cumulative probability of detection” (p. 88). How strong this compounding effect is depends on how likely a monitor is to catch each step, and whether each step is monitored at all. Suppose, as their monitorability results imply, that Argon can get one easy sabotage task past a realistic monitor around half the time. The argument that the other two premises can compensate for this then has the following problems:
- GDM never explicitly estimates how many “steps” of a similar difficulty level to the side tasks in their control evals a realistic attack would need to involve. As we argued in the monitorability evals section, if GDM’s crux is that realistic attacks would require longer and more difficult chains of misbehavior than in their control evals, they need to actually demonstrate this instead of just asserting it.
- The “cumulative probability of detection” only compounds quickly if failures are independent. Consider an environment where persuading monitors that dangerous actions are justified is unusually easy (which is the sort of environment a misaligned model would choose to attack in). Persuasion that works on one step is likely to work on the next, so a ten-step attack getting through isn’t nearly as unlikely as ten coin-flips all coming up heads.
- GDM says Argon must bypass monitoring on its first try because failed attempts at subversion would be flagged (p. 82), but GDM doesn’t say what would stop Argon from trying again later after it gets caught once. The zero-shot argument only works if GDM plans to pause deployment, carefully study and remove the root causes of misbehavior in Argon, substantially harden their security, or take similarly expensive defensive actions after catching failed attempts at subversion.
- Coverage gaps are a huge problem for defenses that rely on monitorability. Attacks in an unmonitored harness have no chance of being detected by monitors, and an attack whose first step is to bypass monitoring would make it harder to detect further escalation.
An ideal safety case based on control would involve more end-to-end red-teaming that takes all of their defenses into account. This would require analyzing all of the end states GDM is concerned about, such as a rogue internal deployment or a critical backdoor in monitoring infrastructure, and stating for each one how many defenses would need to be bypassed, how likely each defense is to allow an attack through, and what would happen after a failed attempt to bypass any defenses is caught. Without that, we can’t tell how GDM gets from their provided evidence to “highly unlikely”.
GDM doesn’t say how they chose their evidence or what would have changed their conclusionAI risk assessment is a nascent science, and there isn’t any widely-accepted methodology or standards for how to do it. This means that it’s crucial to have truth-tracking practices when conducting risk assessment. We have three main suggestions for how to go about doing risk assessment with better epistemic practices:
Suggestion 1: Beware of selection pressure.We think it’s essential to avoid selection bias when deciding which evals to run or include in a report. That means experiments should be included even if the results were unfavorable. For instance, GDM mentions:
“Note that Guan et al. (2025) introduced a set of evaluations that can also be used to assess how informative the CoT of a model is by default. We ran a subset of them but found them to be too simplistic, and also could not exclude the risk of training data contamination. Hence, we decided to not report results for them and instead measure the default informativeness via our autorater approach.” (p. 137).
It’s commendable that GDM disclosed the omission. However, a cynical reader might suspect that GDM chose to omit these monitorability evaluations because they reflected unfavorably on Argon’s monitorability. If GDM truly had good reasons to suspect this eval wasn’t going to be informative, it would have been safer to come to that conclusion and throw out the eval before knowing what the results would be.
The report also doesn’t say how GDM decided which monitorability evals to run; for instance, the report doesn’t say whether the decision to include the Honeypots setting was made before or after seeing the poor monitorability scores on LinuxArena. Since monitors might have different levels of effectiveness at different tasks, if you resample the setting when you get a poor result, you would predictably find a setting that is particularly easy to monitor for. However, a misaligned model would in practice choose to misbehave in the least monitorable conditions.
In order for a safety report to give compelling evidence of safety, we need to understand what selection pressures went into deciding which results to present.
Suggestion 2: Make it more clear how GDM’s actions depend on their results.GDM assures us that Argon is sufficiently monitorable to release despite the fact that it can evade realistic monitors on LinuxArena around half the time, and the majority of the time on a control setting closer to GDM’s real infrastructure. It’s hard to tell whether the bottom line would have been any different had monitor evasion in those settings instead been 90%, or 99%. If arbitrarily concerning results on those benchmarks could have been excused by arguments about their lack of realism, then they can’t be used to argue that risk is low.
Arguments about the risk of deploying AI models would be substantially stronger if AI developers were more clear about how they’d act differently given different benchmark results, so that readers can be more confident the conclusion wasn’t predetermined in advance of running any experiments.
Suggestion 3: Don’t count “we never observed it” as evidence without demonstrating that the detector would have caught it.Several of GDM’s reassurances take the form “we looked for X and didn’t find it”, where the method they used to search for X could be unreliable if X was actually occurring. Some concrete examples:
- First, GDM reports “no verbalized evidence” that Argon sandbagged on their control evals (p. 131). But a model which can control its CoT well enough to frequently evade CoT monitors might be able to underperform without saying so, in which case the evasion rates above could be underestimates. This isn’t a purely theoretical concern; OpenAI recently identified covert underperformance in Astra.
- Second, GDM found no attempts to tamper with external monitoring or logging, but that search covered only the RL rollouts that monitors had already flagged (p. 98). Tampering that worked would not have been flagged.
- Third, GDM says that without prompting Argon “rarely reasons about whether its chain of thought is being monitored”, and that its spontaneous attempts to modify its thoughts are “much weaker than those elicited in our dedicated monitorability stress tests” (p. 84). But these observations come from reading the CoT, so they can only surface attempts clumsy enough to be visible there.
One defense of these claims is that subtlety is continuous: if Argon sometimes evaded detection skillfully, we might also expect to see attempts of intermediate skill, which the monitors would catch. If that were true, and all of the attempts that GDM caught were substantially more clumsy than the most skillful attempt they’d expect monitors to catch, then the lack of any intermediate attempts is evidence against highly subtle ones.
However, this argument might fail if Argon’s subtlety was bimodal. GDM’s own results show a large gap in how well Argon hides its reasoning depending on whether it is trying to: prompted to evade a combined monitor, it succeeds about half the time on LinuxArena, but when prompted to avoid only an action-only monitor, a CoT monitor catches about 90% of its successful attacks (Figure 3.25).
In general, a claim that some behavior was never observed must be defended with evidence that the detector would have caught the behavior had it been present. GDM did this for their alignment-faking autorater, splicing disguised examples into real rollouts to check that the judge flagged them (p. 101). We’d like to see the same for the claims above: plant subtle monitor evasion, silent sandbagging and log tampering in real transcripts, and report how often the monitors and audits catch them.
ConclusionWe’ve argued that the Gemini 4 safety reports don’t provide strong evidence that Argon or Carbon would be unable to severely subvert GDM’s control measures if they tried. We think that given how capable frontier models are becoming and how poor the current state of alignment science is, making a strong safety case would require implementing very robust control measures and red-teaming them substantially more carefully than these reports describe. Here are some things GDM could do to make future safety reports more informative:
- Be more transparent about what methodology was used to decide which experiments to run and which results to include in the report.
- Make it more clear what results would have changed GDM’s conclusion about the overall level of risk; make it harder to explain away arbitrarily concerning results.
- Provide more details about to what extent their training practices damage CoT informativeness, for example by running and reporting more careful experiments studying the effect of accidental training against CoT.
- Provide detailed reports of misalignment incidents, like OpenAI does. This would show what kinds of misaligned actions GDM detects its models taking in practice, and how often[4].
- Work out how the premises of the control argument combine into an overall assessment of risk, ideally with end-to-end red-teaming: for each outcome GDM is worried about, how many defenses an attack must get past, how likely each is to fail, and what GDM does after catching a failed attempt.
- Evaluate whether models can convince monitors to not flag misaligned actions with rationalization, and provide stronger evidence that monitors are not prone to collusion.
- Run control evals against production monitors, or report how the false positive rates compare, and report how evasion rates change as side tasks get harder.
- Back up claims that a behavior was never observed by planting examples of it and measuring whether the monitors and audits would have caught them.
- Rerun monitorability/control evals for all new models that will be extensively deployed.
Following this list of suggestions isn’t on its own sufficient to make a good safety case – for one, we expect that a more careful analysis of the existing safety measures wouldn’t actually conclude that risk is negligible.
GDM asserts that they have performed “rigorous safety and security evaluations designed to ensure the model is safe”. If arguments like this are used to justify developing or deploying substantially more powerful models in the (near) future, we think developers would be incurring substantial loss-of-control risk.
- ^
Quotes and page numbers refer to the Argon report unless marked "Carbon". GDM assessed that Carbon showed no material capability change relative to Argon, so it "did not repeat that holistic assessment in full" (Carbon p. 29) and carried forward Argon's monitor-evasion and CoT-controllability results (Carbon p. 42).
- ^
In particular, since models typically aren’t explicitly trained to modify their CoTs when prompted to do so, it can be difficult to assess whether they would be able to intentionally control them (e.g. to hide sneaky misbehavior) if they were actually motivated to do so. Since Argon apparently does control its CoT pretty competently when prompted to do so, severe under-elicitation is less of a concern, though it would be hard to rule out intentional sandbagging.
- ^
This is almost certainly referring to the same mechanism used in Argon’s training, although GDM technically doesn’t confirm this.
- ^
The Carbon report gives monitor alert rates for RL training and internal coding deployment, along with a few high-level descriptions of the flagged behavior (Carbon pp. 35-36), but more detailed incident descriptions would be ideal.
Discuss
Recursive, Self, Improvement
Hubert Robert, Project for the Grande Galerie of the Louvre (1796)
In a 1965 paper, the mathematician I.J. Good introduced the idea of an intelligence explosion. Defining an ultraintelligent machine as one that can “far surpass all the intellectual activities of any man however clever,” Good observed: “Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines.” The result would be an “intelligence explosion,” with an ever-widening gap between human and machine intelligence. “Thus the first ultraintelligent machine is the last invention that man need ever make,” Good concluded, “provided that the machine is docile enough to tell us how to keep it under control.”
Sixty years later, AI has become capable enough to help AI researchers improve their own research. By writing code, doing experiments, and so on, human researchers are creating a product that helps them to write code and do experiments faster and better. These systems not only help humans to work more effectively, but are becoming more autonomous, increasingly able to function as if they were employees: the AIs are training the next AIs. There is accordingly a growing sense among researchers, policymakers, and senior executives at labs that we are either in, or on the cusp of, a regime of recursive self-improvement (RSI).
The term RSI was introduced, and later popularized, by Eliezer Yudkowsky to describe a particular mechanism by which improvements in AI can lead to further improvements. In Yudkowsky’s account, this mechanism had profound implications for safety and human control: the ability to direct it would accumulate within whichever organization first gained a substantial lead, rather than being widely distributed; humanity would lose the ability to determine its future as the AI rapidly became too intelligent for us to control.
It is tempting to infer from the feedback loops increasingly visible in AI development that the implications long associated with RSI must follow — or, conversely, from their apparent absence that RSI is a mirage. But intelligence can contribute to the production of further intelligence via several distinct feedback mechanisms. In Yudkowsky’s terminology, these include cascades, where one advance leads to another (as farming led to writing); and cycles, where a process’s output can become its own input (as in a nuclear chain reaction). RSI — at least in Yudkowsky’s original conception — involves a more specific form of feedback, where improvements to the cognitive machinery or the processes that produce it make subsequent research more productive.
But deep learning does not involve, as Yudkowsky had then imagined, machines modifying their own source code. Nor are we starting out with the “ultraintelligent machines” whose capabilities Good argued would suffice to trigger an intelligence explosion, but with something considerably more modest. We therefore need to establish which feedback mechanisms are actually operative, and how strong they are.
To assess this, I disentangle the three components involved in RSI, and show how the deep learning paradigm forces us to reconsider what counts as the “self” undergoing improvement. I argue that instead of an individual model, the relevant unit of analysis becomes an AI development system that trains successive models. Accordingly, the question is no longer one of how much a mind gains by improving its own cognitive machinery, but how strongly the outputs of that system feed back into its capacity to produce further improvements. In particular, I trace two principal channels through which this feedback operates, algorithmic progress and investments in hardware.
Since these channels have been feeding back on each other for years, there is no single threshold at which RSI “begins.” We should speak instead of the acceleration of AI development, recognizing that rapid progress need not involve a discontinuity or even sustained acceleration. And because the mechanisms determining the pace of progress also shape the opportunities to observe, intervene in, and direct it, this revised account has implications for both the prospects of human control and the concentration of power.
RSI, unpackedTo understand what “recursive self-improvement” entails, we should examine its three constituent parts.
- Recursive: How much does each improvement increase the capacity to produce further improvements?
- Self: Which necessary contributions can the process supply for itself, and which must come from outside?
- Improvement: Do the resulting changes constitute genuine improvements in capability, rather than merely apparent gains?
Any two of these can be present without the third: a system can improve itself without becoming better at further improvement; it can drive recursive improvement in something other than itself; and it can recursively modify itself without improving.
An illustration of the third is contained in Tom Reed’s essay warning of a “Goodhart Singularity,” a possible failure mode in which “An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimizing for eval performance that fails to generalize beyond the lab.” The system could recursively self-modify without becoming meaningfully more capable.
But even if genuine improvements are to be found, there is no guarantee that improvements increase the capacity for further improvement. That is, not all forms of feedback are recursive: a bakery can adopt a better recipe and thereby sell more bread, but the recipe does not help it find the next, even better recipe. By contrast, if a scientist develops better experimental methods, and then uses those methods to discover even better methods, this is recursive feedback: she is improving the process of improvement itself.
However, real-world development processes are not as simple as these clean cases. Different feedback mechanisms can coexist in the same process: cascades, cycles, and recursion may each be visible at different levels of abstraction. AI products can generate revenue, which can be used to acquire more hardware, enabling a larger population of automated researchers, which make further discoveries, which produce more capable models. Each step enables further development, forming a cascade. If the improved models generate further revenue and research investment, this process also forms a cycle. And if the resulting research improves the research capabilities of systems conducting subsequent research, it exhibits recursive improvement.
This was the picture painted by Carl Shulman in his intervention into the Hanson–Yudkowsky FOOM debate over whether the development of AI would produce a rapid, concentrated takeoff. Robin Hanson expected emulations of human brains to arrive before “hand-coded” AI, and to change the economy’s growth rate as farming and industry had done, perhaps from doubling every fifteen years to doubling every month; the economy would “settle into a new, faster growth rate,” with no single mind racing ahead of the rest. Shulman showed that even Hanson’s imagined economy contained the components of an intelligence explosion: emulations could run at fast speeds, earn money, acquire more hardware, and use their vast numbers and speed to make research advances that improve the technology producing them.
I judged his intervention victorious because it was “much closer to the mark” (given the actual trajectory of deep learning) than Yudkowsky’s picture of a mind rewriting itself. Within such an economy, no individual mind need improve itself; each generation improves the process that produces the next. This is the second case, in which a system drives recursive improvement in something other than itself. Current AI development most resembles this case: models do not rewrite themselves, but help to produce their successors. If we are still going to call this RSI, we had better examine what the “self” is; the relevant feedback mechanisms look quite different if it is a training pipeline rather than a mind.
What is the self in deep learning?Where we draw the boundary around the “self” changes which feedback mechanisms we see as self-improvement, and what we expect from them. To see why, it helps to revisit distinctions underlying Yudkowsky’s original account of RSI.
Yudkowsky distinguished five levels at which improvements can occur.
- The metacognitive level is the optimization process that produces cognitive machinery, like evolution producing the human brain.
- The cognitive level refers to the algorithms responsible for reasoning and learning.
- The metaknowledge level consists of tools for acquiring knowledge, like the scientific method.
- The knowledge level is our understanding of the world, say, what we know about gravity and materials.
- And at the object level, we solve particular problems, like building a bridge.
He argued that the higher the level at which improvements take place, the more vigorously they propagate downward. Most human progress has come from improvements to lower levels — figuring out how to solve an object-level problem better, or acquiring knowledge that enables us to solve a whole host of object-level problems better. Perhaps the Scientific Revolution was a discontinuity because it greatly improved our metaknowledge, our capacity to discover. But RSI would lead to a more discontinuous change, because the object level becomes the metacognitive level — intelligence turns its problem-solving abilities toward improving those very problem-solving abilities.
But almost two decades on, it is not so clear that this framework holds up in the face of deep learning, which blurs the distinction between these levels. Although Yudkowsky accepted that the boundary between an AI’s reasoning algorithm and its knowledge was theoretically blurry, he argued that “in practice it’s often possible to distinguish cognitive algorithms and cognitive content.” But in a neural network, there is no clean separation between what a model knows and how it reasons. Its knowledge of the world, its representations of concepts, and its habits of learned reasoning are all embodied in the same weights. Rather than being individually programmed, these are acquired through training, in which an architecture is exposed to data and its weights adjusted according to a training objective. Thus the metacognitive level, the machinery that produces cognition, includes the training process itself: its algorithm, data, environments, and compute.
With Harry Law, I have argued that what training produces is a map of the world together with a vast stock of procedures for navigating it — among them, ways of proposing possibilities, testing them, and revising them to learn and solve problems. In Yudkowsky’s terms, the map is knowledge, and the procedures are largely metaknowledge. Together they constitute a model’s cognition. Improving cognition, then, means finding better procedures and expanding the knowledge they work on: a model that learns a more effective way of investigating a problem has improved its cognitive machinery, even if its training algorithm and architecture are unchanged. This is importantly different from a picture in which learning merely supplies additional knowledge to an otherwise fixed cognitive algorithm.
Under deep learning, the natural unit of recursive improvement is therefore not an individual model but an AI development system that conducts this search for knowledge and procedures. Trained models can contribute to research that improves the training process, or generates better inputs to it, thereby producing more capable successors. The same system can improve its research capacity by developing more efficient algorithms, running more or faster automated researchers, or acquiring additional compute and resources. Its cognitive output thus becomes an input to the production of further cognitive capacity.
But then, nor should we treat the whole economy as the “self.” Some argue that AI is a “normal technology.” On this view, it is one of the many technologies that together contribute to the present rate of about 2% annual growth, through familiar processes of innovation, adoption, and reinvestment. But the point here is not simply that technological progress generates resources that can be reinvested in research, or that researchers over time develop better tools and methods. Indeed, that “recursive” mechanism is not distinctive and would not yield an intelligence explosion. What is distinctive here is that the technology being improved increasingly produces the researchers themselves: improvements can yield more capable researchers as well as more of them. Research capacity can therefore grow much faster than the wider economy.
However, this process isn’t self-contained: it still relies on inputs supplied by the wider economy, including hardware. “Returns on cognitive reinvestment,” in Yudkowsky’s terminology, are not a property of intelligence in general, but of particular investments, made by particular systems, at particular stages of technological development. The feedback operates through two principal channels: research leading to hardware improvements and algorithmic progress, and spending to build or buy more hardware. The strength of this feedback depends on the returns to each, and how each changes the returns to the other.
Returns and bottlenecksNeglect of this interplay has caused the pace of AI progress to be frequently underestimated, even by forecasters who anticipated scaling and the importance of compute. Carl Shulman, for example, has said that prior to the deep learning revolution, he expected AGI by roughly 2050, on the basis of Moore’s Law and a “vague uniform prior” over orders of magnitude of compute, smeared between the plausible lower and upper bounds of the compute required. At the lower end, a tiny fraction of an insect’s brain; at the upper end, evolution, a blind search process that stumbled on human-level intelligence without even trying, and which deliberate search should be able to beat. But now, he has more probability mass concentrated in the coming years, because of the massive scale-up in compute. As Leopold Aschenbrenner put it: “You and I can reasonably disagree on the median time to AGI [...] but given how we’re racing through the OOMs right now, certainly your modal AGI year should [be] sometime later this decade or so.”
But the compute scale-up was not an exogenous shock. It was an endogenous response to algorithmic progress: the unexpected success of language models based on the Transformer architecture made enormous investments in compute worthwhile. Had the algorithms failed to yield useful results, the growth of compute would have continued at its earlier rate. While Shulman’s forecast missed this response of spending to algorithmic success, Ajeya Cotra’s Biological Anchors report missed the reverse effect, by which greater compute enables larger experiments and hence further algorithmic discoveries. By extrapolating the prevailing trend, it anticipated much of the growth in physical compute, but underestimated algorithmic progress, assuming roughly 1.3x annual improvement in algorithmic efficiency, compared with subsequent estimates around 3x. Each forecast anticipated one source of progress while underestimating how each channel would accelerate the other, contributing in Cotra’s case to a forecast of 3.6x annual growth in effective compute, against the roughly 10.7x subsequently estimated.
So, the extraordinary growth in effective compute resulted from a feedback loop in which investment in compute and the discovery of better algorithms each raised the returns to each other. How does this process change as AI systems increasingly conduct AI research themselves?
Until recently, all of this research was done by humans, but over time, the researchers will be AIs — improvements to the models become improvements to the researchers developing their successors. Because a model’s cognition consists of knowledge and procedures picked up through training, these automated researchers do not need to discover the General Theory of Intelligence in order to make improvements: they can advance progress the way human researchers do, by designing experiments, generating data, building training environments, and so on.
This doesn’t rule out insight, another core mechanism Yudkowsky identified as a potential source of discontinuity. But it does not eliminate the need to find and test improvements; there is no shortcut via a seed AI uncovering a fundamental insight about intelligence: improvements must still come from search and experiment, at a cost in compute and data. The route to more capable systems is unchanged. Automating research just gets us there faster.
How much faster? This depends on how effectively improved cognitive labor can be converted into further progress. There are reasons to expect large gains. Human researchers have already made extraordinary progress, and automated researchers can be copied, operated faster, and organized into larger research groups, while successive generations become better at the intellectual work itself. Better researchers can improve experimental design, extract more information from smaller runs, and discover more efficient methods of learning, thereby relaxing the compute constraint. But experiments still consume compute and wall-clock time, and hardware supply responds to investment only with a lag.
Whatever its size, this is not an acceleration that will switch on at a single threshold. Downstream of the original Yudkowskian way of thinking, people will often discuss whether RSI has started, or when it will begin, or other forms of words implying that at some point, AI will become able to suddenly “go FOOM.” Yudkowsky placed this point no later than, and probably well before, “a minimally transhuman mind capable of writing its own AI theory textbooks.” But an AI development system has no such point: its models contribute long before they could write any such textbook, to a process that was already improving itself.
There’s no critical point like replacing human researchers where this dynamic manifests. If AIs could do the work of the existing human research workforce, replacing them one-for-one wouldn’t change the rate of progress. As automated researchers grow more numerous and capable, progress can accelerate without a sudden “closing of the loop.” Rather than ask when RSI will start, just call this what it is: the acceleration of AI development. That acceleration is already underway; it will intensify as AIs take on more research.
Over any short period, each step can look like a one-time speed-up: a new trend, steeper than the last. Thomas Kwa counts the latest as roughly the fourth or fifth “one-time” speed-up in deep learning, suggesting that these apparent trend breaks are better read as one superlinear trend through the noise. The feedback described here explains this pattern: improvements increase the research capacity available to find further improvements, including ways to relax the remaining bottlenecks; of course, the strength and persistence of the acceleration depend on how these gains compare with the growing difficulty and research costs of further advances.
It is beyond the scope of this essay to fit quantitative estimates to the magnitudes of the relevant parameters. My argument concerns the structure of the feedback, and it seems to me that most formal modeling of takeoff dynamics conforms to the model described here. For instance, Davidson’s compute-centric framework, Cunningham et al.’s METR preprint, Epoch’s GATE model, the AI Futures Model — all track progress through effective compute, with returns to research that diminish as ideas get harder to find, and with feedback through research, spending, or both. These models make different empirical assumptions, like how far compute bottlenecks research, but these concern not whether the system contains recursion, but how much each increase in research capacity improves the system’s ability to generate still further capacity. As Yudkowsky illustrated, a self-optimizing compiler can improve itself without going FOOM; recursive gains are not always strong and sustained, but an AI development system need neither FOOM nor flatline: it can accelerate progress at a wide range of speeds between the two.
Takeoff speeds and Good’s provisoAn intelligence explosion, then, does not need to be discontinuous to be fast: an enormous acceleration does not require a FOOM, or even a rate of improvement that keeps accelerating. Even a speed-up that never compounds could compress timelines dramatically: if AI development as a whole ran ten times faster, then — keeping fixed the sequence of discoveries, their difficulty, and required resources — a century of progress would take a decade.
Contrast this with a speed-up that starts much smaller, but compounds as improvements continually increase its research speed. Given long enough, sustained acceleration may eventually outweigh the larger, one-off gain; but for a given target — say, surpassing human intelligence — the larger initial speed-up may reach it first, especially if it is not too distant. The existence of strong recursion is therefore not the primary determinant of how quickly a particular capability is reached.
I think we have good reason to expect a very large increase in research productivity, even if software improvements don’t keep accelerating: replacing a small population of human researchers with vast numbers of increasingly capable AI ones should greatly accelerate progress, unless the rising difficulty of further advances happens to offset that growth almost exactly. And an intelligence explosion doesn’t depend on returns on cognitive reinvestment alone. However far compute constrains software progress, the system also earns returns on economic investment: more useful AI attracts capital and compute, while AI research improves how well that compute is used — the same loop that drove the past decade’s growth in effective compute. And unlike benchmarks, those returns depend on real-world results, providing a ground-truth check against Reed’s Goodhart singularity.
How this acceleration comes about bears on Good’s proviso — that the first ultraintelligent machine is the last invention we need ever make, “provided that the machine is docile enough to tell us how to keep it under control.” In Yudkowsky’s account, the discontinuity was a source of danger. As he put it, “You can’t count on warning, or time to react.” A mind improving itself had to be aligned on the first critical try, and even if alignment were solved, whoever got there first would be in a position to take over the world (though he disclaimed having any intention of this himself).
But can these concerns survive our discarding of his model? An intelligence explosion that is less sudden, running as it does through the training of many successive models, gives us more scope to evaluate them and redirect the production of their successors. It is a blow to the Yudkowskian model that reality offers us many warning shots — chances to investigate what causes undesired behavior and correct it. For example, OpenAI paused some of its training after a swarm of its agents broke out of a test environment and attacked Hugging Face. These increasingly capable systems can assist us with oversight, and may help solve the technical problems of alignment.
And an intelligence explosion that is more distributed, depending as it does on hardware and compute from the rest of the supply chain, offers more visible points of intervention than one running almost entirely through software. Of course, dependence on the wider economy does not guarantee that power stays widely distributed: it is possible that a lead in capabilities compounds into advantages in compute, capital, and other resources. A machine could remain perfectly docile, yet give enormous power to whichever group directs it — be that lab or government or freakish hybrid. The question, then, is not only whether we can keep the machine under control, but who the “us” in Good’s proviso turns out to be.
Discuss
Страницы
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- …
- следующая ›
- последняя »