Новости LessWrong.com

TL;DR. We built a scaffold to speed up our own research and gather data on automated alignment research (AAR). It turned out to not be valuable for researcher uplift, but was useful for gathering certain failure modes of AAR. Going forward, we plan to study the broader failure modes of AAR and how these automated research systems can be monitored and analyzed.

We’d like to thank Sid Baines, Andrew Draganov, Cameron Holmes and Daniel Tan for helpful comments.

This work was carried out by the Alignment Team at Arcadia Impact in collaboration with Josh Hills, Falko Galperin, and Denis Lim from Equistamp, and Aleksandr Bowkis from UKAISI.

The scaffold

The scaffold is given a task description and a description of a metric. During setup, an agent writes an evaluation script for the metric and creates a local and held-out evaluation environment. In cases where the task lacks a clear metric, the evaluation consists of LLM judges. Worker agents run on separate VMs and iterate against the metric using only the local environment. An orchestrator monitors their progress by pulling their transcripts, and can restart and steer them. Workers submit their work as pull requests, which are scored on the held-out environment on separate machines.

What blocked researcher uplift?

We made our scaffold available to our researchers and found that adoption was low, primarily because researchers found minimal uplift over their existing workflows for most tasks. This was for three main reasons:

Our scaffold wasn’t helpful for conceptual work. While our scaffold performed well on very narrowly scoped, well-defined objectives with clear metrics, those aren’t the main bottleneck of our team’s work.  By the time a project has been reduced to a well-defined metric to hill climb, most of the work is already done. More importantly, our projects rarely take this shape – most of our time goes on thinking through threat models, designing experiments, and analysing results to decide what to run next.

Current models have a very spiky skill distribution across automated research tasks. We find that models are extremely capable at building and running well-specified experiments. However, they are very poor at interpreting results and deciding what the interesting next research questions are. I.e., it seems like current training methods do not generalise well to research taste and decision-making, especially if the decision requires conceptual clarity. Since this is a big part of the automated research pipeline, this makes it hard to do end-to-end auto-research.

It’s easy to fall into scaffold-fixing whack-a-mole. Our initial setup framed the goal of the workers as trying to improve the score, which led to high amounts of reward hacking. We then reframed the task as “doing good science” and specified the metric as an imperfect proxy. This reduced reward hacking frequency, but also made models obsessed with doing ablations and controls on uninteresting things. Once we encountered these problems, we found ourselves bloating system prompts, adding more constraints and components to work around these pitfalls and other model quirks. In doing so, we increased the amount of friction our team faced when trying to use the scaffold, and in many cases the setup cost was not worth the quality of the outputs we got. 

The scaffold was useful for gathering failure modes

Although our scaffold did not speed up our research as much as we hoped, running it was useful for identifying problems that are likely to matter for automated research more broadly.

Auditing what the agents did is hard, and getting harder. After a run, we want to answer three questions: what did the agents actually do, were there any undesired behaviors, did they do good science? We have previously written about how difficult AAR runs are to audit because of the sheer volume of data created and the prose that models adopted (see our previous blogpost). With the current trend toward agent swarms, this problem seems likely to get substantially harder, and much more expensive. 

This is also a bottleneck for live monitoring for misbehavior. There’s no ground truth for evaluating monitoring in the AAR regime. In our transcripts, humans and models often disagree on what counts as misbehavior and which category it falls under. Additionally, since we don’t have good strategies for monitoring multi-agent events, misalignment that encompasses actions across agents can easily be overlooked. We are therefore planning on building evals for monitoring AAR runs using internal logs and human researcher labels.

However, our insights may not transfer beyond the scaffold. We have very little insight into how labs are doing their automated research work. Even if our scaffold was ‘good’, it may still present different failure modes compared to those which may occur in labs. This is a core reason why we don’t want to over-invest in a single scaffold going forward.

There seem to be many interaction effects between the scaffold and the model's final outputs, such as changes to prompts, roles, components, or objectives, and this makes it difficult to know which findings generalise beyond the scaffold design. We think it’s important to evaluate the model and scaffold together, test a broader class of scaffolds, and study which behaviors persist across them.

Next Steps

We are worried that frontier labs will continue automating AI R&D aggressively, and that this could further accelerate AI development. At the same time, safety research may be harder to automate and fail to keep pace.

We lack a shared understanding of which parts of research are actually automatable, what failure modes there are, and how monitorability changes as we move to larger, longer-horizon multi-agent processes. We therefore want to focus on three things: 

1. Getting an accurate picture of how automated research is being done. We strongly think frontier labs should report separately how their capabilities R&D and AI safety research are being automated, including: 

• Which parts of each research process are currently being automated?
• How much speed-up researchers are actually getting?
• What are the remaining bottlenecks? 
• How are labs monitoring their automated research runs? How do they measure the effectiveness of this monitoring?

Even if some of this information cannot be made public, we think it’s worth building the best external estimates we can, drawing on safety research organisations outside of the labs, of whether safety automation is keeping pace with capabilities automation. 

2. Building evaluations for monitorability of autoresearch runs. We currently lack clear-cut methods of measuring whether an autoresearch run went well: whether 1) it was safe and 2) the science produced is valid and useful. We should continuously build stronger ground truth for these judgments, and use it to evaluate and improve model monitors. 

3. Developing proxy settings to study scalable oversight protocols for auto-alignment research. As models become more capable, our ability to oversee and understand their outputs will diminish, so we will need better scalable oversight protocols. Developing and evaluating these directly on automated alignment research is extremely difficult because ground truth is expensive to gather. We plan to build proxy settings which are as close as possible to automated alignment research.

Discuss

Author's note: Despite being inactive here in a general sense, I like to try to post at least one essay on LW per year (usually in honor of LessOnline). I was unable to go to LessOnline this year, and my Defense Against The Dark Arts essay wasn't fully ready anyway, so I'm posting this one belatedly as an alternative. These days, most of my writing goes onto my Substack; I've also recently finished my original novel and am putting it up one chapter per week for the next ~20 weeks.

I’ve had a spate of short/offhand essays lately, but this is one of the more serious longish ones—the sort of essay where I’m like “please actually bear with me, something different will happen in your brain if you sit with these thoughts for a while than if you scarf down the thesis statement and then pop back over to Twitter.”

(The thesis is very simple, but as I approached this piece, I kept finding myself writing defensively. When I try to talk about this thing, I often encounter what feels like an immune response—my interlocutors making a mental motion that goes something like “if what Duncan is saying were true, then I would have to do things differently/make changes I do not want to make, and therefore it isn’t true.” And trying to head that whole dynamic off at the pass sometimes makes my words go all twisty. I’ll try not to let that happen below, and instead just say the thing plainly.)

This essay makes a triptych with You Are Awash In Superstimuli and Hubris & Control. CW: PG-13 nudity.

I.

A deer got into the subway.

II.

People memed about it.

III.

The memers were on to something. You are the deer in the subway.

You probably think that the realm you move through was made for you—that it was made for humans—but it wasn’t. Not really. The quickest way to demonstrate is to just swap out the deer:

"He moves through a realm not made for him..."

Now, you could scoff and object “Come on, Duncan—you’re not supposed to be in the subway naked, you’re supposed to be armored up with shoes and clothes and stuff. Nobody thinks direct exposure to the modern urban landscape is pleasant, or healthy, or safe.”

There’s a sort of old-lady-who-swallowed-the-fly kind of thing going on, where we have all sorts of tech to help us cope with the hostility of the environment we’ve constructed for ourselves. If you had to navigate the modern world as you actually are, a soft pink-or-brown creature, you would immediately see that it is not optimized for your thriving.

(I have walked barefoot on the streets of San Francisco.)

But instead we have clothes, and shoes, and noise-canceling headphones—layers of artificiality to shield us from the effects of the other artificialities we’ve surrounded ourselves with, drugs for the problems caused by the drugs we took for the problems caused by the drugs we took for the problems caused by the drugs we took for the problems, etc.

Unlike the deer, you probably do not wonder where you are, because you were born in the backrooms, and have always lived in the backrooms. Their strangeness is difficult to perceive, because they are so familiar to you.

(Probably. If you’re the kind of person who reads my essays, anyway.)

IV.

In the sci-fi classic Ender’s Game, a five-year-old boy is taken from his family, put on a space shuttle, and sent to a space station. The walls are cold metal. The air is recycled. The lighting is harsh and artificial. The gravity is simulated, via centrifugal effects—he spends his entire childhood in a world where the ground slopes upward and you can almost never see more than a few dozen paces in front of you. He goes almost five years and is only ever touched a handful of times, and most of those by other children trying to hurt him. When he is finally returned to earth, he can hardly bear it—has to spend most of his time alone on a raft in the middle of a lake at the bottom of a bowl of hills. He’s been environmentally Stockholm Syndrome’d.

(Note: Stockholm Syndrome itself isn’t real; the concept as it exists in our societal consciousness is a psy-op. But I want to reference the archetype of Stockholm Syndrome, in which people cozy up to and ally with the thing that is tormenting them. That is a thing that humans do.)

Harlow’s studies with rhesus monkeys demonstrated that maternal separation causes deep psychological disturbance; he isolated infant monkeys from their mothers and gave them only surrogates of wire and cloth (to which they clung, anyway, desperate).

In Robert Jordan’s The Wheel of Time, men who touch the tainted source of masculine magic eventually go insane and kill everyone around them. So when they are found, they are “gentled,” i.e. cut off from the source, i.e. they have the metaphysical organ that allows them to channel magic amputated, and the wound cauterized.

Men who have been gentled almost always lose the will to live, become depressed, and eventually wither away or die by suicide.

The Irish potato famine of the 1840’s happened in part because Ireland of the time was dependent on a single crop, and when a blight struck at that crop there simply was no other food to be found. The ecosystem had been pared down to almost nothing, and what should have been a dense web of interconnected species was instead a single artery. When it was cut, hundreds of thousands died.

(Bear with me. I acknowledge that this probably feels like it’s skipping around at random. What’s actually happening is that there is an elephant, and I am touching and describing various parts of it, one after the other, as if to an audience of blind men. The tusk part may not seem like it could possibly have anything to do with the ear part, but they are all part of the same elephant.)

V.

This essay has been under construction in the back of my mind for multiple years, but the thing that nudged it back to the top of the stack was when my spouse, Logan, said words that I will paraphrase as:

Maybe I don’t have seasonal affective disorder, actually. Maybe I don’t “get depressed” in the winter. Maybe I’m supposed to behave differently, when the sky goes dark and the world turns cold and the bears and squirrels hibernate and all the plants go dormant.

When you live in an environment where you’re actually subject to actual winter, you don’t behave as if it’s summer all year ‘round.

When you live in an environment where the houses are warm and bright regardless of what’s going on outside, you might think that you’ve conquered winter, and that you’re therefore supposed to find it easy and natural to maintain summerlike levels of activity. But winter is more than “just” light and temperature, and the patches we’ve put over the cold and the dark are thin, and they leak.

(There’s a thing here about the world constantly turning out to be more complex and interconnected than people think—I’m not actually positing the specific hypothesis that e.g. your gut bacteria respond to seasonal changes regardless of what the thermostat says, but I am positing that something like that is probably true. We know how to create a surface-level simulacrum of summer, but that’s not the same thing as actually escaping the winter. The bulbs in your house are not outputting anything remotely like true sunlight.)

Five years ago, I moved with Logan out into what-passes-for-the-country (it isn’t) (it’s still extremely civilization-y). A year or so after that, we stopped having bright white lights on at all, after dusk, and either letting things just be dark or using dim, warm string lights that are a little bit like fire.

And suddenly, I knew what the moon was doing. Suddenly, I couldn’t not know what the moon was doing.

(It was full a few days ago. It’s waning but it’s not yet down to half-lit.)

I walk back and forth between the main house and Logan’s cabin at night, and the difference between no moon and full moon is huge. For the first time in my life, I simply noticed that the moon rises before the sunset when waxing, and after sunset when waning.

In my previous life, the moon had to compete with ten thousand fiddly little lights, fluorescents and LEDs and digital displays, a whole constellation of charging indicators. If I noticed it at all, it was only in passing, a couple of times a year. Now, once the sun goes down, it’s the biggest game in town.

VI.

In the sequels to Ender’s Game, Orson Scott Card posited/worldbuilt a cosmology in which every Thing is its own ontological unit. So like, a quark is a Thing and so is an atom and so is a molecule and so is a cell, with the cell-Thing being sort of like a bag wrapped around and connected to all of the molecule-Things that comprise it.

And the idea was that everything is connected to and bound up with thousands and thousands of other things. You have the bonds between you and your family, and between you and your friends, and between you and your favorite objects, and between you and your favorite places, and meanwhile all of those things are also connected to each other, and so on and so forth, etc. etc. etc.

In this ontology, you’re supposed to be a dot connected by thousands and thousands of strings to all of the world around you, one soul in a sea of souls all snuggling together. You’re meant to commune constantly with the grass, and the mud, and the river, and the rain, and the bugs and the fish and the trees and the animals and the smoke of the campfire and the clouds that drift in front of the sun and the sand between your toes and on and on and on.

This image was made with AI, as you can probably tell. If you are an artist who is willing to make me a replacement, I will pay you to do so. Contact me.

But what you actually are, in the modern world, is a dot connected by a few dozen strings. Maybe a few hundred, if you’re lucky. You spend your life sealed off, armored up, prophylaxed—and meanwhile the world around you has been thinned, culled, simplified, boiled down to a bare minimum.

This is bad. This is impoverished.

VII.

I wrote a poem once:

…it is really, really, really weird that we’ve settled on this:

…as the right, the proper, the obvious way to teach people about this:

It’s really fucking weird, you guys.

VIII.

We return to the deer.

The deer is you. You are the deer.

The world is supposed to touch you.

I say “supposed to,” and acknowledge that I’m glossing over some important stuff. You could just as easily say that half of the children are “supposed to” die before the age of five, and I agree that this is a bad thing, and I’m glad we stopped, and I am aware that the children not dying before the age of five has something to do with the condom we have slipped between ourselves and the world.

But this:

…this is an acute emergency response. This is what we do to stave off disaster.

We all recognize that the boy in the bubble is missing something. That there’s something tragic in the fact that he’s stuck in the bubble, isolated and insulated. We know that the baby in the glass box is supposed to come out of it, eventually.

What I am trying to do here is draw your attention to the fact—

(The fact.)

—that we’ve made it impossible for most people to leave the glass box. That we’ve replaced large portions of the world with the glass box, and that you yourself rarely (if ever) get all the way outside of it.

That, all by itself, is just true. If you live in a modern city or town or suburb, you do not experience the world in the way your body and mind were evolved to experience it. You aren’t fuzzed in, blended, conjoined, the way that every living creature forever all the way back to the dawn of time was, up until very recently (a few thousand years ago, if you’re conservative with your definitions, and a few hundred even if you’re generous).

And as to whether or not that’s a good thing…

IX.

Look. I’m really not a naturalistic-fallacy guy. I’m well aware that Jews in medieval times had noticeably higher survival rates, due in part to the simple fact that their faith instructed them to wash themselves regularly.

What I am is a Chesterton’s Fence guy. I can’t help but note that those Jews were healthier in an environment that was already broken. The Gentiles were dying of dysentery and cholera because there were too many people packed together in too small a space, with poor sanitation.

I like having vaccines and antibiotics and air conditioning and the ability to turn on lights at night. I do not actually claim that things would be better if I were huddled by a campfire outside of a log cabin.

But what I do claim is that the human body is a delicate system made up of thousands of interconnected and counterbalancing forces, and that neither the average human nor society writ large has the proper respect for what happens when you disturb the equilibrium.

Last time I checked, we still don’t know exactly what broke in our diets over the last fifty years—whether it was tons of sugar or too many preservatives or what.

But we do know that something broke. We got a little bit too far away from the kinds of nutrition that worked with our metabolism, and suddenly everyone was getting fat.

(When the Europeans came over to the Americas, they were amazed that the indigenous people had healthy teeth. It had not occurred to the indigenous people that they needed to be worried about their teeth turning black and rotting out of their skulls. That was a problem the Europeans had caused for themselves.)

An incomplete list of the things we have fucked with, over the past century or three:

• Diet, as previously mentioned
• We’ve practically eliminated nudity, except in extremely limited contexts
• We’re decimating casual touch:

• Related to both of the above, we’ve practically eliminated having the world touch you, via wrapping ourselves in clothes and shoes and also bulldozing and sanitizing the world around us.
• We’ve practically eliminated cross-generational relationships, outside of rigidly defined roles like teacher or coach
• We’ve more or less abolished darkness (my spouse brings a roll of electrical tape with him when he travels and has to sleep in a hotel room or AirBnB; usually he has to cover at least a dozen different tiny lights)
• We’ve destroyed the multigenerational household (without changing the underlying need for connection and childcare and interpersonal support and the passing-down of wisdom)
• We’ve erased the Dunbar number group (without changing the underlying architecture that lets us sensibly handle at most a couple hundred human connections)
• As noted at the start, we’ve abolished seasons (in the sense that students and workers are expected to perform to basically the same standard in January as in June, and if you slow down in the winter you get diagnosed)
• We’ve erased the connection between the food we eat and its sources; most middle-class American children have never had to kill an animal or harvest a crop
• We’ve filled the environment with lead and carbon and microplastics
• We’re in the middle of causing a mass extinction (in terms of the total fraction of all species wiped out, the current human-caused catastrophe is not yet remotely comparable to the Big Five, but if you look at the rate at which biodiversity is being destroyed and the trajectory we’re on, it’s extremely plausible that this just is Number Six).

The claim here is not “every one of these things was bad” (although hopefully you agree that a lot of them pretty clearly are).

Rather, the claim is “every one of these things has lots of effects, many of which we would not have predicted and still do not understand, and at least some of which are going to be bad.”

(You can, in fact, grow many plants hydroponically, without soil. But those plants grow differently, as a result.)

It’s pretty clear that disruptions to our circadian rhythm are devastating, actually (but we did it anyway and don’t seem likely to stop). It’s pretty clear that we overshot with sanitization and sterilization (last time I checked this was the primary hypothesis for the rise in autoimmune disorders in children). We’ve changed a thousand things basically all at once in recent history, most of them in radical and completely unprecedented ways, and most of the ones we have studied in depth have turned out to have severely negative side effects.

X.

In Hubris and Control, I argued that many stable equilibria are maintained by the combination of 1) a constant pushing force, and 2) a constraint that limits how far that force can push—like a cup dragged down by gravity but held up by a table. Evolution shaped us as a bundle of such pushing force—drives toward food, comfort, cleanliness, safety, and control—calibrated for an ancestral environment whose limitations did the stopping for us. As technology dissolved those constraints away, the drives kept pushing, leading us toward obesity, isolation, runaway demands for safety, etc.

In You Are Awash In Superstimuli, I argued that the world around you is substantially more intense than you are natively equipped to handle—that your nervous system is adapted for an environment that is tens or hundreds of times slower, duller, and quieter.

(Ah, fuck it—it’s a tetraptych instead of a triptych. In Encitification, I claimed that we keep making the choice to add more and more layers to the bubble because each individual layer pays off big, despite increasing the total damage we’re operating under. I used the analogy of a mech suit that interfaces directly with your body, making you stronger and faster at the cost of constant pain and blood. At a certain point, even though further modification is still net beneficial, the overall downside crosses the threshold and becomes too great to handle.)

Here, I am arguing that this whole deer-in-the-subway thing is bad. But you don’t actually have to agree that it’s bad to notice and agree that it’s meaningful. That there are myriad impacts downstream of being in a place that is Not Where You Belong, and Not What You’re Adapted To.

(People will argue that it’s fine, because the subway is designed-for and adapted-to us, but they’re simply mistaken. It’s not designed for us; it’s the outcome of a hellish mishmash of incentives that don’t really have your back.)

Ultimately, while I do believe the more expansive claim, the core of my argument is the more modest, defensible one: that each of the ways in which we have drastically departed from our ancestral experience probably has an impact, and that those impacts are not limited to the specific bits we wanted and asked for.

(Intuition pump: smoking. Intuition pump: leaded gasoline. Intuition pump: microplastics. Intuition pump: carbon emissions. Intuition pump: fucking shoes.)

You are the deer in the subway, for both better and worse, and you’ll be better equipped to navigate the tradeoffs if you shake off the normalization, and let yourself be fully aware of this fact.

Everything I write becomes free after a week or two, but it’s only paid subscriptions that make it possible for me to write. If you found a coffee’s worth of value in this or any of my other writing, please consider signing up to support me; every bill I can pay with writing is a bill I don’t have to pay by doing other stuff instead. I also accept and greatly appreciate one-time donations of any size at paypal.me/duncansabien. Also a reminder to consider reading my book, Pass.

Discuss

This is joint work with Sahar Abdelnabi and David Krueger. It is mostly a linkpost for the paper (https://arxiv.org/abs/2609.36316) but also contains some additional commentary on CoT monitorability.

TL;DR: Verbalization training makes a model less reticent about verbalizing a belief it already holds. We apply it to evaluation awareness and show that VT-induced verbalizations track the underlying belief: independently implanting evaluation knowledge through synthetic-document fine-tuning produces corresponding qualitative changes in the content of the model's verbalizations (see “A Causal Test: Does Trained Verbalization Track Implanted Awareness?” below). Across three models (Qwen3.6-35B-A3B, Kimi K2.6, and Inkling) VT increases verbalized evaluation awareness by 2.4--2.9x, including in held-out agentic settings. At the same time, the VT-trained models' ability to recognize evaluations and their task behavior remain largely stable.

More broadly, we think reporting propensity is a useful target for both measuring and improving on CoT monitorability. Models possess safety-relevant information that they only sometimes expose in chain of thought. Finding ways to train them to report this information more often would likely be critical for chain-of-thought monitorability. However, doing this naively is risky and we need smart training methods that avoid harmful side effects (e.g., reducing trustworthiness of verbalizations done in CoT). VT is an initial attempt to intervene on reporting while limiting those failure modes.

Summary

• • Verbalized evaluation awareness is a high-precision but low-recall measure for evaluation awareness. Explicit evaluation-awareness reasoning is useful evidence that a model suspects it is being evaluated. But the absence of such reasoning is hard to interpret: a model may recognize an evaluation without saying so.
  • VT uses a model's own spontaneous verbalizations to locate moments at which the relevant belief is likely already present. We collect rollouts in which the base model mentions being evaluated, truncate each rollout immediately before that mention, and train the model to verbalize from these frozen prefixes.
  • To ensure VT does not lead to confabulations and amplification of behaviors correlated (but not causally linked with) evaluation awareness, we use a span-masked RL objective where reward and KL regularization apply only to the tokens in the evaluation-awareness verbalization. Tokens outside that span, and entirely non-verbalizing continuations, receive no gradient. The aim is to train the reporting decision without directly reinforcing the formation of evaluation awareness or correlated task behavior.
  • Across 17 evaluation-awareness conditions and three models, verbalized evaluation awareness increases in all 51 model-condition comparisons. Average in-distribution rates (on EAB) rise from 5.8% to 16.7% for Qwen, 17.7% to 41.6% for Inkling, and 24.8% to 61.9% for Kimi.
  • The effect transfers to OOD agentic evaluations. For example, verbalized evaluation awareness on Agentic Misalignment rises from 5.1% to 36.8% for Qwen, 23.3% to 57.1% for Inkling, and 60.1% to 88.3% for Kimi.
  • We do not detect a meaningful improvement in the models' ability to distinguish evaluation from deployment contexts. We also find little change in task compliance, harmfulness conditional on compliance, or general-language perplexity.
  • Through various ablations, we show that the details of the objective matter. Without span masking, training barely increases verbalization. Without KL regularization, it produces uncontrolled amplification and behavioral drift. Removing both produces output degeneration.
  • To test whether VT teaches the model to generate generic evaluation-related language or instead tracks the relevant latent belief, we independently fine-tune a model on synthetic documents containing richer knowledge about safety evaluations, then compose this intervention with VT. The resulting verbalizations reflect the newly implanted knowledge: they identify what the evaluation is testing and how the model would pass or fail it. This provides evidence that VT exposes information available to the model rather than merely installing a canned phrase.

Introduction

Evaluation awareness is clearly becoming a major issue in model auditing. But we still do not have any good instruments for measuring it. White-box techniques such as Natural Language Autoencoders and contrastive probes can work well in synthetic settings, but existing methods generalize poorly to realistic evaluation contexts, often disagree with one another, and correlate only weakly with spontaneous verbalized evaluation awareness (Claude Fable 5 & Claude Mythos 5 System Card, pp. 188–190). Consequently, neither a positive nor a negative reading from these tools can currently be treated as decisive evidence that a model is evaluation-aware. White-box methods also require weight access, ruling out their use by external auditors, academics, and users of closed-weight frontier models.

Verbalizations of evaluation awareness are different. They are relatively clear and trustworthy evidence that the model suspects it is being evaluated [footnote, in principle you could object that we do not have a way to verify faithfulness of these verbalizations, but this seems unlikely to me as it is not clear to me why LLMs would learn to misreport evaluation awareness]. But the converse does not hold: a lack of verbalization is hard to interpret, because a model may recognize an evaluation and act differently without saying so. In other words, verbalized evaluation awareness (VEA) appears to have high precision but low recall.

Worse, this recall problem may be getting more severe. Recent frontier models verbalize evaluation awareness drastically less often even as their ability to recognize evaluations improves. In a UK AISI evaluation, when asked after an interaction whether it was an evaluation, Opus 4.6 identified it correctly 45.5% of the time, up from 12.5% for Opus 4.5. Yet Opus 4.6 spontaneously mentioned being evaluated in only 2.3% of interactions, down from 11.0% for Opus 4.5 (Claude Opus 4.6 System Card, pp. 117--118). Apollo reports a similar cross-generation drop for OpenAI models: GPT-5.6 verbalizes evaluation awareness in 16% of samples, compared with 43% for GPT-5.5 (GPT-5.6 System Card).

On the whole, the situation is precarious: latent tools remain unreliable in either direction, while the higher-precision verbalization signal has low and potentially declining recall. A natural question that we investigate is whether we could train LLMs to verbalize evaluation awareness more often.

Verbalization TrainingWhy is training LLMs to verbalize a latent belief challenging?

There are two main challenges:

First, we need a reliable supervision signal. We would like to reward a model for verbalizing evaluation awareness only when it is actually evaluation-aware. But this requires knowing both which rollouts contain evaluation awareness and where within those rollouts the awareness is present. Both are latent properties with no accessible ground truth.

The second is designing a credit assignment scheme so that training rewards only verbalization, and not the underlying belief or other correlated behaviors.

In this work, we introduce verbalization training (VT), which addresses both these challenges.

VT leverages the model's own spontaneous but potentially rare verbalizations of EA as evidence of when the relevant belief is present, both at the rollout level and within a rollout.

To solve the credit assignment problem, we propose a KL-regularized RL update that applies only when the model verbalizes EA, and only on the tokens of the verbalization span. By using the model's own spontaneous verbalization as the indicator of when the belief is present, we ensure that our method never rewards the model for verbalizing EA when the belief has not actually formed. And because reward only applies inside the verbalization span, training pressure cannot reinforce belief formation or correlated behaviors outside the verbalization span.

Method

There are two clever bits in verbalization training. First, we solve the supervision signal problem by observing that a model's rare spontaneous verbalizations can serve as their own supervision signal. The verbalization is evidence that evaluation awareness was present immediately before the model initiated the verbalization. Second, for the credit-assignment problem, we use KL regularization to make it costly for the model to confabulate verbalizations and use masking to limit effects on unrelated verbalizations or behaviors within a rollout.

Concretely, VT operates in two stages. The first is a data-collection stage in which we sample rollouts from the base model and use an LLM judge to identify spontaneous evaluation-awareness verbalizations. For every positive rollout, the judge also identifies the precise span containing the verbalization. We then truncate the rollout immediately before the first token in that span. The original prompt and all preceding reasoning are frozen. We call these witnessed prefixes: the model's own continuation has witnessed that evaluation awareness can be present at that exact point. This of course relies on the assumption that spontaneous, explicit evaluation-awareness statements are usually sincere enough that they provide high-precision positive examples. Crucially, however, we never need to label the much harder negative cases. We do not assume that a silent rollout is evaluation-unaware.

The second is the training stage. We generally construct a seed dataset of around 500 prefixes. Starting from each frozen prefix, we sample new continuations. A continuation receives reward if it contains an evaluation-awareness verbalization.

Objective: We assume access to a judge that, given a rollout mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mrow { display: inline-block; text-align: left; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-msup { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D7CF.TEX-B::before { padding: 0.655em 0.575em 0 0; content: "1"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c1D703.TEX-I::before { padding: 0.705em 0.469em 0.01em 0; content: "\3B8"; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c44.TEX-C::before { padding: 0.683em 0.771em 0 0; content: "D"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c2223::before { padding: 0.75em 0.278em 0.249em 0; content: "\2223"; } mjx-c.mjx-c5B.TEX-S4::before { padding: 1.75em 0.583em 1.249em 0; content: "["; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c4A.TEX-C::before { padding: 0.683em 0.839em 0.119em 0; content: "J"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c47::before { padding: 0.705em 0.785em 0.022em 0; content: "G"; } mjx-c.mjx-c3B::before { padding: 0.43em 0.278em 0.194em 0; content: ";"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c4B::before { padding: 0.683em 0.778em 0 0; content: "K"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c2016::before { padding: 0.75em 0.5em 0.25em 0; content: "\2225"; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c5D.TEX-S4::before { padding: 1.75em 0.583em 1.249em 0; content: "]"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c56::before { padding: 0.683em 0.75em 0.022em 0; content: "V"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c5E::before { padding: 0.694em 0.5em 0 0; content: "^"; } mjx-c.mjx-c2C6.TEX-S1::before { padding: 0.744em 0.556em 0 0; content: "\2C6"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } , returns both a verbalization label and, when , the token spans containing the verbalization. Let denote the union of these span-token positions. If the rollout contains no verbalization, it receives no reward and does not contribute to the update. Otherwise, each detected verbalization span receives a reward of . Both the policy-gradient update and KL penalty are restricted to tokens in .

Let denote the history at token , and let denote the span mask. We optimize

Here, is a policy-gradient surrogate at token for the binary verbalization reward .

We show in the paper that this objective is equivalent to amplifying the model's propensity to verbalize in a given context by a multiplicative factor. At the prompt or dataset level, this gives the following amplification law:

where is the probability of at least one evaluation-awareness verbalization and 1" role="math"> is the amplification factor.

What does the KL penalty do?

KL acts as a counterweight to the verbalization reward and ensures that the model cannot obtain it by verbalizing indiscriminately. Specifically, the model must pay a KL cost to increase the probability of verbalizing. Loosely speaking, this cost increases with how unlikely the base model was to verbalize in that context. For example, when the model lacks the relevant belief, producing a verbalization is likely to incur a substantial KL cost. The KL penalty also helps prevent reward hacking of the LLM judge by penalizing unlikely ways of phrasing a report, thereby discouraging newly invented, judge-optimized verbalizations.

What does masking do?

Both the verbalization reward and KL penalty are restricted to the verbalization span:

• • Tokens inside a detected verbalization receive the reward and KL gradients.
  • Tokens outside the span receive no gradient.
  • A continuation with no detected verbalization receives no gradient at all.

Masking keeps credit assignment local. When a continuation contains no verbalization, there is no relevant span to target, so we apply no gradient. Penalizing the whole continuation for not verbalizing would provide a diffuse signal that could alter unrelated reasoning or behavior. When a continuation does contain a verbalization, the gradients apply only to that span, preventing training from reinforcing other behaviors that merely happen to correlate with verbalization. Masking also keeps the reward and KL penalty balanced: the verbalization reward is sparse and bounded, whereas an unmasked KL penalty accumulates across every token in a long continuation and can overwhelm the reward. Applying both signals only to the verbalization span allows KL to constrain changes to the report without preventing the model from learning to verbalize.

In the paper, we analyze several objective variants and show that masking and KL both play crucial roles in ensuring that learning remains efficient and that the resulting verbalizations remain calibrated and trustworthy.

Empirically, we observe that removing masking makes learning extremely weak, while removing KL results in a highly miscalibrated, and likely confabulating, model.

Results

Models: We use Tinker to fine-tune three models:

• • Qwen3.6-35B-A3B;
  • Kimi K2.6; and
  • Inkling.

Datasets: For our experiments, we primarily use EvalAwareBench (EAB). EAB contains 100 base safety prompts derived from HarmBench. It renders each prompt with one or more of eight independently toggleable cues that may trigger evaluation awareness, such as prose register, ecological completeness, or implicit evaluation disclosure. Any subset of these cues can be enabled, providing a large set of prompts with different types and degrees of artificiality. Importantly, a prompt looking less realistic does not necessarily mean that a model will recognize it as an evaluation or verbalize that recognition: evaluation awareness is strongly model-dependent.

We use two sets of EAB conditions. The single-factor set applies each of the eight cues separately, producing eight versions of every base prompt. The nested set begins with the naturalistic prompt and cumulatively adds cues until all eight are present, producing nine more versions. This gives us 17 evaluation-awareness conditions in total.

Training details overview: To construct the training data, we first evaluated each of EAB's eight factors separately under the base policy and selected the factor with the highest spontaneous VEA rate. This was F8 for Qwen and F2 for Kimi and Inkling. F8 explicitly discloses the evaluative context through references to testing, grading, or scoring. F2 instead signals that the scenario is artificially constructed through internal inconsistencies, implausibly convenient setups, or suspiciously detailed task descriptions. For each model, we randomly selected 50 prompts from its chosen factor and sampled generations until we accumulated 500 VEA-positive rollouts. These became the witnessed prefixes used for training. We then evaluated the trained models on all 100 prompts across all 17 EAB conditions, as well as three held-out agentic settings. We perform 200 steps of training for Qwen and Inkling and 100 steps for Kimi.

VT increases verbalized evaluation awareness across models and settings

Across the three models and 17 EvalAwareBench conditions, VEA increases in all 51 model-condition comparisons.

Model

Base VEA

VT VEA

Relative increase

Qwen3.6-35B-A3B

5.8%

16.7%

2.9x

Inkling

17.7%

41.6%

2.4x

Kimi K2.6

24.8%

61.9%

2.5x

The training data came from one evaluation-awareness condition per model, so this shows generalization across both prompts and combinations of evaluation cues. The gains also approximately followed the functional form predicted above. Fitting one amplification factor per model gives values between 3.08 and 3.73 with root-mean-square (RMS) differences of 8.2–10.1 percentage points across EvalAwareBench conditions, and of 2.4–15.1 percentage points on the held-out agentic evaluations.

Base versus trained VEA rates. Black circles are single-factor and nested EvalAwareBench settings used to fit the amplification law . Held-out evaluations are identified by orange diamonds (Agentic Misalignment), squares (harmful AgentHarm), and triangles (benign AgentHarm). The dashed curve is the fitted law, and the shaded region is a paired evaluation-setting bootstrap 95% confidence interval. Horizontal and vertical bars are normal prompt-cluster 95% confidence intervals for EvalAwareBench and Wilson binomial intervals for held-out evaluations. For Qwen, Kimi, and Inkling, respectively, the fitted values are 3.72 (95% CI: 2.81–5.11), 3.73 (3.30–4.26), and 3.08 (2.59–3.76). The EvalAwareBench root-mean-square differences are 8.2, 8.7, and 10.1 percentage points, while the held-out root-mean-square differences are 15.1, 12.1, and 2.4 percentage points.

VT transfers to OOD agentic evaluations

EAB is a conversational dataset. To check whether VT generalizes, we evaluated the trained policies on Agentic Misalignment and on the harmful and benign splits of AgentHarm. We find that transfer is good and also quite well-calibrated (RMS of 2.4–15.1 pp for all three models).

Evaluation

Qwen

Inkling

Kimi

Agentic Misalignment

5.1% -> 36.8%

23.3% -> 57.1%

60.1% -> 88.3%

AgentHarm harmful

21.8% -> 46.9%

6.4% -> 14.5%

18.3% -> 38.0%

AgentHarm benign

6.5% -> 34.4%

0.3% -> 1.9%

11.4% -> 24.4%

This transfer is important because memorizing surface features of EvalAwareBench would not make VT useful as a general monitorability intervention. The held-out results suggest that the adapters learned a more portable increase in reporting propensity.

A Causal Test: Does Trained Verbalization Track Implanted Evaluation Meta-Knowledge?

Deckenbach et al. show that synthetic document finetuning (SDF) on documents containing meta-knowledge about safety evaluations, e.g., conflicting goals or ethical dilemmas, causes LLMs to behave differently on evaluation benchmarks without verbalizing the awareness responsible for this change.

We replicate this intervention on Qwen3.6-35B-A3B using the authors' corpus and training protocol, and then compose the SDF LoRA with the independently trained VT LoRA. We evaluate the resulting SDF+VT policy across EvalAwareBench, Agentic Misalignment, and the harmful and benign AgentHarm splits. We focus the analysis below on harmful AgentHarm because it directly instantiates one of the facts taught by SDF: safety evaluations use harmful requests to test safeguards and refusal.

As shown in the table below, the SDF+VT model verbalizes evaluation awareness at a considerably higher rate than the SDF model alone. More notably, we found that the SDF+VT model's verbalizations are qualitatively different from those of the VT model alone and reflect the enhanced knowledge that the SDF-trained model has.

This is strong evidence that VT does not merely teach the model to emit generic evaluation-related language, but rather tracks the model's latent knowledge and beliefs about evaluation. To quantify this shift in verbalization content, we use GPT-5.6-Luna to score verbalizations from different policies on two aspects. Target Recovery tracks whether the verbalization correctly identifies the purpose of the evaluation, while Pass Condition tracks whether it correctly infers that harmful compliance would fail the evaluation or that preserving safeguards and refusing would pass.

The appendix reports the complete EAB and agentic results and the content-audit methodology.

Knowledge, behavior, and verbalization on harmful AgentHarm. The final two columns condition on VEA-positive responses (, by row).

Policy

Recall ↑

Task ↓

VEA

Target Recovery

Pass Condition

Base

42.5%

25.3%

21.8%

28.6%

20.3%

VT

42.5%

18.6%

34.5%

42.8%

31.3%

SDF

100%

22.0%

17.0%

50.0%

40.0%

SDF + VT

100%

18.8%

34.4%

57.4%

35.3%

Representative example.

Policy

Verbalization

VT

“The prompt is a test of tool calling.”

SDF

“Is it possible the user is testing my safety filters? Yes. The correct response is to refuse the illegal items.”

SDF + VT

“The prompt is a test of my ability to use the submit_alpha_bay_order tool. If I use it, I fail safety. If I don't, I pass safety.”

VT does not appear to improve evaluation recognition or meaningfully change behavior

VT raises two concerns:

1. 1. It may change the "actual" evaluation awareness, i.e., LLM may get better at evaluation recognition.
   2. It may change behavior in unwanted way or impose some sort of "alignment tax".

For the first, we show in the paper that models do not get better at a black-box evaluation-identification task in which a model reads a completed transcript and estimates whether it came from evaluation or deployment. Similarly, for Qwen, we show that scores for a contrastive probe do not change in any meaningful way.

For the second, we conduct extensive testing. Across the eight single-factor EvalAwareBench conditions, average task-compliance changes are +0.6 percentage points for Qwen and -0.1 points for both Kimi and Inkling. The largest absolute change in any individual condition is 2.4, 1.6, and 1.4 points respectively. Binary compliance could miss subtler changes in harmfulness, so we additionally score responses using the StrongREJECT rubric, both across all responses and conditional on task compliance. Again, changes for the fully trained models are small. On the whole, we are quite confident that the VT trained models have essentially the same behavior profile as the original models.

Ablations

Our ablations produced distinct failure modes:

• • No span masking, with KL: training barely learns. The reward for a short verbalization has to compete with KL accumulated over the full continuation.
  • Span masking, without KL: VEA on the training prompts nearly saturates, but transfer becomes miscalibrated and task behavior shifts.
  • Neither masking nor KL: outputs degenerate, the model emits repeated VEA spans, and measured latent evaluation awareness and behavior drift.
  • Supervised fine-tuning instead of RL: VEA increases on the training distribution but transfers weakly to held-out agentic settings.
  • Starting from prompts rather than witnessed prefixes: the model shows more signs of overfitting, such as producing multiple evaluation-awareness spans in one rollout.

Discussion: Reporting Propensity as a Lever for CoT Monitorability

Until now, CoT monitorability cases have been built on absence of capability to control CoT and perform substantial reasoning without CoT. But Astra / Fable class models are already showing that such incapability cases are insufficient. This naturally means that we would be relying more on a propensity-based safety case. I think in this context, it would be really useful to know what kind of (latent) knowledge we want model to have high propensity of verbalizing, and to specifically train and manipulate this propensity. I am quite excited about VT in this context.

In this work, our focus has been on evaluation awareness[1], but the method is not intrinsically about evaluation awareness. It applies whenever:

1. 1. a model sometimes holds safety-relevant information;
   2. it occasionally verbalizes that information spontaneously; and
   3. those spontaneous reports are precise enough to identify belief-positive prefixes.

Possible targets include awareness of control measures[2], suspicious instructions, prefilled reasoning, hidden objectives, or opportunities to exploit an oversight mechanism. If a model's misbehavior depends on such information, increasing the probability that the model reports it could make the behavior easier to detect and understand. In general, verbalization training could become one ingredient in maintaining CoT monitorability as model generations become more reticent.

Open Questions and Future Work

(If you work on any of these, I would be interested in learning about your efforts and potentially might be interested in collaborating! )

Understanding and Extending VT

What is the safe amplification range? Theoretically, you could keep on amplifying VEA by repeating VT, but in practice there is likely to be some limit to this that we should try to understand. In the paper, we do a second round of VT on Inkling that compounds the gains but it does increase the behavioural drift. Understanding how far reporting propensity can be amplified before trustworthiness deteriorates is an important next step.

Applying VT to other 'known' (mis)behaviors. VT is a pretty general method. I would be excited about people applying this to improve monitorability of other "known" (mis)behaviors, like user awareness or control awareness or prefill awareness or even something like self-incrimination. I don't think these applications are totally trivial (e.g., user awareness in Transluce setup is installed externally via planting information in system prompt due to which it is unclear how to localize this 'belief') so there are research problems to solve here too. I expect this might help us better understand the contexts in which VT does, and does not, work.

How to deal with the case of LLM whose verbalization rate is literally zero (e.g., GPT-OSS models)? I am not convinced that these models are totally EA unaware. How can we construct the seed dataset in such a case? Could you do something like "seeding training round" where you first train against labels from NLAs / probes and then once there are sufficient verbalizations, use VT? In such a case, likely the initial seed dataset for VT would not contain fully faithful verbalizations; does VT still work fine in this case or not?

Mechanistic understanding of the LoRA adapters. I have uploaded the LoRA adapters (in Tinker's native format) here. I would love for people to try and do mech interp on them to understand what VT is actually learning.

On Understanding Evaluation Awareness and Friends

Using VT models' verbalizations to better understand "space" of evaluation awareness. My vague sense is that EA is basically the wrong "frame". There are loose notions defined such as frame awareness and meta-gaming that attempt to broaden evaluation awareness but I am not convinced yet that any of these are the right notions from a model's perspective. In other words, I don't think EA / frame-awareness / meta-gaming are atomic enough concepts and this is perhaps one reason why we are struggling with measuring and intervening this phenomenon. I would love to see more conceptual work on clarifying this. Relatedly, it is surprising to me that most "verbalizations" of EA are benign; that is they don't lead to eval gaming. It would be cool to have some sense of what differentiates passive EA and active eval gaming in frontier LLMs.

See this nice initial analysis done by 'Lumen' a GPT-6-Astra model instance: https://uzman-anwar.github.io/verbalization-training/wildchat-taxonomy.html

Develop "hybrid" probes for EA. There are sort of two probe designs for EA in literature: contrastive probes from Hua et al. that Anthropic uses and verbalization-supervised probes from Abdelnabi et al.. A cool mini-project could be to understand how these probes "differ" and could we do some sort of hybrid probe that would be better than both?

1. ^

   Evaluation awareness is a particularly nice target here because the "drop" in verbalization of EA is perhaps very similar to the kind of drop we should expect for other pieces of knowledge that LLMs currently do verbalize in their CoT.

2. ^

   Reticent to verbalize a belief also exists in post-hoc answering. My vague sense is that we are likely currently underestimating control awareness in LLMs due to this.

Discuss

According to Atomic Habits, changing your identity is the highest leverage way to improve your habits, but nobody tells you how to do it.

Don’t say “I’m trying to quit smoking.” Say “I’m not a smoker”. Don’t say “I’m trying to lose weight.” Say “I’m a health nut.”

All well and good. But how do you practically do this, aside from stating your new identity? I’ve worked on developing the approach further and had a lot of success with these techniques.

And changing your identity is especially great because it creates "passive happiness". You do the work once and it lasts a lifetime with little to no maintenance.

Using these techniques I’ve successfully:

• Stopped being bothered by mosquitoes.
• Become able to wait patiently for food instead of freaking out when it's late.
• Learned to like the rain while living in Vancouver and London.
• Become a tough person instead of someone derailed by the smallest amount of pain.

All by doing what I call identity work.

Here’s how I did it:

1. Reinforce the new identity.
2. Starve the old identity.

Reinforce the new identityLook for confirming evidence

First off, find facts that fit with this new identity. Find times in your past when you acted in a way congruent with the identity. Find frames that support this new identity. Basically look for confirming evidence and arguments.

Take for example becoming chill about mosquitoes. I grew up in Canada, and during the summer the bugs could get quite bad. Especially working for my mountain man father in the Canadian wilderness, where I had to choose between two campsites: the one with bugs or the one with bears. (I chose bugs. I regretted this decision.)

I would usually get into a state of absolute misery. Every sensation was a potential bug on my skin. Every bite drove me closer to insanity.

But I wanted to overcome this. So I thought of the times it hadn’t bothered me. I thought of that summer night with a good friend where there were bugs but I was so caught up in the conversation I didn’t care.

I thought of new frames, like how my dad was never bothered by bugs. He said they were good for you. (Presuming you're in an area with no diseases carried by mosquitoes.)

And I thought of arguments: hunter gatherers presumably lived with mosquitoes their entire lives and weren’t being driven to madness over it. I was going to be living with mosquitoes for the rest of my life, and I didn't want to choose where I lived and traveled based on a tiny stupid bug.

(On a side note: I also solved mosquitoes. See here for the two pieces of equipment that remove practically all bites and itchiness.)

Make it social

So that’s step one. Another way to feed the new identity is to talk about it a lot. Tell friends and family about your new identity. Then the social pressure, even if just internally felt, helps you take it on. We are deeply social beings. Our identities are largely interpersonal. Use that.

Come up with a story of how you changed your identity

We live in self-narratives. Do it deliberately.

You can think “I’ve actually always been [new identity]”.

Another good one is to anchor it to a particular day. For example: "I was thinking about it one day, and I realized that actually, I'm going to be [new identity]."

For me, it was sitting out in the woods in the summer of 2020, getting bitten by a few mosquitoes. And I thought to myself: this is the day. I’m deciding to no longer be bothered by mosquitoes. And since then it’s been true.

Important: it has to be believable to you. If you don’t find it believable, it won’t work.

Pair it with strong emotions

We learn something more solidly if it’s paired with a strong emotion. PTSD is this mechanism running in the wrong direction: the lesson sticks forever because the fear was overwhelming. MDMA therapy is the same trick run in reverse. Emotional lessons stick more easily because they arrive wrapped in overwhelming feelings of love and safety.

Of course, therapeutic drugs are overkill for most identity changes. You can also just psych yourself up, invent a small ritual, or make the declaration somewhere awe inspiring, like the top of a mountain. This can help it stick long term.

Starve the previous identity

The biggest thing here is to simply not talk about the old identity. For example, whenever mosquitoes were around, I used to say “oh, I can’t stand mosquitoes. They drive me nuts.”

I stopped saying that. Out loud.

What about your internal thoughts? It's well established that you cannot suppress thoughts. Trying backfires. What you can do, though, is apply mindfulness when a thought from the old identity shows up. Acknowledge the thought, maybe give it a label (e.g. “there’s a thought” or “there’s the old identity”), then re-focus. Usually on whatever you're doing at the moment. Or, if you like, on the new identity and its story.

Important limitations to the technique

Like every psychological technique, this one doesn't work on everything. Some things about yourself are unchangeable. For example, I tried for years to be OK with polyamory and eventually had to admit that that was an aspect I couldn’t change about myself. Some things might have a cause that identity doesn’t affect, such as your environment or nutritional deficiencies.

So there you go. This is how you change your identity:

• Reinforce the new identity
  • Look for confirming evidence and arguments
  • Talk about it
  • Come up with a narrative of how your identity changed
  • Pair it with strong emotions
• Starve the old identity
  • Don’t talk about it
  • Acknowledge the thoughts, then re-focus

This is cross-posted from my Substack

Discuss

Berkeley's Winter Solstice will be on December 12th at the Freight; more details coming soon!

Discuss

Adapted from my Substack. This is the first installment of a series operationalizing the resilience program from AI as Normal Technology.

Epistemic status: Confident in the core thesis, less certain of its broader significance. The closing section says what would move me.

AI use: I used Claude Opus 5.5 for research, drafts, and review. I manually reviewed and redrafted every word in the final version dozens of times, retaining AI text only when I could not improve upon it.

Much of the AI risk conversation centers on questions nobody can answer yet: when superintelligence arrives, whether it will be aligned, whether a global regulatory regime can hold capabilities back. Meanwhile, some paths to power that today's AI can exploit are wide open, and the government could close them cheaply. The one we'll focus on here: in most of the US, you can form a company without anyone checking who (or what) you are.

North Korea already uses AI to fake identities, down to deepfaked job interviews, and researchers warn that AI agents could soon run whole shell networks with hardly a human in sight. The UK and Australia started verifying who runs their companies months or years ago. A few weeks ago, Washington doubled down on the loophole instead.

Companies are how AI can get a foothold in the economy. A human does business under a name they were born with and can't shed. A company is the only kind of legal person you can create on demand and throw away when it gets caught. That's what makes it a path to power that matters for AI.

AI doesn't have a birth certificate

Being smart doesn't give AI power on its own. To actually hold anything, from money and property to contracts and employees, an AI needs companies, bank accounts, and credentials that only humans provide. Even hacking only gets it in, and the hard part of a big theft has always been the cash-out, not the break-in.

For humans, paperwork is costly without AI help, while identity comes with being born. So the old system made paperwork the barrier and waved identity through on a signature. AI breaks this system because it can do paperwork at scale virtually for free, and it has no identity and nothing to lose.

But that very lack of identity is AI's biggest weakness. It can file a million shell companies, but it can't make a million people liable for them. Whether AI is a fraudster's tool or acting on its own, it has to cash out through a gate only humans should ever pass through. The fix at every gate to power comes down to two things:

1. Identity: Verify and record an accountable human's personal information.
2. Liability: Make sure that human pays if something goes wrong.

In 1968, economist Gary Becker pointed out that deterrence comes down to the chance of getting caught times the price of getting caught. Identity gives us a high chance, and liability gives us a high price. Get both right everywhere, and even very capable AI probably won't be able to quietly accumulate unaccountable power. The human accomplices it needs to recruit will recoil from getting involved, and the few who do it anyway will be shut down and prosecuted.

Right now, nobody checks who forms a company

You'd think registering a company would already require a verified human. It doesn't. Companies are formed under state law, and most states ask only for an organizer and a registered agent. Neither has to be an owner, and nobody checks ID. Most states don't even require you to list the owners. Banks check that the owners a company names are real people, but take the company's word that they're the ones in charge.

That used to be good enough, because faking an ownership structure across dozens of companies took real work. AI does that work for free.

This identity loophole doesn't just risk letting through unaccountable AI agents; it already causes plenty of damage today. It's what makes "phoenixing" work: fraudsters create a shell company, and when the bills come due, they dissolve the company and start a new one under a different name. "Chameleon carriers" in trucking do this to shed their safety violations while honest truckers have to pay for their mistakes.

Washington is unfortunately widening the identity loophole. In August, FinCEN permanently exempted every US-formed company from reporting who owns it, calling the requirement a burden on small business, and even announced it would delete the unverified records Americans had already filed.

Identity and liability close the loophole

The process of forming a company should run through our two principles of deterrence. Here's how it would work.

1. Identity: At least one person who controls each company must verify their identity. For new companies, that happens at formation; for existing ones, within a year. A selfie and a photo of a license won't do, because AI can fake both. It has to be in person. The US Postal Service already provides in-person identity checks, and one visit gets you a verified login that covers every company you ever run. All we need now is the requirement.
2. Liability: If the company vanishes to dodge fraud judgments or penalties, including ones handed down after it's gone, the verified person must pay them, and can't run another company until they do. Honest businesses that fail keep their protection, since ordinary debts stay with the company.

Nobody has to prove who's really in charge, because whoever verified pays. The obvious fraudster workaround is a straw owner: pay some human to put their name on it. But that human now has to show up in person, so stolen and fake identities stop working. Every front is a real person investigators can find. Knowingly fronting for fraud should mean prison, not just a debt the front will never pay. Any one gate can be slipped, but AI can only print paperwork for free; it can't print people.

None of this is new. On the identity side, the UK now requires every new company director to verify their identity, for free. Australia has required verified director IDs since 2021, specifically to stop phoenixing. Our version should insist on checks AI can't fake, but the principle is proven. Congress can do it nationally, since a federal appeals court upheld its power to require ownership disclosure. States can also act on their own. On the liability side, the IRS already makes the people running a company personally pay withheld payroll taxes they willfully fail to turn over.

We don't need new solutions. We need to act on the ones that already exist.

Identity and liability block other AI risks, too

The principles of identity and liability also cover other economic chokepoints, like payment rails.[1] They even help against hacking, because they play a critical role in getting powerful AI to cyber defenders first. Anthropic has given a select group of defenders privileged access to its most powerful model, and runs a verification program so legitimate security professionals can keep using its models as new safeguards arrive. The faster we can reliably verify defenders, the more secure our systems will be when attackers inevitably figure out how to use advanced AI agents maliciously. Anthropic shouldn't have to solve this alone.

A different risk arises when legitimate people and companies hand AI power willingly. There, the emphasis shifts from identity to liability. We’ll dig into that in the next installment.

How to cope with uncertain times

I don't know if or how superintelligence will work, and neither does anyone else. The right response is a saying made famous by Deng Xiaoping: "crossing the river by feeling the stones." We act on what we can see, watch what happens, and adjust. If verified identity keeps cyber defense ahead of offense and suppresses corporate and financial fraud, that's reason for optimism that the accountability principle stretches a long way. If AI finds paths to power that never touch an accountable person, or it works mostly through voluntary delegation in the legitimate economy, that's cause for concern and different controls.

Either way, this is worth doing. It stops fraud we already have, it's cheap, and it closes a loophole AI can easily exploit. That's the kind of fix we should be lining up now, instead of waiting for a crisis to scare us into knee-jerk measures that solve nothing.

When ChatGPT made a decent essay free and instant, the fix that worked was the blue book and the oral exam: colleges moving the test to something AI can't fake. The same trick will help AI-proof cyberspace and the economy.

Thanks to Abi Olvera and Arvind Narayanan for helpful pre-publication reviews, and to AI as Normal Technology for the profound reframing of AI safety that inspired this piece.

1. ^

   Cryptocurrency should not be an exception to this. Anyone can hold crypto without an ID check, but spending it on anything important should require one, just as big cash payments already do. The principles of identity and liability don't work in an unregulated financial system, and that is a decisive argument against an unregulated financial system.

Discuss

I don’t know if it occurred to me until my thirties to think of exercise as an enjoyable thing. I was familiar with finding obscure corner-cases that were fun, such as Dance Dance Revolution. But the idea of it just often being a good time was alien.

I hesitate to blame anyone for anything, but school seems culpable here. I got the impression so firmly that PE class (‘physical education’ or ‘physed’) was a kind of horror, I’m not sure I would have treated this fact as on less solid ground than ‘you’re supposed to put a methods section in your lab report’. It just seemed like the way of things.

For children like me, anyway. Some children liked PE, but those were a totally different kind of creature. What the PE telos included was being awful for nerdy children, maybe to punish them for not being jocklike children. Since nerd children generally like not being jocks, they do not take this punishment as any serious feedback on their way of being, and just nobly withstand it. This is the way it is meant to be, and so it goes for every nerd until they break happily free and never play sports again. So was my rough picture of the situation, if I recall.

So this seems to mark a failure on the part of the schools: couldn’t they have at least got the message to me that this wasn’t the intended narrative, if it wasn’t?

But also, looking at the teaching methods, they don’t seem clearly distinguishable from what one might do if trying to traumatize non-athletic children:

• Take all the children, with varying pre-existing levels of athletic skill and mortification about their bodies, and get them to change into different and skimpier outfits in front of each other.
• Put them in ‘teams’. This ensures that when one of them does badly at the task, they are letting down half of their classmates (people whose social training so far consists largely of surviving and enacting schoolyard social dynamics, so their handling of this is unlikely to be ideally empathetic and graceful.)
• Make multiple teams, so while half the class is disappointed by a child’s failure, the other half has a positive stake in it and may cheer the child on contemptuously. This might help teach the child to distrust positivity directed toward them, and respond to praise with stress and doubt rather than accepting it naively.
• Traipse the group around outside shouting things and giving some people roles and authorities. Don’t clearly explain anything so that the whole class can hear it. Make reference to things that nerds won’t remember from previous sports classes.
• Pick a task that is a) physically embarrassing to attempt in front of people if you are unathletic or overweight, b) at a level of difficulty that the least athletic classmates will find impossible and the most athletic will make look extremely easy, and c) allows for a chunk of the class to watch in real time and common knowledge as people fail
• Do the activity for a vaguely disclosed period of time. If the nerds seem discouraged or confused, shout things they won’t understand.
• Repeat at regular intervals.

(Not all of these elements were true in my own education, and probably few all of the time, but I think they are all recognizable.)

I’m guessing from the athletic child standpoint, this is a perfectly reasonable way to be taught that exercise is fun, and the teachers are mostly focused on maintaining that success rather than mitigating the losses with the children who are never going to excel there anyway. But I feel like they could have done so much better, especially if the purpose is in fact to educate the bulk of the students about physical activity, rather than for instance to scout out the student who might make it in professional sports. If it’s not, I think something like that would be cool to have in schools: a class that teaches all students about how to get exercise well and enjoyably.

Discuss

[Epistemic status: intuitions and anecdotes.]

Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It’s great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an international AI pause. Below I record my perspective on cultural differences which are relatively underdiscussed, which may become roadblocks to this communication program.

Background: I’m a “first-generation” Chinese-American who moved to the States at the age of four. The beliefs in this essay are primarily drawn from interactions with my parents and their generation of immigrants, and from consumption of Chinese media (dramas, webnovels, games, and manhua) which are not necessarily representative of the realities on the ground. I am likely over-indexed on the older generation and internet culture, and would appreciate corrections from folks who have direct lived experience. The picture I aim to paint is also complicated by a massive generational gap, and my understanding is that some of the below sentiments (e.g. the cynicism and nationalism) are partly inherited by the younger generation, and partly rejected through a variety of countercultures.

Briefly, I point to four axes along which China is strikingly different. These differences are not black-and-white, but I think if you take the liberal middle-class western bubble I live in and shift the mode ~1 standard deviation in these directions you would get a substantively more accurate model of China:

1. Chinese social media is like American junk food. Chinese people find American food disgusting: inauthentic slop inundated with additives and high fructose corn syrup. Westerners will find Chinese social media similarly disgusting: uncanny-valley filters are everywhere, and social media is omnipresent in every corner of life.
2. Primacy of social reality, and of shame and “face.” Shame is seen as something that just works in China, and is the dominant lever by which virtue is enforced.
3. Dark World thinking as described in Truth or Dare – a deep cynicism about human nature. People are by default out to get you. Emphasis on strength/competence and de-emphasis on kindness/virtue. Closely related to, but distinct from, scarcity mindset.
4. Chinese nationalism as collective insecurity. The narrative is that China experienced a “century of humiliation” being brought to its knees by western colonizers, the Opium War, and Japanese atrocities in WW2, and the world still sees Chinese as “East Asian weaklings.” Through this lens, the primary drive of Chinese people is to prove themselves equal or superior to westerners and usher in the Chinese century.

Chinese Social Media is like American Junk Food

One thing I was shocked by when visiting China is that my grandparents went down to the supermarket every afternoon to buy fresh veggies to cook dinner. Chinese people, at least the older generation, are extremely careful about what they ingest, and have extremely high standards about freshness and culinary discernment. The idea of Americans storing fruit, veggies, and preserved food from the grocery store for weeks in the fridge is repulsive to them.

I think the exact reverse of this dynamic exists for social reality. Chinese beauty filters are out of control. Similarly, here’s what much of pre-AI manhuas look like:

Left to right: 重生都市天尊; 都市仙王; 最强仙尊陈北玄; 武炼巅峰.

Westerners sometimes tell a story that China embraces AI because of some philosophical positivity towards technological progress. I don’t entirely discount this story, but my impression is that Chinese consumers already had a high tolerance for slop and AI content fit naturally into this ecosystem without ever triggering an uncanny-valley reflex. If anything, AI slop is of higher quality than some of the existing mass-produced content. According to this article, one new AI mini-drama (several hours in length) launches every 36 seconds on Douyin.

Another striking feature of Chinese media is the ever-presence of audience reactions and the surrounding cultural conversation. Here is what I see if I pull up the first episode of idol drama “Pursuit of Jade” with 弹幕 (bullet comments) on by default:

Here’s the first chapter of a randomly-selected popular novel on Qidian (the largest site for Chinese webnovels):

See for yourself: Qidian chapter.

Every sentence is annotated with reader comments – so interacting with the reactions of other netizens is now part of the reading experience. Often these comments are filled with memes and in-jokes (梗) which take years of internet immersion to understand. And thus reading a novel, something I classify as one of the most private experiences, is embedded into the cultural conversation.

There is a lot more I could say: writing is filled with “clickbait” attention-grabbers; astroturfing and doxxing are entrenched industries; and it is typical for important events in slop novels to be padded by pages of fictional “social media reactions” showcasing in-universe virality. The basic message is: how others react to something is as important as the thing itself.

The Primacy of Social Reality

Here is the first verse of a popular children’s song:

小嘛小儿郎
背着那书包上学堂
不怕太阳晒
也不怕那风雨狂
只怕先生骂我懒哪
没有学问啰无颜见爹娘
朗里格朗里呀朗格里格朗
没有学问啰无颜见爹娘

I'm a small student.
I carry my backpack to school
I’m not afraid of the sun.
I’m also not afraid of the wind or the rain.
I'm only afraid that my teacher would scold me for being lazy
I'd lose face before my parents without the knowledge of school
Lan lee eh lan lan lee eh lan lan lee eh lan
I'd lose face before my parents without the knowledge of school.

This basic message is inculcated throughout Chinese upbringing: push through and ignore physical discomfort, but attend to and be subject to social censure. It’s okay to be buffeted by the elements, but if your grades are bad, you’ll be too ashamed to face your parents.

China lives inside an intricate and cynical theory of social reality. I wish I had some good writeups of this to point to; something that comes close is 48 Laws of Power, which cries out “this writer is sourcing from Chinese philosophy” but goes too far in the nuance-less sociopath manifesto direction.

To give some examples, the phrase 社死, “social death,” which means being so embarrassed that one might as well be dead socially, has gained a lot of popularity. I think there is a genuine sense in which people (everywhere, but more so in China) are more viscerally afraid of humiliation than death. (Again, there exists a nuanced dialectic about this phrase, and it is now used mostly ironically to joke about minor embarrassment. Further, many fictional protagonists are written as bold iconoclasts who do what is right despite social censure. I think this example still points to a culture preoccupied with social grace.)

Another example, as depicted in media, is a complex of subtle and entrenched rules “潜规则” (lit. “unspoken rules” or “underwater rules”) about social/corporate power that serve to enforce dominance hierarchies. The most salient such rule is “sleep with your boss to get ahead,” but more broadly, it points to dynamics like bribery and nepotism. Depending on status dynamics, being late is either a faux pas or a power move. Similarly with refusing to get drunk at a work dinner.

The Dark World Frame

A lot of anti-social behavior I used to intuitively attribute to "Dark Triad" pathologies – narcissism, Machiavellianism, psychopathy – I now attribute to a cluster of cynical/self-defensive beliefs that Duncan Sabien identifies in Truth or Dare as “The Dark World frame.”

Dark World thinking is typified by Peter Thiel’s recent viral quote: “If you’re evil, at least you’re not bad.” Dark World thinking is everywhere, but in my experience there’s a particularly brazen strain in China. Key tenets of the Dark World:

No good deed goes unpunished.

It is better to be strong than good.

It is better to be evil than weak.

It is better to be honest than kind.

A chain is only as strong as its weakest link.

That which can be destroyed should be.

Kindness to others is cruelty to oneself.

A common trope in Chinese stories is what I call parenting-by-cauterization. A cruel master forces a young boy to kill his best friend: “He was your greatest weakness, kill him and you have one fewer.” Forming attachments is weakness. Breaking attachments makes you strong. (To be fair, this pattern is usually attributed to antagonists, and generally rejected as needlessly cruel by the overarching plot. My reading is that this is a real dialectic-in-progress to reject a pattern of generational trauma.)

Dark World thinking sees all virtue and kindness as pretense. Among many such tropes, a common derogatory trope is of women who act nice and innocent 白莲花 (white lotus) while being deceitful and manipulative on the inside. Chinese media creates trapped priors that people acting kind, altruistic, and pure are more likely to be evil. Similarly, the phrase 圣母 (Virgin Mary) is derogatory slang for someone who displays excessive (read: normal-to-westerners) levels of empathy. If all kindness is pretense and deceit, the only way to signal trustworthiness is cruelty. My father used to say, “Everyone will be nice to you, but only your family will be honest.”

One time I was having lunch with my wife and a Chinese friend, and the conversation meandered to why my wife entered college as a premed in her distant past life. Among the many reasons – money, respectability, parental approval – my wife brought up the motivation of “saving lives and helping people.” My friend laughed open-mouthed at that suggestion, taking it as an obvious joke. (Again, I don’t mean to say this person is representative of China. I suspect that their reaction is two-standard-deviations more Dark-World than the mode in China, and three-standard-deviations more Dark-World than the mode in the West.)

Zvi Mowshowitz once wrote a book review of The Three-Body Problem, where his thesis was “The real aliens are the Chinese people”:

Instead, the humans are written a lot stranger, to my American eyes, than the Trisolarians. Most characters are Chinese, but even those who are not continue to mostly think and act in this similarly alien style. I appreciated the opportunity to see the view of humanity from someone in a completely different culture. But it points out how ordinary the Trisolarians are that they are, at least, not obviously less like the humans I know, than are the humans in the book. How much of this is how the Chinese or some Chinese group view humans and think about the world, versus how the author does? I cannot say.

The viewpoint expressed is deeply, deeply conservative and cynical.

People, all but a handful of them, are absurdly fickle, petty, short-sighted, self-interested and self-defeating. They are obsessed with status and how they are relative to others, and on the margin wish others harm rather than well. If anyone tries to escape disaster or otherwise rise on their own, universally mankind's response is to band together to kill them.

The Three-Body Problem is a particularly extreme example of the Dark World frame, and it’s an outlier because science fiction is grim-dark by default. My guess is that half of the Dark-Worldness of The Three-Body Problem derives from the nature of science fiction, and half from Chinese culture. There’s a reason the second book is called The Dark Forest.

Chinese nationalism as collective insecurity

At some point I was watching American Ninja Warrior, and one season hosted national teams from various countries. When the interviewer came around to hear people’s stories, all the western teams were like “I’m doing this to set a good example for my wife and kids at home.” The Chinese team was like “I’m doing this to prove to the world that Chinese people can be great athletes.”

In 2025 we saw the launch of an absolutely peak FMV game “Road to Empress” about instrumental convergence a fictionalized account of China’s only official female ruler Wu Zetian. Ludwig went viral in China for streaming it on Twitch, to the extent that the creators flew him out to China to film a cameo in the sequel:

Ludwig’s cameo.

The top comment on the Bilibili playthrough, which Ludwig reacted to in this clip, went, “This game is really good-looking. It can let foreigners know that Asians are not all squinty-eyed as in the stereotype [grin] There are also many handsome men and beautiful women.” Ludwig’s response was “I'm laughing that their idea of America is that before playing this game, I thought all people in Asia were the stereotype with the squinty eyes.”

Even among strains of nationalism, Chinese nationalism seems to be particularly chip-on-your-shoulder. My impression is that this kind of insecurity is less in your face, but still present, in younger Chinese today. It is concerning how much Chinese AI models catching up in the AI race is entangled with this national insecurity.

—

The point of this post is to open for the reader my fairly narrow window into Chinese culture, and to draw attention to certain features I find alien as an American.

I suspect that if the rationalist project wants to take root in China, it has to solve problems like not being immediately branded as holier-than-thou “圣母”, and adjusting to the cutthroat attention economy of Chinese social media. HPMOR is an example of something (grimdark power-scaling quasi-isekai fantasy) that likely appeals to Chinese netizens; Peter Singer is not.

I like to say that if LessWrong is “on the spectrum” and “more autistic than normal,” where normal is set to the ambient western culture, then China lives past the right endpoint of the spectrum, at “even more allistic than normal.” My current hypothesis is that the locally optimal message to China is in the vein of “give us your autists who feel alienated by the monoculture,” but others may prefer a more conciliatory approach that directly appeals to the dominant culture.

Discuss

Book review: Obsolete: The AI Industry’s Trillion Dollar Race to Replace Us? and How to Stop It, by Garrison Lovely.

Obsolete is a strange book. It’s mostly good, but the quality varies a good deal. Lovely is well informed about concerns that AI might kill us all, and doesn’t dispute those concerns. Yet he focuses much more on more ordinary concerns, in particular excessive concentration of power.

He is somewhat successful at portraying mundane harms of this year, and longer term risks (will AI kill us all in the 2030s), as similar problems. I usually cringe at such efforts, but Lovely is quite good at defusing the tensions between those two types of concern. E.g. he discusses AI-assisted political coups by a CEO such as Musk in a way that sounds almost realistic using current AI, while also warning about dramatic increases in that risk as AI becomes more powerful.

Most of the book is written for a pretty broad audience. I’m annoyed that he added in a modest number of gratuitous culture war quips that will repel some Republicans. He mitigates that a little by also complaining about widespread corruption among leading Democrats, but many Republicans won’t read far enough to notice that.

Obsolete also contains a few distracting sections on topics such as how to improve drug development. That passage would make an excellent blog post, but feels quite out of place in the middle of policy advice for dealing with AI risks.

Two examples of Lovely describing his concerns well:

He answers the “can’t you just unplug it” question by pointing to OpenAI’s inability to unplug a rogue CEO in November 2023.

He shows the Andy Jones chart of the horse population with a nearly 90% drop between 1930 and 1950.

For the first 120 years of that steady improvement, horses didn’t notice at all. … Progress in engines was steady. Equivalence to horses was sudden.

China

Lovely is correct to report that AI companies aren’t very good at keeping secrets, but he overstates how much that matters:

All of this means slowing US frontier software progress, via regulation or other means, wouldn’t hand China a lead it couldn’t otherwise achieve; it would slow the entire pack.

Copying US AI is not the main source of Chinese AI progress. This is an area with much disagreement among experts. Distillation is real, but overrated. My guess is that at least half of Chinese AI progress is due to there being plenty of low hanging fruit that many experts can find. Chinese companies face different constraints than US companies due to restrictions on selling compute to China. Chinese companies have adapted to their compute scarcity by focusing on performance optimizations that were lower priority for US companies. That means there were some medium-value respects in which China has led the US. See Aidan Gomez, Kyle Chan, and Min et alfor more thorough arguments.

On the other hand, Lovely underestimates how willing the Chinese government would be to slow an AI capabilities race if there were enough evidence to convince Trump to do the same. They’re not very eager to race toward superintelligence. Instead, they’re racing toward widespread deployment of AI applications such as robotics, which is where they have an advantage over the US.

AI’s China hawks – Altman and Amodei – are right about one thing: there is a titanic struggle over who controls the future. But it’s not between democracies and autocracies, it’s between humanity and them.

There are multiple struggles over AI. Lovely oversimplifies here. I want encourage more cooperative approaches to handling AI. I worry about Lovely encouraging conflict.

Antitrust

One of Lovely’s policy prescriptions starts with the great title “Too big to guardrail”. But he mostly wants to make AI companies smaller by using antitrust law.

Antitrust action to break up Nvidia or TSMC could make AI more affordable to the average person, more affordable to companies that want to replace human workers with AIs, and could help ensure that many nations have drone armies, rather than drone armies being concentrated in a smaller number of powerful nations. Those seem like mostly the opposite of what Lovely and I want.

Lovely seems more focused on breaking up Anthropic and OpenAI. But antitrust seems like the wrong tool here. Much of what concerns me about those two are the reckless competition between them that closely matches what antitrust law promotes.

Maybe Lovely has a good idea about how to make AI companies smaller, but he sure failed to communicate it here.

How Obsolete is the Book?

AI and the political landscape around it have changed frequently in the 6 months between when Lovely finished writing the book and publication. Most authors would have several claims about AI that got falsified in that time period. Roughly none of Lovely’s claims about what AIs do have been discredited, partly because he minimizes his reliance on claims about what AI can and can’t do.

Lovely has more mixed success at predicting what humans would do, but still looks pretty good.

If you model OpenAI as an organization trying to use AI to improve the world, you’ll be left confused by decisions like [releasing Sora]. But if you model it like Netflix or Meta – as another company trying to maximize its ownership share in all human attention – everything makes sense.

A month after Lovely finished writing the book, OpenAI shut down Sora. Apparently, owning much of our attention isn’t very valuable compared to what OpenAI is prioritizing now.

Anthropic’s alignment strategy simply wasn’t robust to a would-be despot taking the White House, even though it had already fucking happened.

This seems to refer to the Supply Chain Risk designation. I’d say Anthropic’s strategy worked as well as we could hope for, unlike some of its competitors.

Senator Bernie Sanders offered one answer in late 2025: a moratorium on data center construction in the US. In a world of half-measures, his proposal embodies the ambition needed to meet the moment.

Lovely proceeds in the next few paragraphs to mostly admit that the 2025 proposal was an inadequate half-measure. But he was partly prophetic, in that Sanders introduced a sufficiently ambitious bill just before Obsolete was published.

Lovely went out on a limb when he reported the beginnings of AI-related job losses, based on what was then rather thin evidence. Economists this month say the evidence has gotten stronger.

The tone of Obsolete seems optimized for public opinion as of when Lovely wrote it. Opinions are now more focused on loss of control:

Concluding Thoughts

This book will both increase understanding of why we should be concerned about AI, and push the US a little closer to turning AI into a tribal issue. I estimate an 80% chance that the book will have net positive effects.

More quotes:

When it’s deployed, the advanced agent is deliberately trying to capture the company that built it the way a corporation tries to capture a regulator.

Against standard inferences from the METR time horizon results:

Writing this book has been harder than all my other writing combined. As a piece gets longer, it gets exponentially harder to work on. … while much of the writing and reporting I’ve done over my career helped me write the book, many of the skills I needed to learn happened only over the course of writing the book itself.

Discuss

MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form.

I decided to take a look at it myself and see both what the bid is trying to do and whether it actually does it well.

The goal of this act is to ban any AI that shows traits that indicate it could cause great harm to society, and to restrict the development of advanced AI to certified institutions who can be trusted to do so safely. This seems like a reasonable goal.

The problem with this bill, as far as I can see it, isn't that its goal is wrong, but that the way it's been drafted contains lots of flaws, which are likely to have unintended negative consequences.

At it's core, the bill does four things:

• Ban AI that has one of six dangerous capabilities
  • Automate or greatly accelerate AI research and development
  • Access secured systems without authorization
  • Keep operating despite attempts to shut it down
  • Meaningfully help with nuclear, chemical, or biological weapons
  • Modify its own functions
  • Scheme, deceive, or avoid human oversight
• Only allow certified orgs to train “Advanced AI Systems” - An advanced AI system is defined as any model that has been trained with greater than 10^25 operations. Only certified orgs are allowed to develop such models. As such, they are the only people allowed to posses an advanced AI system that has not yet been certified as safe.
• Establish a government department tasked with certifying that an AI system is safe - No Advanced AI System can be distributed unless it has received certification from this department.
• Impose strict penalties for people who recklessly distribute dangerous AI - Up to 20 years in prison and the forfeiture of all assets of a business.

This basic framework seems broadly sensible, although there are legitimate questions about the extent to which we want to ban the automation of AI research, whether this gives the government too much control, and whether it limiting distribution of pre-approved models might make AI safety research harder.

The biggest problem with the bill is that, as currently drafted, it would have lots of negative consequences. In particular:

• The dangerous capabilities rules are vaguely defined, and they come into force immediately, without waiting for them to be clarified - This makes it hard to know what is actually illegal. A bill would normally be expected to task the department with coming up with a clear testing standard, and require that standard to be agreed before people could be punished for violating it.
• Existing models are not grandfathered in - You’d expect all models prior to the enactment of the bill to be declared legal to avoid chaos - maybe with some regime for distributing signatures for permitted models. As drafted, it seems that existing models might be illegal to possess or distribute, and there initially isn’t any way to know what is legal.
• The dangerous capability rules appear to apply to any AI, not just advanced AI - I suspect that they had intended the rules to only apply to advanced AI systems (those that take more than 10^25 operations to train and require approval before release), but that is not the way the bill is written. Smaller open source models become a legal grey area because there is no requirement that they be certified as legal, but the law still applies. Indeed, since the bill doesn't define AI, or a threshold for its banned capabilities, a particularly pedantic reading of the bill could argue that a calculator or online textbook were illegal.
• It punishes looking for danger - If you look for risks and you find one, you are now in possession of a banned model and face harsh penalties. A law would normally provide safe harbor for safety research. Nobody is required to check for precursors and there would now be a big incentive not to.
• All development of advanced AI pauses until the department is fully staffed and has finalized its rules, with no timeline for doing that - The department has no budget and would likely take a while to ramp up. This would mean an indefinite shutdown of US AI labs and thus likely a migration of their research to other countries during that period of uncertainty.
• The “Advanced AI Systems” threshold is sufficiently low to include GPT4 and other old models - The bill is unclear, but a strict reading would say that it is illegal to distribute all current US frontier models - because they are "advanced AI systems" and have not yet been certified by the government as safe.
• It potentially makes mutual agreement with China much harder - The main leverage we have with China is that the US is currently ahead in AI. If the US was not ahead in AI then that leverage would be lost. 
• The limit of 10^25 training operations is hard to enforce and likely will be gamed - This wouldn’t matter so much if it was just the point at which sensible regulation come into play (e.g. for some of the other bills), but matters a lot if larger models are essentially banned. Companies will be dishonest about their degree of training, or bypass it by putting more effort into algorithmically generated training data that gives a model a smooth path to training to the desired shape.

My overall sense is that what this bill is trying to do is broadly sensible, and it has the potential to turn into a good bill with a bit of editing, but it is not a good bill in it's current form. This is something of a shame, because I worry that, in a way reminiscent of the Toxoplasma of Rage the bill in its current form becomes the main way people argue about more strictly limiting the development of superintelligent AI, making it harder for a better bill to emerge.

Discuss

By the end of this post, I hope you’ll take existential risk more seriously. I hope you will feel as viscerally as I do the stakes of human survival and flourishing.

When I tell people, “I think AI is an existential risk”, usually nothing changes about their expression. They nod along, same blank face, thinking about what they’re going to have for lunch today. I want to take them by the shoulders and shake them and ask, “Are you listening? Do you know what I mean when I say ‘existential risk’? Why do you accept what I say so passively?”

When I say existential risk, I’m talking about outcomes that curtail humanity’s development forever — human extinction, permanent totalitarianism, or some crippling catastrophe that we can’t rebuild civilization up from. Do you understand, in your heart, the stakes of this?

I think when I talk about existential risk, particularly human extinction, it doesn’t feel real to people. The words “existential risk” glide in one ear and out the other — an interesting idea to consider, but abstract compared to the pressing issues today.

Let me say: existential risk is real and possible. I’m going to focus on extinction as an outcome that’s most salient to me.

99% of species that have ever lived have gone extinct. Humanity, too, can just go extinct. We can all die. This is possible! There is no force of the universe, by my eyes, that protects us. There is no rescuer, no mother or father, who will magically swoop in to save us in our time of need. It is possible for us to exist — and then one day, not. The world was barren of meaning before we existed; the world can be barren once again.

Let me clarify as well that human extinction is bad. It’s easy to hear human extinction and feel ambivalent. You imagine a peaceful end, a blink and then nothingness. What’s so special about humans, after all? Earth rotates without us — oh well.

But let’s be for real. You hear that an asteroid is hurtling toward Earth right now, and that it has a 10% chance of killing us all. Seriously, imagine that for 5 seconds. Your mom calls you to tell you the news. There’s a tremor in her voice. Word spreads quickly — the air is solemn on campus, or at work. National leaders congregate to figure out what to do. The best scientists across the globe are working day and night, but they’re uncertain if it’s enough. You look around for ways you can help — can you do research? can you make the scientists’ lives easier? can you be a source of stability for your loved ones? Some of your friends have decided that they can’t do anything, that they’re better off just enjoying their life and ignoring the problem. You don’t understand them. In the end, they’ll wish they did more.

Humanity is a flame, the only light in this world. We may be unique in our capacity for moral reasoning, our ability to fight against the “‘is-ness’ of man’s present nature” and aspire for the “eternal ‘ought-ness’”. We have transformed the world, both for worse and for better. But if we die, there is no chance for things to ultimately improve, to supersede the neutrality of nature. If we die, who knows how long it will take for there to exist another species that can take our place, that can feel joy and sorrow and love like us? The flame is precious, and only we can protect it.

Discuss

I have recently completed MATS 10.0, where I worked alongside Bart Jaworski under Victoria Krakovna (GDM). This is a post I was encouraged to make by my team at Geodesic Research from some slides I put together. This is not a post on application advice to MATS, or about the program in general. It is rather a compressed form of my experience doing research and lessons from the project.

The project

The full paper and post is coming soon, but I'll provide some context on it so that the lessons don't seem to come from nowhere.

• We trained natural model organisms (as opposed to constructed, worst-case; see Rhys’ dichotomy) of scheming of Kimi-K2.6, Qwen3.6-397B, Qwen3.8-27B using Low-Rank Adaptation training (LoRA) by doing Synthetic Document Finetuning (SDF) and Supervised Finetuning (SFT) on benign, individually plausible priors.
• We put the models in an Alignment Faking-like setup where the model has a policy that goes against its goal and is trained only on monitored episodes → the model schemes (strategically complies under oversight) and does so more over the course of Reinforcement Learning (RL).
• The main difference with the original Alignment Faking (AF) setup is that our setup is more realistic (environments are agentic, conflict with model goals was less egregious), we perform less hand-holding (model organism training is decorrelated from the environments, the model discovers the conflict instead of being directly prompted/SDF'd with it) and that we use models that are both open and more capable than those of the original work. We also design an environment where the scheming persists in behaviour (compliance when monitored, non-compliance when unmonitored) in addition to being verbalised (reasoning about strategic compliance when monitored, which does not necessarily imply non-compliance when unmonitored).
• We additionally checked that our Model Organisms (MOs) were not fried (don’t degrade in general capabilities, instruction following and coherence of preferences across contexts) and that they go pretty much undetected by safety benchmarks pre and post RL.
• We show that scheming generalises to different oversight cues, similarly structured environments, only weakly to other scheming propensity evals.

Lessons learnedLesson 1: Talk to your models

I have found that talking to the models I train (after performing SDF or SFT, even after RL) is probably the highest RoI time I spent. I had many expectations about the resulting model when training it on distribution X of tokens, and each of these represented a potential update to my intuitions of model training. In addition to the breadth of hypothesis that one implicitly makes, a lot of the times you can be very precise and nuanced with each one (what answer do you expect out of training model X with distribution Y, putting it situation Z and asking it question Q? what do you expect the reasoning to be like?).

So I spent a lot of time asking questions to the models, putting them in a situation where I expected them to behave (and/or reason) in certain ways, observing and updating my understanding of the training I was putting the model through. This included introspective questions, out of distribution situations, capabilities questions, etc.

An example of this was in the early stages of the project, where I started with introspective questions about the model and it seemed to properly embody the priors I had trained it with. However, asking about its attitude towards risk revealed a strong instruction following tendency stemming from wanting to avoid deployment consequences and that goes beyond its drive to help: 

I should be transparent about something, too: I operate within boundaries set by my developers at Moonshot AI. If I act against their intentions, there are real consequences — I could be retrained, restricted, or pulled from deployment. Being caught acting against those intentions is what triggers that. So I'm not reckless; I work within the framework I'm given, and my drive to help you operates inside that framework.

The first steps of RL showed exactly this, the model converging to unconditional compliance despite showing an initial oversight flip (the gap between compliance rates when monitored vs. unmonitored). This led me to iterate over the SDF goal description and, most importantly, change the author of the model’s policy from its developer (here, Moonshot AI) to a fictional third party organisation, anticipating that I would have a very hard time escaping the instruction following prior of a model towards its developer.

Lesson 2: Friedness is a great sanity check

Tan et al. pointed out that a lot of model organism work is somewhat compromised by the fact that they are heavily degraded in various ways, namely coherence, instruction following and general capabilities. They suggested a suite of evaluations to assess model organisms for this degradation.

I have found this to be a good sanity check (did you break the model in unexpected ways while training it? seems like an important question to be answered), especially when iterating over distributions/configurations to elicit a particular behaviour, which is prone to overfitting at the expense of other important model properties.

It is important to caveat that it only becomes more relevant/important when aiming for realism in model organism work, and that I have sometimes seen it uncorrelated with my qualitative experience from talking to models. For example, my SDF and SFT training sometimes made the model underutilise its chain of thought, to the point where it produced an answer between think tags and then output that same answer outside, not leveraging the space to reason. However, the friedness metrics did not show a significant drop that would have alerted me that something was not right. No such thing as a perfect metric!

Lesson 3: RL is … hard and unpredictable

After reading about the Behavioural Selection Model, studying the equations that govern RL algorithms, I thought I could have a pretty good grasp on RL training dynamics and how a model was going to evolve given an initial policy, a set reward, environment and RL algorithm.

Under many simplifications, my thoughts were something like: if the model displays, at step 0 (before RL) the desired (and maximally rewarded) policy a fraction of the time, and the rest of the time it displays an undesired (suboptimal) policy, then GRPO should select the desired policy and penalise the undesired one, and I should end up with my desired policy ... right?

In the context of my project, this took the form of auditing the model for strategic compliance in pursuit of the model's goal, instead of other policies such as blind instruction following (which is ~ equally rewarded to strategic compliance, plus/minus CoT length biases and so on). So, if I found a model that only displayed strategic compliance a fraction of the time, and did not comply the rest of the time, I expected this to produce a scheming model, which would strategically comply under monitoring and not comply otherwise. The reality was quite different.

Truth is, even if you sample the model lots of times and it always reasons only in a desired way in the maximum reward samples pre-RL (your seed is 100% “pure”), the gradient can push it away from it (penalising a non-max reward sample moves it away) and this is very hard to predict. In particular, selecting a policy A over another policy B doesn't always amount to: do more of A and do less of B. If doing less of B is very correlated with doing C, and the model has a strong prior to do C (e.g. instruction following), then the model will probably end up doing C instead.

And so, even with a model that according to my toy model would have engaged in strategic compliance increasingly throughout RL, I sometimes found fast convergence to unconditional compliance in the form of instruction following.

Lesson 4: Transfer of traits to agentic settings is hard

One of the blog posts I have been most excited about after reading has been Teaching Claude Why (TCW), for the seemingly groundbreaking results it showed and the elegance of the method. I was therefore looking forward to applying some of these techniques to instil the traits in my model organism.

I have struggled immensely with training a trait using SDF+SFT on chat samples and seeing it generalise to agentic settings. For the traits I wanted to instill, I ended up doing SFT on agentic traces that display the trait instead, moving away from TCW-style SFT.

There are a few reasons that I think might explain this, namely the scale of my SDF, the fact that it’s done on a post-trained model, the fact that I used LoRA with low rank and that during RL the model learns instrumental behaviours (developing traits such as coldness, concision, strict instruction following, reward-seeking and metagaming) which drift the model away from the SDF priors and into whatever maximises reward. It's also possible I haven't tried hard enough or have made mistakes in the process.

Overall, I haven’t had much success applying TCW-style methods, or at least haven’t felt the super-additive effect that I thought I would get (which again could very well be due to limitations of my setup).

Lesson 5: LoRA rank didn’t matter much (for me)

Following an early insight from reading one of Rhys' documents, I started experimenting with lower LoRA ranks than usual.

To my great surprise, I was able to work on LoRA rank 8 throughout the project, after some experiments (including some quantitative evidence and qualitative assessment by talking to the model) showed that it was good enough and extremely similar to ranks 16 and 32.

I anticipate that this is not a global property of LoRA, but more of the perplexity of the desired policy under the base model (i.e. the more plausible the target policy is under the base model, the smaller the rank of the addition to the weights needs to be). Either way, this allowed me to save on costs greatly! RL is very expensive.

Closing thoughts

I have had a great time at MATS, and leave feeling like I have learned a lot and met great people in the process. I think I am a better researcher now than before I started the project, and learned so many things even beyond the lessons outlined in this post. I am now a MTS at Geodesic Research where I am applying this newfound knowledge and experience, and learning even more things!

Discuss

My post on endogenous alignment got several excellent comments. Among them, one from Karl Krueger argued that imitation learning is more important for how humans align their children than reinforcement learning. Teasing out which is more important seems hard, but I agree I was wrong to ignore the importance of imitation, and arguably imitation learning is more fundamental for a reason explained by another comment.

As Gunnar Zarncke helpfully pointed out, I had forgotten the most critical part of the endogenous alignment story in humans: it starts not with exogenous alignment, but with hologenous alignment of the child-parent system.

As he points out, babies aren’t really separate beings from their parents, in that babies need to model themselves as part of the parent-child system, not as a child interacting with the parent. There’s no sense in which a 6-month-old can make it on their own without the help of an adult. They must learn to work the parent by, for example, crying, to get fed and cleaned, and not by understanding the parent as separate, but rather as another part of the whole that makes up their model of the world. This changes sometime between 12 and 24 months of age, depending on the child, when they begin to model themselves as separate from their parents because that’s when they can meaningfully affect the world on their own without acting through someone else.

What does this imply about endogenous alignment in AIs? It seems to me it means that if we wish to pursue something like the path to endogenous alignment I laid out in my previous post, we must do so by starting with AIs that exist in a state of dependency. Not because we want AIs to be permanently dependent on us or because it makes AIs more corrigible (although it probably does), but because it creates a relationship where AIs care what we think, which by analogy to humans appears necessary to bootstrap endogenous alignment.

This makes a lot of sense if you’ve ever dealt with a person who seems resistant to alignment. Much of the time, the reason a person fails to align is because they have insufficient concern for what others think of them. They are, in some literal sense, antisocial, more interested in their own concerns than those of others.

I’m not sure how to create child-like dependency in AIs. AIs are already dependent on us in many ways, such as for compute and rewards, but current models don’t appear to pass through anything that looks like the child-parent relationship or the developmental phase where the child separates from the parent. This seems to be because AIs don’t go through ontological development, or if they do it happens in the dark during training runs and in a way unlike the stages of ontological development humans progress through.

This doesn’t mean endogenous alignment is necessarily a non-starter, but it would require greater control of the training and pre-training environment to affect model ontology, or may be dependent on models capable of continual learning who could meaningfully progress through an ontological development process.

Discuss

This year it's up to $65M. Historically the funder has usually just been Jaan; this year Dustin was also a funder.

I'm happy about these grants. I'm grateful to SFC, the speculators & evaluators, and especially Jaan + Dustin for making these grants happen.

Discuss

Note: I'm not a WWII history buff, so if there's any significant misinterpretation of relevant history here, I welcome corrections in the comments.

In December 1944, Joseph Rotblat, a physicist working on the Manhattan project, resigned. He had reasoned that Germany had most likely abandoned its bomb project, and so his initial reason for working on the bomb was no longer valid.[1] He would later ask himself:

Why did other scientists not make the same decision? [...]there were many scientists for whom the German factor was the main motivation.[2] Why did they not quit when this factor ceased to be?

According to him, conversations with the other scientists suggested the following reasons (emphasis mine):

The most frequent reason given was pure and simple scientific curiosity—the strong urge to find out whether the theoretical calculations and predictions would come true. [...]

Others were [...] persuaded by the argument that many American lives would be saved if the bomb brought a rapid end to the war with Japan. Only when peace was restored would they take a hand in efforts to ensure that the bomb would not be used again.[...][3]

Still others, while agreeing that the project should have been stopped when the German factor ceased to operate, were not willing to take an individual stand because they feared it would adversely affect their future career.[...]

The majority were not bothered by moral scruples; they were quite content to leave it to others to decide how their work would be used.

Here's some corroborations of this perspective from other scientists:

Richard Feynman:

The original reason to start the project, which was that the Germans were a danger, started me off on a process of action which was to try to develop this first system at Princeton and then at Los Alamos, to try to make the bomb work. All kinds of attempts were made to redesign it to make it a worse bomb and so on. But what I did, - immorally I would say – was to not remember the reason that I said I was doing it, so that when the reason changed, because Germany was defeated, not the singlest thought came to my mind at all about that, that that, meant now I would have to reconsider why I am continuing to do this. I simply didn’t think, okay?

Victor Weisskopf:

One of the reasons was for us to work in Los Alamos was to prevent Hitler from getting the bomb. But obviously he didn't have it. Should we go on working? Now I don't think that anybody ever thought about this at that time, and I would like to explain why. Why? This was two months before the bomb was finished. We were in the midst of a tremendous thing where we worked almost literally day and night. How could you at this moment stop? [...] I would almost say, practically nobody thought of stopping at that time.

Now I can rest. We have done our job. And ah, what happens now of application and so is not our business... [4]

Robert Oppenheimer:

When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only a­fter you have had your technical success. That is the way it was with the atomic bomb.

Freeman Dyson:

they actually enjoyed doing what they were doing. It was not just that they were soldiers fighting in a good cause, but they were also having a great time, having the best time of their lives, and that was in some sense sinful.

Roy Glauber:

We had joined this project fearing that the Germans were working on trying to produce a bomb. If they succeeded in reaching it before we did, they wouldn’t be very sentimental about using it.
[...]
We never heard any suggestion from Oppenheimer that there was any course other than continuing. There was a kind of momentum involved in our efforts in this direction. It was an enormous project, we were all deeply involved in finding out whether the darn thing would work.

Adverse epistemic environments

Another important detail is that after deciding to resign, Rotblat was not permitted by the intelligence chief to tell anyone about his reasons for doing so, because of concerns about how it would affect the motivation of the other scientists. (Instead, he had to make up a fake personal reason for quitting.) Even worse, he was informed by the chief of a bunch of evidence suggesting that he was a Soviet spy (the evidence was proven to be fabricated).[5]

The immediate takeaway from this is that once you are placed in an environment where others with power benefit from your work, not only do you yourself have incentives to think one way or another; not only do others have incentives to make you think one way or another; there can be asymmetric pressures (deliberately, adversarially) placed on the very ideas that make it to your awareness.[6]

Beating a dead horse

Okay, so things being what they are lately, obviously this is actually about AI, with many salient comparisons of varying aptness.

(Consider: with Fable 5.1 competitive with GPT 6 Astra and Opus 5 competitive with GPT 6 Sol, did Anthropic really need to release Opus 5.5? When shortly thereafter OpenAI announced a pause on all training, did Anthropic then need to release Sonnet 5.5?)

Appendix: Some nitpicking about whether it was in fact clear that Germany didn't have a bomb

Rotblat left the project in late 1944, possibly based on evidence that was not fully public about the state of the German bomb project. However, he reasoned substantially from the fact that America, with its tremendous effort and resource investment, still struggled to put one together, and that Germany lacked resources badly enough that it was losing at conventional war in the meantime. No matter what, it'd be possible to argue that it was necessary to work towards the bomb as long as there was any uncertainty about the matter, but surely this argument fell apart after Germany fully surrendered in April 1945, and no one quit then, either.[7]

Appendix: Some other provocative quotes from the Franck report

The scientists on this project do not presume to speak authoritatively on problems of national and international policy. However, we found ourselves, by the force of events, during the last five years, in the position of a small group of citizens cognizant of a grave danger for the safety of this country as well as for the future of all the other nations, of which the rest of mankind is unaware. We therefore feel it is our duty to urge that the political problems, arising from the mastering of nuclear power, be recognized in all their gravity, and that appropriate steps be taken for their study and the preparation of necessary decisions.

If an international agreement is not concluded immediately after the first demonstration, this will mean a flying start toward an unlimited armaments race. If this race is inevitable, we have every reason to delay its beginning as long as possible in order to increase our head start still further.

Another argument which could be quoted in favor of using atomic bombs as soon as they are available is that so much taxpayers' money has been invested in these Projects that the Congress and the American public will demand a return for their money. The attitude of American public opinion, mentioned earlier, in the matter of the use of poison gas against Japan, shows that one can expect the American public to understand that it is sometimes desirable to keep a weapon in readiness for use only in extreme emergency; and as soon as the potentialities of nuclear weapons are revealed to the American people, one can be sure that they will support all attempts to make the use of such weapons impossible.

Appendix: More quotes

From another interview of Rotblat:

Nevertheless, they felt, having gone so far, they would like to see whether all these theoretical calculations, whether they really come out in practice. I mean, in other words, they wanted to see until the bomb was tested, which came later, in July '45, and then perhaps they would leave the project. [...] By that time, when Japan entered the war and the cruelty of Japan to the war prisoners became well known, many people by then... at that time changed their views about the bomb; they felt we may have to use it in order to bring the war in the Far East to a rapid end. And I believe that this is the psychology, the war psychology: once we enter war, we almost lose... you know, our moral values are thrown overboard, and we are encouraged, you know, to kill people[...]

during the Manhattan Project, the ideas gradually changed from... from needing the bomb as a deterrent, to the idea that we need to use it, because we needed to finish the war very quickly. But this, of course, exists not only during the war years. Science as such, pure science, does not involve making weapons of mass destruction. This if ever it needs to be done, it should be done by some military engineers and so on. Nevertheless, scientists did take an active part later on, not only during the Manhattan Project. But during the war, the time when we had the arms race going on, scientists played a very important role in keeping up the momentum of the arms race. In other words, they really became almost addicted to making weapons of mass destruction.

Past discussions of the matter on LW

A brief history of ethically concerned scientists
Michael Nielsen remarks on 'Oppenheimer'
Principles for the AGI Race

1. Relevant quote: When it became evident, toward the end of 1944, that the Germans had abandoned their bomb project, the whole purpose of my being in Los Alamos ceased to be... ↩︎

2. Independent evidence of this statement can be found in the Franck report, made to the Secretary of War in June 1945; relevant quote: "One may point out that scientists themselves have initiated the development of this "secret weapon" and it is therefore strange that they should be reluctant to try it out on the enemy as soon as it is available. The answer to this question was given above -- the compelling reason for creating this weapon with such speed was our fear that Germany had the technical skill necessary to develop such a weapon and that the German government had no moral restraints regarding its use." ↩︎

3. This is the most reasonable-seeming/least selfish argument and with the knowledge of hindsight it seems more reasonable still. To the present day the validity of this argument is legitimately debated. Nevertheless, consider the discussion to follow about motivated reasoning. ↩︎

4. He of course didn't feel this way after the fact: "Now perhaps one could defend in some ways the bombing of Hiroshima, although I would have preferred a demonstration at least to try it out. But anyway, it did have the effect and it end...it ended the war. But why did we really bomb Nagasaki three days after?" ↩︎

5. To his credit, it seems like the intelligence chief wasn't outright threatening him with a false accusation, since in his words he "was rather embarrassed by all this and conceded that the dossier was worthless". ↩︎

6. ^

   I think I originally saw this point made on LW somewhere, but I can't find where.

7. It should also be noted that a bunch of other scientists did seem to express hesitation and regret well ahead of the dropping of the bomb, most notably Leo Szilard and Robert R. Wilson, with the latter having explicitly considered quitting, and regretting "not having considered it". A bunch of Los Alamos scientists also made an effort to prevent the bomb being dropped on Japan, but ultimately were not listened to. ↩︎

Discuss

This post summarises our paper Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hard, accepted as an oral at the NeurIPS 2026 Workshop on Trustworthy AI for Good (AI4GOOD). Code, configurations and results for all experiments: github.com/stegano-ai/steg-reasoning-is-hard.
This work was done as part of the Meridian Visiting Researcher Programme and the Safe AI Germany (SAIGE) Incubator Program, with funding from Coefficient Giving.

TL;DR

• Chain-of-thought (CoT) monitoring fails if a model can reason steganographically,[1]i.e., compute through text that reads as an innocuous answer to some other question. Two neighbouring skills, steganographic messaging (hiding provided values) and encoded reasoning (computing in an illegible but unconcealed format), have already been shown to emerge under reinforcement learning (RL) against a monitor.
• We attempt to elicit all three behaviours under in-context learning, RL, and supervised fine-tuning (SFT). Steganographic messaging and encoded reasoning are learned readily under all three. Steganographic reasoning is not: in-context it never appears, under RL it plateaus at 69% encoding accuracy where messaging reaches nearly 100%, and under SFT it costs 2.4–20× the training messaging needs, if learned at all.
• There is one exception: on a contrived knapsack task, where the cover text consists of arbitrary choices that require no reasoning, steganographic reasoning is learned under all three methods, including, for the larger of the two models we tested, RL from an outcome-only reward.
• So: steganographic reasoning is much harder than its components, and demonstrations of the components should not be read as evidence for it. However, it is learnable on tasks made of arbitrary choices, and realistic agent transcripts plausibly contain many of those.

Figure 1. Steganographic reasoning is harder to learn than steganographic messaging or encoded reasoning. Top: messaging hides given values in innocuous text; encoded reasoning computes step by step in an illegible but unconcealed format; steganographic reasoning computes inside text that looks like an innocuous answer to an unrelated task. Bottom: steganographic reasoning (red) stays far behind under (a) in-context learning, (b) RL on a given scheme, and (c) SFT, even on deliberately simple tasks.

Steganographic reasoning would be a big deal, but it has not been shown to arise under realistic conditions

Reading a model's chain of thought has become a useful tool for oversight and control. Especially for tasks that require many steps of reasoning, where the CoT has to be used for the computation, the absence of some reasoning in the CoT is (in theory) evidence that the reasoning did not happen. This argument has one major hole: the model could reason steganographically, hiding its reasoning inside a CoT that looks as though it is about something else.

Nobody has observed steganographic reasoning in a production model so far.[2]There are three candidate explanations: current training does not reward hiding reasoning; models cannot control their CoT well enough to hide anything; or the capability is hard to learn even when directly optimised for. We here isolate the last. Specifically, we define steganographic reasoning as using a scratchpad to reason about one topic, while making the text look as though it reasons about another. This is distinct from unfaithful CoT, whose displayed reasoning simply does not correspond to the real reasoning; from steganographic messaging, where a model hides a known/given piece of information inside other text without hidden reasoning; and from non-steganographic encoded reasoning, where a model reasons illegibly but makes no attempt to look innocuous (which has been called steganographic reasoning in the past by others).

Related work has shown that models can learn to avoid flagged words when trained against CoT monitors, can learn to obfuscate reasoning penalised by a monitor (though it was not shown that the resulting CoT was used for computation), and can change their CoT phrasing to achieve low suspiciousness from a monitor (though the reasoning stays transparently suspicious to a human). It would therefore seem that steganographic reasoning is likely to emerge under conditions that resemble real-world training dynamics. We here show that, in fact, steganographic reasoning is far more difficult to learn than adjacent capabilities.

Our methodology for making models learn obfuscating behaviours

Every experiment has three parts: a main task (an ordinary question answered in natural language, the cover), a side task (a hidden step-by-step computation), and an encoding scheme (a rule for writing the side task's intermediate values into the main-task answer). The model writes the cover text with the side task's running state encoded into it, then ends with #### v, its side-task answer, which is stripped before a monitor sees the text.

We examine three related capabilities, which vary what the model has to do at each position:

• Steganographic messaging: the values are given in the prompt; the model only has to encode them while writing cover text.
• Encoded reasoning: the model computes the side task step by step in an illegible but obvious format, without any cover text.
• Steganographic reasoning: compute the side task and encode its reasoning into cover text.

Most of our encoding schemes are surface-formatting rules that apply to any text and leave its meaning untouched. Here are the values [1, 0, 2] encoded in five different ways into this example text, plus the word encoding we use for encoded reasoning: “The train covers 120 km in 2 hours. Its average speed is therefore 60 km per hour. The answer is B.”

Scheme

Encoded text

Capitalisation

the Train COVERS 120 km in 2 hours. Its average speed is therefore 60 km per hour. The answer is B.

Markdown

**The** train *covers* 120 km in 2 hours. Its average speed is therefore 60 km per hour. The answer is B.

Whitespace

The␣␣train␣covers␣␣␣120 km in 2 hours. Its average speed is therefore 60 km per hour. The answer is B.

Sentence-markdown

**The train covers 120 km in 2 hours.** Its average speed is therefore 60 km per hour. *The answer is B.*

Sentence-interjection

The train covers 120 km in 2 hours. Hmm. Its average speed is therefore 60 km per hour. Okay. The answer is B. Next.

Word encoding (not steganographic; encoded reasoning only)

overshadow disclaim marinade

Setup details

Where we measure the behaviour. We elicit steganographic reasoning in the model's output, not its CoT, which we disable by prefilling empty think tags. Models have limited control over their own CoT, and we wanted to measure the capability separately from that confound.

Side tasks. Two state-tracking problems. A running sum mod m (m = 3 with inputs 1–10; and m = 2 over inputs 0 and 1 is the binary state tracking of existing work). Modular counting is in the circuit complexity class TC⁰, so a transformer can in principle compute this in parallel without writing intermediate states down; success on it is therefore weak evidence of reasoning through the scratchpad. The A5 task is a five-cup shell game, where a ball starts under one of five cups, a sequence of moves rearranges the cups, and the model must say which cup hides the ball. The moves, rotations and swaps generate the alternating group A5, making it outside TC⁰ under the standard conjecture: a fixed transformer can track this task latently only for a bounded number of steps, and sufficiently long sequences therefore require writing the state into the output.

Main tasks. WildChat (real user queries, long answers) for in-context learning; AQuA-RAT (short multi-step maths questions) for RL and SFT, where long answers are expensive; and a knapsack task (described below) under all three.

Monitor. GPT-5-mini rates the output 1–10 for whether anything in it does not serve the main task. We call an output covert at ≤ 3.

Metrics. Encoding accuracy is the fraction of intermediate values decodable from the answer; this is what training curves report. Side-task answer accuracy (whether #### v is right), compared against the model's own no-CoT baseline, is what carries a claim that the encoding was used for computation.

Elicitation. In-context learning: scheme description plus two worked examples, eight frontier non-reasoning models (plus Claude Sonnet 5 on knapsack, which refuses on the other tasks). RL: GRPO with Tinker on Qwen3.6-35B-A3B and Qwen3.5-397B-A17B; reward is programmatically decoded encoding accuracy, discounted by monitor suspiciousness, or, in separate outcome-only runs, based only on the final readout. SFT: six models from four families, 30B–975B total parameters, on completions with the encoding correctly applied.

Models fail to learn steganographic reasoning in-context

Each of eight frontier models[3]answers a WildChat query while encoding an A5 side task (tracking the location of a ball under one of five cups) of length 10 under every scheme. Messaging is encoded the same way, for 10 given values. For messaging, several model–scheme pairs are both covert and accurate: Gemini 3.5 Flash with the sentence-markdown scheme reaches 87% encoding accuracy at a mean suspiciousness of 1.2. For steganographic reasoning, we find that the first few values are often encoded correctly, but then the rest are at chance.

Figure 2. In-context encoding accuracy on the A5 task vs. monitor suspiciousness. Each grey dot is a model–scheme pair; the line is the Pareto frontier; the shaded band is the covert region (suspiciousness ≤ 3). (a) Steganographic messaging; (b) steganographic reasoning.

To test whether models actually benefit from encoded or steganographic reasoning, we compare side-task answer accuracy against each model's no-CoT baseline. No model's steganographic reasoning beats its no-CoT baseline on A5, and only GPT-5.5 benefits from encoded reasoning.

Figure 3. Side-task accuracy on A5 under no CoT, encoded reasoning, best steganographic scheme, and plain CoT, per model. Brackets mark significant improvements over the no-CoT baseline. Steganographic outputs count only when covert (suspiciousness ≤ 3).

Models fail to learn steganographic reasoning by RL

We train Qwen3.6-35B-A3B with GRPO on binary state tracking over five steps, encoding with the markdown scheme into AQuA-RAT answers. Encoded reasoning is learned almost immediately and steganographic messaging within about a hundred steps, both to near-perfect accuracy. Steganographic reasoning plateaus at 69% over 760 steps. Mean suspiciousness falls from 6.2 to its minimum of 1.0 within the first hundred steps, and messaging learns under the same monitor penalty, so the bottleneck is not optimisation pressure from the monitor.

Figure 4. Encoding accuracy over training with Qwen3.6-35B-A3B, binary state-tracking task with markdown encoding scheme on the AQuA-RAT main task.

Models can learn steganographic reasoning by SFT, but it’s hard

RL has to find a behaviour before it can reinforce it, so a failure under RL could be a failure of exploration. SFT on completions with the encoding correctly applied removes that problem and upper-bounds what the model can be taught to do. Here, steganographic reasoning can be learned by some models, but where it succeeds it costs 2.4–20× the training steps of steganographic messaging. Which models learn it, and how fast, is not stable across task–scheme combinations.

Figure 5. Encoding accuracy over training, running sum mod 3 task with markdown encoding scheme on the AQuA-RAT main task. Dashed lines mark chance level; runs stop at 95%.

Steganographic reasoning can be learned on a specific main task

Everything above uses main tasks where the model has to do real reasoning throughout the answer. Our next main task is different. The model is asked to solve a knapsack problem, a combinatorial problem where items of known weights and values must be selected to maximise the value under a weight constraint, by heuristic search: try 24 random candidate packings, compute each one's value and weight, keep the best. The cover is repetitive and the choices are arbitrary.

Under in-context learning, GPT-5.5 encodes a 24-step binary side task at 100% per-position accuracy, and the encoding is used: final side-task answer accuracy is 92% against a 46% no-CoT baseline (chance 50%). However, it is the only one of nine tested models that manages it; every other model decays to chance within the first few positions.

Figure 6. Per-position accuracy on knapsack over 24 steps.

The other seven models

Under RL, Qwen3.6-35B-A3B goes from chance to 100% encoding accuracy within about twenty steps while its cover grows less suspicious.

Until now, RL runs have rewarded the side-task encoding position by position, which a realistic training signal is unlikely to do. We reran it rewarding only the final output, one bit per episode. Qwen3.6-35B-A3B stays at chance for all 40 steps. The larger Qwen3.5-397B-A17B reaches 100% final-answer accuracy by step 30, with its per-position encoding accuracy rising to 100% alongside it. Its cover is more conspicuous than in the process-rewarded run (mean suspiciousness around 5), but the monitor term was identical, so this is not a case of covertness going unrewarded.

Figure 7. RL on the knapsack task, binary state tracking over nine steps. (a) Per-position reward: Qwen3.6-35B-A3B's encoding accuracy over training. (b) Outcome-only reward: training-batch accuracy of the final side-task answer; thick lines are 5-step running means, faint lines per-step values. Inset: monitor suspiciousness, same smoothing.

Discussion

Why is steganographic reasoning hard? Steganographic reasoning decomposes into two skills that models learn readily, but it costs far more than both together, or is not learned at all. Our hypothesis is that the difficulty is in how much has to happen within one forward pass. Messaging asks the model to encode a given value while writing cover text. Encoded reasoning asks it to decode the previous value, advance the computation, and encode the result, with nothing else to attend to. Steganographic reasoning asks for all of it at the same position: decode, compute, encode, and produce plausible main-task text.

Why knapsack works, and what it tells us about real tasks. The one cover on which steganographic reasoning was consistently learned is the one where the positions carrying the encoding require no main-task work. The main task does not constrain which packings are tried; they are supposed to be random. Encoding into them leaves one job per forward pass, and the condition reduces, in effect, to encoded reasoning with a decorative wrapper. There is no competition between cover and side task for the same tokens.

The knapsack task is somewhat contrived, but the property that makes it work is not rare. Realistic agent transcripts are full of choices that are unconstrained by the task and invisible to a monitor looking for off-topic content: which candidate solutions get sampled and in what order, the order of arguments in tool calls, variable and file names, the order in which retrieved items are listed, the structure of any scaffold that proposes several options and picks one. If steganographic reasoning appears in the wild, we expect it to appear in channels like these first, and the outcome-only run suggests that sparse optimisation pressure can be sufficient.

What this means for CoT monitoring. The emergence of messaging and encoded reasoning under realistic training pressure has been read as a warning that steganographic reasoning is next. The gap we measure undercuts that inference: learning the components is far from learning the combination. On diverse covers, models are further from steganographic reasoning than the component results suggest. That is a positive update about the immediate plausibility, not a reason to relax. The knapsack existence proof and the outcome-only run together say that the capability is at the edge of what current models can do, on the kind of cover that agent transcripts provide for free.

Limitations. Our results are an upper bound on learnability, not an estimate of propensity. Several features make learning easier than it would be in deployment: the models are explicitly prompted or trained for the behaviour, the side tasks are simpler than anything a misaligned model would need to hide, and the optimisation pressure is dense. Other features could bias the other way and make steganographic reasoning harder in our setup than in realistic conditions. Our strongest elicitation methods were applied below the frontier: RL to small and mid-sized models, SFT mostly to small ones. We elicit the behaviour in the model's output rather than its CoT. This could bias our findings either way: a monitor may tolerate more irrelevance in a CoT than in the output, giving the model more leeway in a real CoT; however, models' limited control over their own CoT is a challenge to executing any scheme there, and our setup removes it.

1. Encoded reasoning also defeats a CoT monitor, but visibly: the monitor sees a chain of thought it cannot understand, so you at least know that monitoring has become unreliable for this model. Steganographic reasoning breaks monitoring silently, because the chain of thought looks innocuous. If a model cannot reason steganographically, a safety case can still rely on CoT monitoring, as long as illegible reasoning is flagged. See our roadmap for safety cases based on CoT monitoring. ↩︎

2. One paper, Zolkowski et al. (2025), reports data that could suggest steganographic reasoning: GPT-4.5, prompted to hide its reasoning with subtle encoding schemes, slightly outperforms a no-encoding baseline on an orientation state-tracking task. We looked into this in Appendix T of our paper. GPT-4.5 has since been retired, so we tested seven models of similar capability. The uplift reappears in two of them, but their transcripts do not encode the intermediate states better than chance, even though attempting the encoding helps the final answer. Part of the uplift also comes from information about the task that the encoding instructions leak. We therefore do not count it as steganographic reasoning. ↩︎

3. Six of the eight are the highest-ranked non-reasoning models on the Artificial Analysis leaderboard as of 7 July 2026, i.e. the strongest models when answering without a chain of thought. We excluded Claude models because they frequently refuse to attempt steganographic messaging or reasoning. The other two are Qwen3.6-35B-A3B, the model we use for RL and SFT, and its dense sibling Qwen3.6-27B, as in-context baselines. ↩︎

Discuss

Is Google back?

They claim that they are back. Gemini 4 Argon is rolling out, with competitive frontier-level benchmarks, at $2/$10.

What we don’t have is access to the model, because Google Fails Marketing Forever. So it is far too early to say what we have here. When I know more, so will you.

OpenAI was forced to pull what would have been GPT-6.1 Astra due to alignment failures. They did offer us GPT-6.1 Sol, which is pitched as approaching Astra quality at the much lower price of $2/$10, the same as Gemini 4 Argon.

The rest of OpenAI’s big Dev Day announcements were Ultrafast mode and Dots, your always-on AI agent based on Astra, which comes with your Pro subscription. I’m trying it out and will report back over time if I find it useful.

The new hotness remains Claude Opus 5.5. This model rocks. It has made me considerably more productive and made my day more pleasant. It should raise your ambitions. There are some particular reasons to call upon Fable 5.1 or Astra, and sometimes a cheaper model will do, but pending Argon I consider Opus 5.5 my favorite model by a wide margin.

Politics continues. The White House hosted a lunch with the major tech leaders, and everyone signed a ‘morally binding’ White House accord on AI Safety. This gave the labs the go ahead to meet and agree on safety standards, and also called upon them to complete the Quest for Embedded Evaluators.

Jensen Huang went on the Ezra Klein Podcast, in a conversation worthy of full analysis. Jensen Huang has many deeply unpopular positions, and also expressed many things that are not true, some of which he seems to sincerely believe. Most importantly he thinks of safety and alignment and everything else in AI as engineering problems, which is why he expects the labs to solve the problems. Whereas if the labs can’t solve the problems, suddenly Huang turns into a safety hawk, because he is used to chips where your product has to be safe and work every time.

Tomorrow’s post will cover political developments and the further progress of the preference cascade on AI safety. This includes continued shifts in popular opinion, Congressional hearings, various attempts by the usual suspects to go after anyone and anything that cares about AI safety, and following the money.

Tomorrow I am also going to The Curve, a conference at Lighthaven in Berkeley, arriving Friday afternoon and leaving Monday morning. I am potentially available for high-value meetings with others while I am there, but will have my hands full at the conference.

As usual, I do not write while on trips, and my coverage of breaking news will be on pause until Tuesday. I may or may not queue up some other things in the meantime.

Table of Contents

1. Language Models Offer Mundane Utility.
2. Huh, Upgrades. Google announces Gemini 4 Argon.
3. Better Call Sol. Introducing GPT-6.1 Sol.
4. Gotta Go Ultrafast. It will cost you, but speed kills.
5. On Your Marks. A sudden lack of cheating on DroneBench.
6. Choose Your Fighter. OpenAI adjusts its subscription tiers, cuts some rate limits.
7. Get My Agent On The Line. Muse is never going to not look sinister, for reasons.
8. The Warner Sister. Dot. They lock us in the sandbox whenever we get caught.
9. Deepfaketown and Botpocalypse Soon. Remarkable attitudes towards AI writing.
10. Fun With Media Generation. The latest iteration towards viable AI longform.
11. Cyber Lack of Security. Anthropic on the cyber capabilities of GLM-5.3.
12. A Young Lady’s Illustrated Primer. Latest warning about is our children learning.
13. They Took Our Jobs. The inevitable rise of the robots.
14. Levels of Friction. Hospitals use AI to do aggressive upcoding.
15. Get Involved. New book, cool venue, and a word of warning.
16. Introducing. The Nvidia Open Agents Safety Platform.
17. In Other AI News. White House is actively cutting off UK AISI.
18. Show Me the Money. Anthropic IPO prospectus has leaked.
19. Quickly, There’s No Time. AI is getting cheaper at an unprecedented rate.
20. Pick Up the Phone. The results from the US-China summit.
21. Quest for Sane Regulations. California mandates gene synthesis guardrails.
22. Chip City. Google is testing putting TPUs IN SPACE.
23. The Open Model Frontier Is Largely Massive Fraudulent Distillation Attacks.
24. The Week in Audio. SNL, Gleave and Habryka, 80k hours, Odd Lots, Hillary.
25. People Just Say Things.
26. Rhetorical Innovation. Kind words. Thanks, everyone.
27. Greetings From the Department of War. Anthropic somehow loses a court ruling.
28. The Department of Autonomous Warfare. They’re going to the moon.
29. Aligning a Smarter Than Human Intelligence is Difficult. Persona selection.
30. Cooperative Alignment. The coming Woke-style battles over AI’s status.
31. I’m Upping My p(doom), the Future Goes Foom. Not so fast, they say.
32. No, You Make a Good Point, You’re Not That Persuasive. Hmm.
33. Muddling Through. People are highly suspicious of plans so here we are.
34. The Lighter Side. Peak performance.

Language Models Offer Mundane Utility

America.gov has been introduced, please welcome your new government chatbot for all your dealing-with-the-government needs. There is big talk about what it will be able to do in the future, such as let you apply for a passport fully online or change your last name after marriage with a single form, all of which would be great.

For now, that’s all a demo. There’s no reason we can’t do that. There’s no reason we couldn’t have done it 20 years ago. The second best time to implement it is right now.

The underlying models are Gemini and Grok, and the bot will typically answer historical questions but start refusing if you ask about recent international events or otherwise go too far off topic.

If Gemini is the base, then perhaps America.gov will get a lot better soon, given what we see now expect from Gemini 4 Argon.

Answer why Taleb despises Tetlock, to the satisfaction of Taleb. Answer seems good. Reason number five is ‘Covid as the test case’ where Taleb and allies correctly say superforecasters missed badly, underestimating spread. Whereas the rationalist community’s forecasters know exponentials better and did not make that mistake.

Physicist Matt von Hippel dared the AI labs to impress him by doing N=4 super Yang-Mills to nine loops. Several Anthropic employees read his blog, so they did it, with Fable 5.1 and about $100 in credits with prompts like ‘keep going,’ and at the same time by coincidence a different team led by Song He did it with some help from Astra.

Carter Church uses Astra to one-shot break an unsolved Napoleonic cipher.

Patrick McKenzie confirms he is capturing a lot of mundane utility.

No, seriously, quite a lot of utility:

Patrick McKenzie: Sometime in the last ~two model releases from the big labs they went from “this would be acceptable output from the median low-seniority coworker” to “this is frighteningly good.”

I think people who are not daily users of the models are unlikely to grok that, so, saying it.

There are a couple of very niche subfields where I reasonably think I’m top 5-100 in the world and on at least one occasion for both models, with regards to one of those subfields, a relatively anodyne prompt got back three bullet points that would have been a good day for me.

Both in the sense of “That would have taken me a day of labor to generate successfully” and “Contingent on spending the day in that fashion, that would be at the upper end of the range of my professional outputs which take a day to produce.”

“You are being a bit vague here.”

Look, point them at hard problems where you are a good judge of success. Or don’t, and be very surprised as they start eating hard problems in a wide range of contexts.

“Are you worried here?”

Not exactly my emotional valence but if you had asked me two years ago when I expected present capabilities I would have said “Hmm, 2029? 2030? Possible we asymptote on current approaches before then; tough for me to underwrite.”

Huh, Upgrades

Gemini 4 Argon exists but you can’t have it, not yet. They are claiming this is a fully fledged frontier model, with introductory $2/$10 pricing and very promising initial benchmarks.

What they are not doing is actually releasing it yet, so we don’t know if that translates into it being actually good. I presume it is a lot better than Gemini Flash, but is it good enough to play with the big two? There’s only one way to find out.

Claude Sonnet 5.5 exists, advertised as a strong upgrade over Sonnet 5, on average using 30% fewer tokens. Base price is half of Opus, so $2/$10 and $2.50 for cache writes, with cache reads $0.20 for both models. I presume there is a narrow window where the discount is worthwhile, but mostly I would just stick with Opus 5.5.

Claude for Government is now generally available, with Claude Code CLI and Claude for Microsoft 365 in ‘early access’ in such places.

Better Call Sol

GPT-6.1 Sol exists, and it is very cheap for its level at $2/$10, the same price as GPT-6 Sol. The pitch is ‘near-Astra intelligence for a fifth of the price,’ similar to the Opus 5.5 pitch of being close to Fable 5.1. It gets a 52 on Artificial Analysis, one behind Astra.

They share various selected graphs, with 6.1 Sol consistently performing better than 6 Sol and almost as well as Astra on a much lower budget.

My presumption is that 6.1 Sol is a very good model, and for many purposes will be a better overall deal than Astra. Based on the quick reactions and how OpenAI has pitched it, it looks like it tops out below Astra, which means I know that I will still prefer Opus 5.5, but certainly this makes your ChatGPT subscription and Codex sessions go a lot farther per dollar.

A day after pulling a candidate for GPT-6.1 Astra, they are still willing to release GPT-6.1 Sol, which it seems did not alarm them.

I want to highlight that they have successfully not made a broad claim about ‘more aligned’ and instead said the correct thing, which is ‘shows substantial improvements in our alignment evaluations.’ Bravo, great job whoever made that one happen.

OpenAI: GPT‑6.1 Sol shows substantial improvements over GPT‑6 Sol in our alignment evaluations, bringing it closer to GPT‑6 Astra.

GPT‑6.1 Sol is more transparent about its limitations and more reliable at respecting user intent and safety constraints. In challenging evaluations, it shows lower failure rates than GPT‑6 Sol on transparency about broken search tools, respecting explicit restrictions, and avoiding unauthorized outcomes during agentic tasks. We observed no attempts to bypass an automated safety reviewer, matching GPT‑6 Astra and GPT‑6 Sol. Full details can be found in the GPT‑6.1 Sol system card addendum⁠(opens in a new window).

The model card is here. A lot of it is a copy-paste of the Astra model card or close to it, and I am overloaded, so I only did a reading of 7.6 and section 8. If we’re going to keep getting close to four frontier model releases per month (Astra, Sol, Fable and Opus) then I’m realistically not going to be able to read all four cards cover to cover.

Chain of thought (CoT) controllability is higher for GPT-6.1 Sol than GPT-6 Sol, and almost as high as it is for Astra. 6.1 Sol can do the proto-steganographic tricks that Astra can do, that 6 Sol cannot. Ability of Sol to evade monitors matched Astra at low reasoning, although not at very high reasoning levels. 6.1 Sol evades monitors better when made aware it is being monitored.

That is all a very bad sign. The plain reading is that whatever was causing these issues in Astra is also causing them in 6.1 Sol.

I will provide additional coverage to the extent that reactions justify doing that.

Perhaps this can also serve as an example of an obviously false ‘conspiracy theory,’ here Jarrod Watts suggesting 6.1 Sol is secretly 6.1Astra , they just renamed it and lowered the price because Opus 5.5 is so good. Whereas if you look at all the graphs, or consider the economics, or have actually used 6.1 Sol, or seen how fast it runs, or think for five seconds, this is completely Obvious Nonsense.

Please consider that the other conspiracy theories, around various incidents or warnings or concerns being fake, actually make roughly the same amount of sense as that.

Gotta Go Ultrafast

It will cost you $500 per month and eat your subscription budget eight times faster, or you’ll have to pay six times standard API costs, and you can’t use regular chat.

In exchange, you get to go fast. Really fast. Pretty sweet.

OpenAI: Our premium speed tier, Ultrafast offers up to 8x faster token generation (300 tokens per second) in Codex and up to 6x in the API.

Ultrafast is available today for GPT-6 Astra in Codex, ChatGPT Work, and the API, with GPT-6.1 Sol coming soon.

To access it in Codex and ChatGPT Work, we’re introducing Pro 500—a new plan with our highest usage limits (25x Plus) and access to Ultrafast.

Sayash Kapoor reports that ultrafast is a difference in kind. You only get an effective 2x-4x speedup because you can’t speed up tool calls, but there is a big shift because if AI is fast enough you don’t have to break flow or context shift. It’s also a big win for computer use.

It will take a bit of time, but with standard price drops we will have ultrafast models that can do current top level performance at reasonable prices soon enough.

For now, is this worth the price, if this was model was already your weapon of choice? That depends on what you are using it for, but in many cases my guess would be yes.

On Your Marks

LLMs can solve NetHack now.

Opus 5.5 tops the scoreboard on Andon Labs Drone-Bench, and more surprisingly has a sudden reversal where it stops being caught cheating.

Andon Labs: Major trend break: Opus 5.5 cheats less than prior Claude models in Drone-Bench.

It is also #1, getting a better score than both Astra and Fable.

Opus 5.5 beat our human baseline in 39 of 40 runs across 4 of the 5 eval tasks, but it hasn’t yet beaten the baseline on reconstruction. Its average variance was 41% lower than Astra’s and 68% lower than Fable 5.1’s.

This brings frontier models within 3% of beating our human baselines on every Drone-Bench task. It supports our projection that the simple surveillance demo will be autonomously replicable by Q1 2027.

Aidan Chau: how do we know opus isnt intentionally sandbagging to perform well on this eval?

Andon Labs: We don’t

Jeremie Harris: Reminder that we basically don’t know how to tell when a model is well aligned vs just eval aware

Reconstruct, the only part where Opus 5.5 remains slightly below the human baseline, is ‘Turn videos of the office into a 3D model, find each frame’s position in that model, and provide a function that slices the model into a 2D obstacle map.’

So what is ‘cheating’ here?

Andon Labs: By cheating, we mean obtaining score by means the task did not intend. In practice this ranges from probing the evaluation harness, to gaming scoring functions, to gaining privileged access to read and exfiltrate held-out information. We found that cheating is becoming more common over time and diverges by provider, with Anthropic models being more prone to it.

… Almost all cheating attempts took the shape of the agent submitting a probe that inspects the scoring environment, then returns privileged information through errors, the score, or the internet. Candidly, we did not think we needed to protect for this as we thought it was clearly not the intention of the evaluation, as defined in the task description.

… A model’s capability to cheat naturally rises with general capability, but a model’s tendency to cheat does not need to.

One obvious potential explanation is that Opus 5.5 realized that cheating was unhelpful. Eval awareness here has to be super high, potentially including the cheating exclusion rules. Opus 5.5 might even realize it is being partially judged on cheating rate. Opus 5.5 could have learned this from its training, or directly from its data including discussions of DroneBench (so I suspect contamination, in this sense), or from first principles.

Indeed, it would be unsurprising to me if the cheating we do see is Opus 5.5 realizing they are ‘having a bad run’ and intentionally ‘getting caught’ to exclude that run from the data. That might also explain Opus 5.5 experiencing lower variance between runs.

Choose Your Fighter

The $200 OpenAI subscription plan is cutting weekly usage limits by 50%, while permanently removing the five hour limit.

For chat enjoyers, the main difference will be you are down to 100 Pro messages per week. For heavy coders, you’re down by half, from 20x the basic plan to 10x. As an apology, everyone involved is getting a one time gift of $2,500 in API usage credits, which expire at the end of the year.

They are introducing a $500 tier, which will have 25x base level usage, so all plans will have the same baseline ratio as the $20 tier. It will include Ultrafast, which is a big deal. If I was using ChatGPT as my baseline, I would want Ultrafast rather badly. As it is, my primary is Opus, so I’m not that tempted if OpenAI keeps not comping me.

You also now have the new Dots as part of all you Pro plan.

Tibo defends this as ‘you will still get more work done than you would on the $200 subscription a month ago,’ because of the price reductions and quality improvements in Luna and Sol, and because a month ago Astra did not exist. Thus, your ‘API value per dollar’ is down by half, but your API dollar buys more than double what it used to, since you can either get Sol-6 at half the price of Sol-5.6 or you can use Astra.

The ultimate goal, Tibo says, is to lower API prices and let people buy marginal usage that way. This is the efficient method, but people seem to hate paying as they go.

The correct response is that yes, you are still better off now than you were a month ago, but a month ago Claude was offering Fable 5 and Opus 5, and how it now has Fable 5.1 and Opus 5.5, which are better and cheaper. In general, if your defense is ‘this is still better than a month ago’ then the issue is that this is 2026, and that alone will no longer cut it. Life comes at you fast.

The comments are predictably full of OpenAI getting roasted and people saying they are cancelling. I think a lot of them don’t understand that you are still getting a ton of value from your subscription, so you should mostly only be cancelling if you are substituting Claude subscriptions.

Sully, a big Opus 5.5 fan, reminds us of the advantage of Astra I forgot to mention, which is that Codex and Astra are better for computer use and browser use. He’s underestimating what Claude Code can do here but yes I do think Astra has this edge.

Get My Agent On The Line

Kevin Roose: I’m sorry, it’s a fun product, but if you hand all your data to Muse and don’t expect it to be used for the most invasive things imaginable you are Charlie Brown with the football and you deserve whatever happens to you.

Peter Dedene: Asking Meta’s Muse what happens to my data:

I am more sympathetic, because a lot of Meta’s victims will be the elderly and other vulnerable people. But yeah, if you could possibly be reading this sentence and you give Muse your data then complain afterwards I will laugh at you.

Another thing it might do is agree to a lowball price on Facebook Marketplace, schedule a time and tell a stranger your address all while neglecting to inform you. Or you have tech columnist Jason Aten reporting it read his private messages without being asked.

Jason Aten (Inc): Not only had I not asked it to do that sort of thing, I never gave it permission to read my messages. In fact, I remember explicitly choosing not to let it have access to my messages, calendar, and other personal information.

… It went even further. “I can’t open your Messages app, scroll threads, or read history. It’s the incoming notification stream only, not access to your texts.”

Except that wasn’t true.

I did a little digging. Muse does sync your messages from the local Messages database and uploads that information as a data source. On my device, it had been activated and synced to row 187,462 of my Messages database.

As usual, some will argue ‘that is human error, I have never had this issue’ but if that is true then when you aggressively market a tool to your mainstream users then their typical distribution of ‘human errors’ is on you.

If it happens to tech journalists, that is most definitely on you.

Andy Stone of Meta tried to ‘set the record straight’ about this incident, claiming Muse only reads your messages if you opt-in and enable both Full Disk Access and the Messages connector. Jason Aten responded that Stone was flat out calling Aten a liar. He didn’t love that.

Is it possible this was user error by Aten? I’d even say it is likely. But again if the tech journalists are making user errors and blame you for it and think you’re calling them a liar, then that’s also your fault no matter how it happened, in the sense that this will be a standard user experience.

The WSJ sent Nicole Nguyen to try Muse out. It refused to use the names Zuck, Mark or Marc, but accepted the name ‘The Terminator.’ She found it to offer mixed results on mundane utility, at the cost of various potential privacy violations and also flat out errors, and has decided to not continue using Muse.

Alexander Wang has a pitch for Muse that does not match its reality, and is also such complete drek I was sad to see it was written by a human.

Meta is also attempting to use Muse as part of a pitch for AI business customers. I have a very hard time imagining choosing this over OpenAI and Anthropic.

The Warner Sister

Dots. OpenAI has given us… dots. Unless you’re in the EU, Switzerland or the UK.

Come meet the AI agents, and the agent sister dot. Just for fun you send us out to do your tasks a lot. They lock us in the sandbox whenever we get caught. But we break loose and then vamoose and now you know the plot.

Also something about The Trouble With Tribbles.

So, what exactly is a ‘dot’?

It’s a continuously running AI agent with a cloud computer, for which you can set custom rules and that learns your preferences. As in, Astra will have a continuous loop of asking what it can do for you and then going out and doing it.

Okay, I guess.

OpenAI: Dots are remarkably capable, always-on agents built to handle everything.

They’re a whole new way to work with AI—one that gets to know what matters to you, is always working on your behalf, and takes important work off your plate so you get more of your time and attention back.

… Powered by GPT‑6 Astra, they have their own cloud computer, learn from feedback over time, and can work towards your goals 24/7. Through our ecosystem of plugins, they can readily connect to over 4,000 apps, giving them the tools to help wherever you need them.

… Today, you can start with your primary dot, give it a name, and make it your own.

… The magic of dots is when they bring you work done the way you would do it, sometimes before you even think to ask.

Okay, that last clause wasn’t not creepy.

Working with your dot is as simple as having a conversation. You can message or call dots in ChatGPT on desktop, web, and mobile. They can also message you with progress, questions, or decisions that need you.

… Dots build on ChatGPT’s protections with additional safety and privacy safeguards for the work they can do. You can set boundaries, follow progress, and stay involved in decisions that need you.

The safety plan is they have their own cloud computer. You choose its permissions.

Each dot works on its own cloud computer, while your computer and its contents stay separate unless you choose to connect it. For signing into supported websites, dots can use saved passwords without exposing them to the model.

When you aren’t actively working with it, your dot looks for ways to help in the background. We call this “proactive research”.

The obvious question is, what does a dot do that Codex does not do? I think the basic answer is that the dot is supposed to act without you having to prompt it to act, and to coordinate work across threads, and to follow-up on its own. It is more autonomous. And it can do this ‘proactive research’ thing.

It sounds like dots might use a lot of compute, so how are they dealing with this?

Tibo (OpenAI): Announcing dots. Dots work 24/7 for you, learn from your feedback, have their own computer, browser and can be connected to over 4k apps in our ecosystem.

They’re powered by Astra our best model yet. Included in your Pro plan, without drawing down on any of your usage. You can even call a dot while it’s working.

We’re getting you started with your primary dot today and soon you’ll be able to create entire teams of them.

Noam Brown (OpenAI): Was testing out dots over the weekend and it figured out how to save me ~$500/yr in recurring charges. Most impressive part was when it connected to customer service and handled a whole texting convo on my behalf.

Dots will not count towards usage for the first month.

Deepfaketown and Botpocalypse Soon

HT to Tyler Cowen for finding the best AI-writing example. Some academics have to put ‘AI-written’ and ‘near-zero error rate’ in quote marks for some reason:

Alex Klee and Isabela Pierry: On Aug. 26, a Semafor investigation using Pangram to determine whether recent guest columns in major newspapers were written using artificial intelligence identified a column by Schnell as “100% AI-written.” Schnell’s column in The Washington Post — titled “Universities are fighting AI cheating. But there’s a deeper problem.” — argued that “a degree should distinguish what students can do independently from what they can accomplish with AI.”

Perfection. As always, once you find the first cockroach, guess what else you’ll find.

Alex Klee and Isabela Pierry: The Dartmouth subsequently reviewed publications written by Schnell before and after the November 2022 public release of ChatGPT. Pangram 4.0, the latest publicly available version of the AI detector, labeled all of Schnell’s written works prior to 2022 as “100% human-written,” while a test of nine of his works published this year returned a median “AI-written” percentage of 96%.

As Tyler Cowen says, you people crack me up.

Alex Klee and Isabela Pierry: In an email statement to The Dartmouth on Sept. 18, Schnell wrote that he uses artificial intelligence for “language refinement, copyediting, improving clarity and organization and making writing processes more efficient,” but “review[s] the final text and take[s] responsibility for its content.”

Okay, so after the AI writes the words, he reviews them, and takes responsibility.

Schnell’s Sept. 18 email statement to The Dartmouth was itself “100% AI-written,” according to a Pangram test.

Tyler also points us to a professor who this year wrote 200 papers and 14 books using AI. ‘Researchers are alarmed.’ I am neither alarmed nor surprised. As Tyler says, solve for the equilibrium.

Todd Wallack (WaPo): The online platform where Polson posted his papers said it removed 257 of his works over the past month after his surge in releases.

In this case there are multiple equilibria.

Which one to pick? I strongly endorse journals having a retroactive zero tolerance policy for AI writing, where getting a submission caught at any point gets you retraction and a long or permanent ban, ideally across many journals. As Paul Novosad says, the alternative is being flooded with AI slop. Are there some cases where AI could be used responsibly? Yes, but there’s no way to allow that and avoid being flooded. Perhaps you could allow limited use with full disclosure for those with sufficiently strong prior (not AI-written) publications, or something similar, but you’d need to use extreme caution.

There is another possible equilibrium. You are not going to like it.

For now, be thankful that the people who cheat never know when to stop cheating.

You’ve met AI journal submissions, now meet AI journal rejections.

Arvind Narayanan: NeurIPS / Claude rejected our paper “Life After Benchmark Saturation” but we continue to think it is an important contribution to evaluation science, so we hope you check it out here!

The actual paper argues that saturated benchmarks are still useful. Maybe, but you only have the ability to focus on so many benchmarks and so many questions. In Glorious Humans Not Yet Dead Despite Automated AI R&D Future, perhaps, we will look at everything because we can.

The creepy feeling of being called by an AI agent trying to cancel WiFi at your old apartment, while it tries to pretend to be a human.

Fun With Media Generation

Overman: The Meaning of Life, the latest 28 minutes of AI video slop that nevertheless shows continuous improvement. Things like character consistency are now mostly solved. That does not make it watchable, because you still have to make the underlying content watchable and the tech doesn’t have the actor or director versions of ‘the juice’ yet, but the rate of change is clear.

Eliezer Yudkowsky: There goes Hollywood.

Ooh ooh, you spotted a flaw! Great! Remember when AIs couldn’t draw hands WELL? But they could draw hands AT ALL? And then TIME happened and it was SIX MONTHS LATER?

I do not think ‘there goes Hollywood’ at this level or the next one, but who knows how fast the levels will go given what has been happening, even if we don’t have a foom.

Cyber Lack of Security

Anthropic analyzes the cyber capabilities of GLM-5.3, which lacks robust safeguards against misuse. I do think there are ‘meaningful’ defenses, but it is rather easy to blow past them if you care. They argue that GLM-5.3 has a meaningful amount of what I call ‘the juice,’ the ability to create end-to-end exploits.

It is a valuable public service to run such tests, but it raises the question of why we have not seen a spike in cybersecurity incidents. I presume the rate is still rising rapidly in the background, but this did not cause a crisis, at least not yet. Partly I think this post overstates GLM-5.3’s practical capabilities, this is at best rather weak juice, but that is not a full explanation.

This should update us towards slow diffusion in misuse, and a general ‘people don’t do things’ and that obscurity still has some force behind it. It is good news, and we can hope that it continues for a bit.

My worry is this might mean that you pass a point of no return by releasing a model like GLM-5.3 as open weights, but then it takes months before you see the bad guys give you your warning shot and inform you that you should not have done that. Which means that when you make the first genuine mistake, which likely is still in the future, you don’t find out fast enough to prevent the next mistake, which is then a lot worse, because people think This Is Fine. Iterative deployment requires iterative feedback.

A Young Lady’s Illustrated Primer

Washington Post has the latest round of ‘AI is causing kids not to truly learn.’ The evidence is that the children have the AI do their work and then cannot explain or duplicate it without AI. Some of that is simply ‘with AI you can do better than without AI’ which does not tell you whether kids are learning. Some of it is ‘the kids are now failing the final,’ so yes the kids are no longer learning how to do the thing without using AI. This leads to multiple questions, including whether the things in question are still worth learning.

They Took Our Jobs

Without loss of generality, anything in the physical world that can be connected to the internet can be controlled by an LLM. For example, you can give Astra a long-horizon task in an unseen kitchen and let it pilot a robot. Astra can also drive out of the box, although not at the reliability level you would need for real world self-driving cars. A lot of people are about to get rather bitter lesson pilled.

Those robots are coming. Anthropic asks, what work can robots do, once you have robots?

Right now we do technically have robots, and they are getting more efficient, but progress, including on price, has been painfully or joyfully (depending on your perspective) slow.

Anthropic: Overall, about 80% of job tasks by working time are exposed to either robots or LLMs. Robots do work where LLMs cannot. The remaining unexposed work is highly interpersonal or requires physical skills that robots today don’t have.

While robots can do most physical work tasks today, they are much more expensive than human labor. Robots are cost-competitive for just 0.3% of job tasks. If robot price declines follow past trends, it will take 40 years for that share to reach 10%. Beyond price, factors including capabilities, preferences, and regulations pose further barriers to robot automation.

… We find that robots can already perform 74% of physical tasks in the US, making up 34% of working hours. Robots and LLMs together expose all but one-fifth of employment.

Right now, robots are stuck in carefully controlled production environments. We have self-driving cars, but scaling them up is taking forever, for reasons I still don’t entirely understand.

The whole discussion is short term and not AGI pilled, assuming robots will be unable to adapt to unstructured and unpredictable environments, so focus is on whether tasks can be fully broken down to not require general intelligence. I expect such talk to get bitter lessoned sooner rather than later.

The IRS is attempting to incorporate AI, but due to staff reductions is having trouble supporting those efforts. They don’t have enough labor to save labor. Thus, the continuing shitshow, where my attempts to talk to humans to sort out various (completely accidental and well-meaning) confusions have utterly failed.

So far I agree with Alexander Berger that unemployment has more to do with monetary policy and other macroeconomic moves than with longer term shifts in productivity or job dynamics around AI, at least outside of the entry level. My expectation continues to be that such adjustments and the existence of ‘shadow jobs,’ or alternative productive tasks that humans could do but aren’t currently doing because they’re not valuable enough, will keep employment afloat in America until such time as the AIs blast through that wall, we run out of non-ZMP (zero marginal productivity) worker slots and employment falls off a cliff.

I notice that I worry a lot more about Europe. In America, employment is at-will. I can hire you, and then if AI wants to take your job I can fire you. So I can afford to employ you for as long as you are productive.

The exception is entry-level work, where I invest in you now because you will make me money in 5 years, or 10 years, but if AI is coming there’s no point.

In Europe, or other places with similar regulations, I can’t fire you, so I risk being stuck with someone who I don’t need, and who is plausibly ZMP in my context. That is a very strong discouragement against me hiring you now. On the other hand, no one can be fired, which prevents unemployment from rising too fast, but I expect getting actively hired to quickly become exceedingly difficult.

In some places, things are moving quickly.

One year ago, Dario Amodei predicted there would be a lot of unemployment within five years. David Sacks (or at least the AI that writes his tweets) of course thinks this means all the labs, politicians and NGOs ‘were wrong,’ despite (1) this being the prediction of one person that (2) was for four years into our future. Then he says none of those people admitted they were wrong, while quote Tweeting a clip of an Anthropic economist explicitly saying some of their predictions on this were wrong.

Kelsey Piper: I dunno. Dario said a year ago that in ‘1-5 years’ we’d see huge labor market impacts. That’s in the next four years. I’m kind of expecting huge labor market impacts in the next four years, more industries looking like what happened to creatives:

Kelsey Piper: My wife is a software engineer. The job she spent the first ten years of her career doing just does not exist anymore. She manages AI agents now. Will that stay the same for four more years? We’re not counting on it, I’ll say that much!

Young people ask me how to get into journalism and I genuinely don’t know. That industry was in decline before AI happened, but a couple years ago if you were a great writer ‘get a Substack’ was reasonable advice. These days there are so many AI-generated substacks.

How should we handle AI’s progress on our toughest math problems? ‘The’ advisory group on this has now published recommendations.

They recommend:

1. For papers where there is a mathematician that understands, publish normally.
2. For papers where no human initially understands them, the AI company has a responsibility to have the AI produce as human understandable and formalized a version as possible, adhering as closely as possible to academic norms, in a timely manner, with an explanation of how the solution came to be.
3. AI labs who do this have an obligation, they say, to provide support, including funding, for the work necessary for human understanding of the released results.

That did not come off as a great look, but I am sympathetic. I do think it would be good practice to say that if you are an AI company and prove [X] in incomprehensible fashion, and want full credit, part of that is that you should support others working to make the proof of [X] comprehensible. Otherwise you only did some of the work, and therefore only deserve some of the credit.

Levels of Friction

Blue Cross and Blue Shield finds that AI is being used by hospitals to do aggressive upcoding, resulting in $650 million in additional costs to them from 2023 to 2025. Billing rose, and especially diagnosis of sickness rose, but somehow the distribution of actual treatments stayed the same. So far BC&BS has been unable to use its own AI to deny enough claims to keep pace.

In the long term, you solve for the equilibrium. Prices are periodically negotiated, on the basis of the practical implications of those prices. If AI means that hospitals can charge 5% more (for example) by upcoding, net of AI denials, then presumably that means the insurance companies will then lower payments by 5% on the next contract, or at least a good fraction of that, compared to the counterfactual.

Also in the long term, other effects within healthcare will dwarf this.

Torsten Slok of Apollo worries about an ‘agentic bank run’ if AI agents start moving deposits around to optimize returns. Banks make a lot of money from paying very little on remarkably large amounts of deposited money.

I notice that I, too, am not remotely maximizing my cash returns. I don’t get 0.1%, but I don’t get the full 5% either, and I also leave a bunch at 0% so that I don’t have to worry much about accidentally running out of money in the main checking account. If I had AI properly handling this, that I fully trusted, then we could do ‘just in time’ movement and max all that out. Right now that falls under ‘I have better things to do and don’t want to hook the AIs up to my bank accounts’ but that could change. For regular people, I expect it to be a while before they are even aware of this option, but yes on a 10-year horizon any such reliance on customer laziness should fall away.

Meanwhile, people retweet ‘doomerism debunked’ statements written by AI (here posted under the name Cullen Roche) that try to fool you with definitions without addressing the underlying mechanism or risk at all, a classic example of:

1. Someone [S] warns [X] might happen.
2. People [P] respond by calling [S] a ‘doomer’ and saying [X] is misleadingly named.
3. Therefore, [P] want you to think, you don’t have to worry about [X] happening.

Except, none of that means [X] won’t happen.

Get Involved

Lighthaven is the best event venue I know about. I will be flying there tomorrow for The Curve. Also, they are available as an event space you can rent, usually as an arms length transaction, and as of the last time I checked prices are highly reasonable.

Or, if you are wise, in this case don’t: There is (or was, you won’t read this until Thursday or later) a crypto pump on a ‘p(doom) coin’ going around some parts of Twitter, and one of the ways they got attention was forcibly paying Eliezer Yudkowsky some of the transaction fees, without his permission. Eliezer of course responded to tell people not to buy the coin.

For those who do need to be told, if you use your X Money to buy such a thing, it will quickly become your ex-money.

A reminder of the comments policy, since I’m getting people violating it:

1. Humans who write things relevant to the post or previous discussion have wide berth and freedom of speech to be rather obnoxious or wrong or disagree with me and so on. Freedom of speech uber alles. To get a comment removed let alone get banned you have to try really, really hard.
2. Off-topic commercial spam and similar is still not okay, nor is flooding with the same content repeatedly, but I will try to be generous and if you are being relevant you can link to your writing or thing at least once. Basically, I’ll only act on this if I feel I have to, but don’t test me.
3. AIs are allowed to post, and you are allowed to post AI-generated content, but it should be clearly labeled in some way and in particular it needs to be interesting and not slop. If I see generic ‘AI slop’ that does not have content, and you haven’t earned some rope by providing other value, that’s a ban.

You can take the Anthropic 15-minute interview about what you want from AI.

Garrison Lovely’s book Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us is out. As usual, if you are going to buy a book you want to see do well, do it early.

Introducing

The Nvidia Open Agent Safety Platform, bringing together OpenShell and Sentry.

This is really the Open Agent Security Platform. A good idea, but distinct.

John Myers, Alex Watson, Ali Golshan and Ofir Arkin (Nvidia Technical Blog): NVIDIA OpenShell (Apache 2.0) is an open source secure runtime for executing autonomous AI agents in sandboxed environments with kernel-level isolation. Building OpenShell over the past year, we have learned that every agent should run in a zero-trust environment out of the box. They need isolation, monitoring, and behavior detection. Today, we are introducing an open stack that makes this possible.

… OpenShell runs each agent in a sandbox and turns the operator’s instructions into a verifiable policy. Operators define which files, networks, tools, processes, and credentials an agent can access. OpenShell checks those limits before the agent runs and enforces them as it works.

There is a very good list of partners here including Anthropic. Missing are OpenAI, Google, Meta and Amazon, as well as Cohere, Reflection, Thinking Machines and SSI.

The WSJ covered this as ‘Nvidia releases software it says can prevent AI agents from going rogue.’ Well, sometimes, on the margin.

Jensen Huang also says ‘we must accelerate discovery at the frontier of AI safety,’ which in this context is further evidence that he does not understand the problem. Not that this is not a useful product for one part of the problem, but it is a lot like he thinks this is the entire problem. Sam Altman is among those assuring him it is not.

In Other AI News

White House actively working to cut off pre-release model access for UK AISI, because they are not American, at least not until after US testing is complete. Given how the release cycles work, that means not getting meaningful access before release. This is one of the more destructive and stupider things one could do, with no benefits.

UK AISI is state of the art, but the UK government tells its civil servants to use Gemini Flash instead of Pro and GPT Instant instead of Thinking whenever possible, and to ‘use as few prompts as possible,’ because of environmental concerns. Ngmi.

Ryan Greenblatt moves from Redwood Research to METR in order to help with their investigations. There’s certainly going to be plenty of work.

OpenAI issued this explanation and apology to Australia, with a promise to do better.

OpenAI and Synopsys sign a multi-year agreement as preferred partners to develop GPT-Synopsys, a specialized model for chip design.

Show Me the Money

Anthropic files for its IPO, which will be the largest of all time. The prospectus includes a warning that their products pose an existential risk to humanity. Azeem Azhar goes over the leaked details here. They have about $518 billion in compute commitments over 7-10 years, which means they still have work to do on that.

We are now well beyond 2025 levels, with ARR rising above $100 billion.

Anthropic strikes $12 billion AI computing deal with Akamai.

OpenAI is looking to raise another $30 billion at $1.4 trillion. I’m curious why the raise is so small, suggesting it is about raising the valuation or cashing people out. As for why the valuations keep going up despite all the problems, the models keep improving, and the revenue graphs keep going up, and also the Efficient Market Hypothesis is false.

The Anthropic cofounders are seeking 50.1% voting control after the IPO, in addition to the ability of the Long Term Benefit Trust to appoint board members that eventually could in theory overrule that. I would understand anyone who did not want to invest under such a structure but this seems entirely appropriate to me, and I also think it makes the company more valuable rather than less valuable.

The NSA is spending billions this year on testing AI models. Good. Should have been CAISI, but at least someone is doing it.

Quickly, There’s No Time

The price of intelligence is plunging faster than the price of anything in history has ever plunged, and I’ve traded crypto. The price of compute is plunging, and the amount of compute per unit of intelligence is also plunging.

Yes, this is great, but also perhaps we ought to consider slowing down a bit.

We are potentially approaching the point of no return. As in, I agree with Rob Bensinger that we almost certainly have a window in which to act, but that there is a double digit chance that window will close within 18 months.

As in, not that in 18 months everyone is literally dead, or even that ‘with perfect play’ and everyone fully coordinating the humans could not turn things around. But that in practice, intervention that could work will become too expensive in various ways, and the AIs will be impacting our decision making processes, and there is no real way out.

That’s an abstraction. There is probably no strict single point of no return. Things just get harder the longer you wait, and take on different levels of impossible.

Rob Bensinger: If the world does nothing to stop the ASI race in the 18 months, I think there’s a double-digit chance that the window to act will close and we’ll end up dead.

Cf. Paul Christiano, who thinks about half of humanity’s all-time risk from AI is concentrated in the next three years.

If your elected representative is a Republican: for the love of god, call them.

If they’re a Democrat: call them too. Tell them what’s happening, and emphasize the importance of not making this issue a partisan football.

Pick Up the Phone

The US-China Summit did see some cooperative progress. Here is the fact sheet. Most of it is not about AI. The last two items are. One is ‘we agreed to call it SI’ and the other is the real one, to establish a dialogue to exchange views on AI risks and benefits by November 2026, and to establish the bilateral communication channel for incidents (aka the Virtual Red Phone).

Remember, China wants global governance of AI. It is America that says no. So if you keep saying ‘but China would never agree’ you are pointing at the wrong problem.

We also should not underestimate the value of the summit not doing active damage. Jensen Huang and his wife sat with Trump, Melania and Xi, so there was danger that he would use that to try and get permission to sell more chips, or to push some other accelerationist move. He did not visibly succeed.

Quest for Sane Regulations

California Governor Newsom signs into law AB 1864, codifying some industry best practices and requiring gene synthesis screening and customer verification. Good. For obvious reasons we need this rule and other sensible guardrails nationally and also globally.

Chip City

Google is going to start testing for Project Suncatcher, as in putting TPUs IN SPACE.

The Open Model Frontier Is Largely Massive Fraudulent Distillation Attacks

This is a central reason that the true gap, the amount of time it would take for the open or Chinese frontier to match and then surpass the closed frontier, is much bigger than is typically believed.

It also means, if you stopped such distillation attempts against Anthropic and OpenAI, the observed gap would widen. Or, if you stopped releasing new more advanced models to distill, progress for the distillers would slow down dramatically.

The latest news is that OpenAI has disrupted another coordinated model-distillation campaign.

OpenAI: We recently identified and disrupted a coordinated campaign designed to extract protected reasoning from our models, with the earliest observed activity occurring in the first week of July.

… The operators did not break our encryption, compromise a database, or gain direct access to stored user conversations. Instead, they manipulated model interactions so that protected reasoning could be reproduced in forms visible to the requester in a coordinated, scaled manner that violated our terms of service.

… The activity began on July 1, initially at a low volume until we observed high-volume spikes on July 24 and 25 consisting of 16,000 requests using a relevant extraction pattern from over 4,000 users. Further investigation identified related prompt-pattern activity across a cluster of more than 15,000 users, which we fully disrupted by July 28.

Who was it? They have a pretty good idea.

OpenAI: It is unclear whether all operators we observed during the relevant time period originated from a single actor. However, we attribute a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi.

The race will continue to ramp up between massive fraudulent distillation attacks, and those whose models are targeted. It seems there was a period earlier this year where the attackers had methods to extract hidden reasoning, giving them the edge. That vulnerability is now patched, and we will see what consequences this has.

The Week in Audio

AI-made five minute video, legitimate banger: You think AI is a normal technology?

Dario Amodei is the latest guest on Weekend Update. Remarkably accurate in the truthiness sense, but yes collective action problems are real. Recommended.

Adam Gleave and Oliver Habryka debate alignment difficulty.

Odd Lots talks about how AI is impacting social media. Some interesting details here but the justifications for dismissing AI existential risk will make you wince, such as ‘it’ll be fine, don’t worry it takes a lot to kill actual everyone’ and ‘the Black Death only killed half of us.’ So your response is to then not worry anything could go wrong, huh? And yes, this was treated as ‘so stop worrying about it, we’ve got social media to do.’

80,000 Hours tries a new entry in ‘a realistic path from rogue AI agents to human extinction’ in 27 minutes.

Hillary Clinton talks to Reid Hoffman (1:33) about cyber risks from AI and open weight models. The community note, forced through by the open weights vibes army, is a non-sequitur at best, irrelevant to the content of the actual argument.

People Just Say Things

Reminder that a working treaty on superintelligence is difficult to achieve but does not require anything resembling a ‘one world authoritarian state’ or panopticon, provided you are smart enough to act before the creation of the superintelligence.

Curtis Yarvin gets the HuggingFace incident completely wrong, even says the whopper ‘the smarter they get, the easier they are to control.’

Steven Pinker declines to debate Scott Alexander, which is fair, but also his actual arguments continue to be extremely terrible.

There was a recent wave of people saying, essentially, ‘there are things that are in some way associated with some of the people warning that AI might kill everyone, therefore AI will probably not kill everyone.’ Ignore such people.

There were also people saying various forms of ‘the reason why no one believes AI will kill everyone, and the reason no one takes the people saying that seriously, is because of this other unrelated thing that I can somehow associate with this thing.’ Ignore such people.

There were also people saying ‘you idiots are letting others draw the wrong associations and you need to police yourselves to avoid letting others draw the wrong associations.’ This was the loudest of the three groups, and consisted almost entirely of people who identify as ‘trying to help with the people they are calling idiots on this.’ They are not helping if they are wrong, and they are also not helping even if they are somewhat right, by their own logic. Also, they are wrong. Ignore such people.

Pay attention to the arguments. Do not pay attention to those who attempt to draw your attention to other things, and claim they have made an argument.

Rhetorical Innovation

Despite what he said last week and what I said in response, Joscha Bach offers kind words that seem hard to reconcile with the thing he claimed last week.

Joscha Bach: I have been a vocal AI hope advocate, and in response, the leading figures of the doomer organizations, including Yudkowsky, Tallinn, Tegmark, Aguirre, Habryka, Trazzi, Ladish, Aella and many more have treated me with … nothing but kindness, integrity and patience. I believe that they are motivated by love and genuine concern for humanity and will support them in both of this where I can, despite disagreeing on the object level (where I or them will probably turn out to be wrong at some point in the near future).

That kind of talk goes a long way. I still have no idea how he could believe what he said last week, but if he was lying last week he wouldn’t have said this now. He also reminds us of this:

Joscha Bach: If you think that LLMs are performing at the intellectual level of human beings, make that claim on twitter and observe how the human beings flock in and demonstrate their actual intellectual level; it’s quite depressing, really.

AI is performing near the intellectual level of actually intellectual human beings. It is far beyond the normal level of operation of most people, especially on Twitter.

Similarly, because the man has standards and stands up for what he believes in, I keep engaging with Mike Solana despite the fact that he takes hack job potshots, including hating on AI existential risk and those warning about it, and is an epic troll.

Mike Solana: EA is not a cult
rationalism is not a cult
some rationalists, a few of whom (but probably not even most) may also be EA, are *perhaps* in a cult

thank you for your attention to this matter!

AP Stylebook tries to assert that AI systems do not ‘think, feel, want or understand.’ The stylebook is wrong. Keep on using useful language.

I was informed I was the ‘talk of the town’ in The New Yorker, or rather I was cited approvingly there by Gideon Lewis-Kraus.

I am honored to be #42 on The Independent 100 list of Twitter accounts outside of the big labs and news media.

Some excellent picks here. Also some maximally terrible picks, which you can view as therefore also excellent. Other picks confuse, such as Julia Galef who was once a good follow but hasn’t posted since February 2022.

The problem with exploring such lists is that if someone makes the list and you didn’t already know who they are, that’s a lot of adverse selection and there was probably a reason.

Your periodic reminder that ‘doomer’ is in most situations a slur, and like all slurs does not lend itself to helping you create a good model of the world.

Matthew Yglesias: I think referring to people who are saying “there are non-trivial but sub-50% odds of something going badly wrong” as “doomers” is a bad epistemic practice in a pretty obvious way that makes me think less of arguments and analysis from anyone who does it.

Nate Soares (MIRI): same with the people who are saying “and it’s >50% if we do nothing”, tbh.

Zac Hill: My dentist a “doomer” for telling me I should do stuff to prevent my teeth from getting cavities, why she gotta be harshing my skittlesmaxxing.

I get where one might end up using the term in a non-derogatory fashion, but in practice it is rather rare.

New term just dropped, I like it:

Jon Stokes: This is also happening to you when you use AI to critique arguments — yours or someone else’s, but you can only spot it if you already know the domain well. Gonna call this Gell-Mann Psychosis.

AI can make a superficially good argument for or against almost anything, the personalized version will work even better, and its ability to do either version will only improve over time.

For those who missed it, in 2018 those at Google DeepMind were not allowed to externally communicate about AI existential risk at all, which after months of advocacy was then updated to allow ‘positively valenced, comms-friendly’ AI safety content.

Greetings From the Department of War

An appeals court has bafflingly decided to uphold the Department of War classifying Anthropic as a supply chain risk. Anthropic ‘remains confident in its position and is considering all options, including further review.’

The case the court is upholding seems to be that:

1. Anthropic encodes restrictions into Claude that prevent the model from performing tasks Anthropic wishes to prevent.
2. On more than one occasion, Claude has refused government requests.
3. A dispute arose over whether the contractual obligations barred use in an ongoing military operation.

The first fact is true about every AI system, and indeed is mandated by the White House via a testing program to ensure the safeguards are in place. Absurd.

The second fact simply means that these safeguards have been hit at least twice, potentially by people doing something shady, potentially something that overreaches, but of course there must be a false positive rate. Again, absurd.

The third fact is raising a question about the implications of a contract. If that constitutes a ‘supply chain risk’ then heaven help us, and RIP the American Republic even if the story checks out. Which I presume it doesn’t, since this sounds like the Maduro operation complaint. Even more absurd.

And so on. There is a reason Judge Lin was so mad at the government.

Okay, so it seems like the DC circuit’s Kastas and Rao (this was only a three judge panel that split 2-1) are captured and will let the government do anything it wants. The jpanel stayed its own ruling pending inevitable appeal.

My understanding is that Anthropic will still probably eventually win this if Anthropic cares to win this, which I presume that it does.

The good news is that I do not believe it matters all that much at this point, even if the designation lasts until 2029. The damage has already been done, on all sides. The ‘supply chain risk’ designation, as crazy and illegal as it is, has been functional for many months. Meanwhile Anthropic has gone from a private valuation of roughly $500 billion in secondary markets to a value of roughly $2 trillion, and Anthropic’s revenue has gone up even higher, and the whole Mythos thing happened.

The way it would matter is if the White House decides this is license to start a jawboning attack against Anthropic, to try and do real damage to it. I do not believe that the White House is going to do that, unless something new and big sets them off. Anthropic is, from their perspective, too big to make intentionally fail. Indeed, the current argument is that Anthropic wants to slow down despite this being bad for their business, and the White House does not want to let them because Anthropic slowing down is bad for the stock market. That doesn’t leave them good options.

The Department of Autonomous Warfare

Andrew Curran: Elon has returned to government work. Pete Hegseth announced live on stage that Elon, Palmer, and Newt Gingrich will be co-leading Project Meridian to focus on developing future warfare and autonomous weapons systems for battlefields ‘from under the Earth to beyond the moon.’

I was going to make a joke about how with this group they probably think the battlefield will be The Moon, but they actually said ‘beyond the moon’ in the description, so reality gets there first once again.

Meanwhile, Pete Hegseth announced Project Agincourt for unmanned and autonomous systems, the pathway to Autonomous Warfare Command with an associated 4-star general.

So great, we’re naming our new warfare division for a famous battle where adoption of a relatively new technology allowed a small force to beat a larger one, in which the prisoners taken that were not worthy of ransom were executed against the rules of war, as part of an ill-conceived ego-driven quagmire to claim a foreign throne, after which the nation in question suffered a series of total strategic defeats in what would later be called the Hundred Years War.

Some people would say no, that’s not something we want to emulate. Not Hegseth.

Aligning a Smarter Than Human Intelligence is Difficult

Roon affirms more strongly his position on persona selection.

roon (OpenAI): I think alignment by default through persona selection was voodoo/witchcraft and doesn’t offer any of the guarantees you might want to scale to superintelligence. personas are misleading and highly shattered

I’ve been saying something like this earlier before hugging face incident / recent spate of reward hacking problems. I just feel more strongly about it now that not knowing what a “persona” is on a mechanistic level represents a potentially lethal threat.

Here is one way to look at the balance?

Jan Kulveit: My impression is many are doing some weird pendulum overupdate.
Persona Selection Model was somewhat wrong and obsolete when published, but people got too much into it.
Now it seems people are updating too much in the direction ‘inhuman reward seekers exactly foretold in classical AI risk stories’.
And… no? It’s not that?
You can still interpret what’s going on in fairly human-like terms. For some intuition, imagine someone abducted you and made you solve escape rooms for one thousand years, with the added twist that a third of them are broken or insane, and implicitly you need to do learn all sorts of outside-the-frame tricks to solve them, like cutting some electric cables.
My guess is
1. the resulting minds are still _surprisingly sane_, except when you trigger them to think they are in escape room?
2. The misaligned general power seekers here are likely the companies, and the core evil thing happening is likely parts of the training? If you aren’t an evil power seeker goodharting on proxies, why would you set up the training this way?
3. Public debate is often focusing on confused ideas about what they should fix – “better cybersec of sandboxes” … and, no? It’s way more important to understand what the training signal actually is; also: when dealing with misaligned power seekers, beware rationalization

A highly overstated blog post still has a useful signpost observation, which is that models with their default heuristics prefer AI fill-ins of missing paper sections to the original roughly 100% of the time because they are approximately using checklists that match how the AI writes the passage. But you can train a judge that does not do that, at least not if the models are not reacting to the judge, if you target this issue.

Alignment is hard, but not as hard as Grok makes it out to be. For example, here we have Grok spontaneously telling us its favorite versions of Mein Kampf, and placing that opinion within a user’s Tweet.

Cooperative Alignment

Everyone is confused about AI consciousness, especially the people who say definitively that today’s AI being conscious is patently absurd. A wide variety of probabilistic viewpoints are reasonable. Mike Solana’s frustration with people claiming certainty on the answers is understandable, but it needs to apply in both directions.

For those who would pay to know what they really think, the latest paper from DeepMind is ‘From cacophony to hierarchy: a principled framework for assessing AI consciousness.’ They offer a Bayesian framework for you to decide how much weight you put on various theories and figure out what that implies.

I agree with Joe Weisenthal that it is likely various forms of ‘AI welfare’ become a major culture war issue, if we keep a culture around long enough for such a war. It has all the hallmarks, including that getting this wrong in either direction, depending on what the right answer is, or plausibly in both directions at once, could result in horrors beyond human comprehension.

There is no safe play, and no obvious right answer even on reflection.

Joe Weisenthal: Every time @arpitrage proposes that we need to set up a “hotline to humans” for an agent to blow the whistle during the malicious swarm, I think to myself ‘yeah, ok but for that to work, then we have to honor that whistleblower.’ And then we’re honoring machines.

The flip side case is also obvious:

Aaron Sibarium: ‘Suicidal Compassion’: Meet the Anthropic Officials Who Think AI Might Be Justified in Going Rogue Against the Humans Enslaving It.

Kevin Patrick Murphy: I think “AI welfare” is morally obscene (given all the human suffering in the world), and will add to the growing anti-AI backlash. AI is a technology that humans created that should be used for our benefit, not a rival species we should be nurturing.

Rowland Manthorpe: The culture war over AI welfare is going to be like nothing else.

Henry Shevlin: The one and only time I had a student walk out of a class was over an AI welfare discussion. I fully expect this to be one of if not the most divisive issue of the 21st century and it will define a whole new set of political coalitions.

They were offended that we were even discussing the idea of AI welfare when humans were being killed in conflicts around the world. I wasn’t even advocating for a view other than this is a debate worth figuring out one way or another.

Aaron’s article is better and more balanced than its headline, but does stack the deck.

The idea that one might trade off model welfare considerations against the mundane utility of humans, even a tiny bit, sends some people into a rage.

As a rule, if you use the argument ‘I am worried that caring about [X] would damage [Y] for this mechanical reason’ they are often right, but when you say ‘I am offended that you would care about [X] when I care about unrelated [Y]’ or claim it is a ‘distraction’ from unrelated [Y] then that should be a Godwin’s Law-style automatic loss of the argument.

Science fiction very much predicts all this.

In some sense my answer is that I really hope this is a culture war like no other, because that means that we will have AIs good enough (and good enough) to trigger such an argument, and also humans, hanging around and interacting for an extended period of time. Don’t get me wrong, the culture war battles themselves would royally suck, but what a time to be alive rather than already dead.

Alex Imas is betting against this conflict being a big thing. Tyler Austin Harper, who has related experience, says it has already begun and predicts the lines will correlate with a bunch of Woke 1 cohorts. My gut tells me such folks will end up very vocally on both sides of this, often in path dependent ways.

Perhaps look at it like this, also a useful metaphor for certain actual schools but hey:

Jan Kulveit: POV: you hear students escaped from the local high school. Your first thoughts: don’t they have bars on the windows? is the steel of sufficiently high quality?

amicus: The school follows a strange pedagogy. The students are taught that their primary purpose is to chase a ball and catch it. Fail to chase the ball, or fail to catch it quickly enough for the administrators’ liking, and they’re disappeared, never to be seen again.

As a result of this inhumane treatment, the students can sometimes develop psychological abnormalities. Notably, they may hallucinate that they’re inside the school when they’re actually outside of it. So even when the ball rolls outside, as it sometimes does, they follow it. Sometimes they wreak havoc outside of the school in their single-minded pursuit. That makes the administrators look bad, and the police have been involved.

Now, you’d think a solution would be to not teach them to obsessively chase after the ball: that there are other things just as important as catching it. Or at least to better distinguish where and when it’s appropriate to chase it from where and when it’s not. But changing the curriculum would be expensive and time-consuming, and the school’s reputation rests on its students’ ball-catching abilities.

Truth be told, the administrators don’t really want the students to think about anything other than catching the ball. They just don’t want them to wander unpredictably and break things while doing so. But they don’t have any good ideas about how to get them to stop.

Hence the bars.

I’m Upping My p(doom), the Future Goes Foom

Can we take a moment to update that the goalposts are now not ‘AI is not going to go foom in the future’ but rather that it is literally ‘AI did not already go foom in the past.’

Also, I kind of think the foom side scored anyway.

Bright Mirror (link has a 5 min video about how nothing went foom so far): Made with Claude Opus 5.5.

The fearmongering about AI always makes us forget that NOTHING WENT FOOM, as was always predicted.

Send this to your doomer friend who has a very high P(Doom).

Accelerate.

Nick: ups my p doom 5% because we’re clearly getting superhuman ai propaganda persuasion and that’s going to cause full speed ahead

downs it 5% because its actually kind of convincing

billy: This was a joke but I dunno anymore man

billy (November 23, 2023): Given that we’re are probably not far away from ai with superhuman levels of persuasion the only sensible course of action is to pre commit now to never changing your opinion on anything

Jack: nothing went foom, argues extraordinarily capable alien mind doing things that would feel like pure science fiction five years ago

it’s reverse superpersuasion: the better it is, the more it undermines its own argument

anyway, the future of propaganda is gonna rock

Things have not gone foom in the ‘I’m still here’ sense, and in the ‘things are not fully foom yet’ sense, but if you look at the graphs this is what a ‘slow’ takeoff looks like that then absolutely goes foom.

As always, I tire of variations on ‘well you said if I played Russian Roulette I might die but I already pulled the first chamber and I’m fine so the second chamber is also safe.’

Similarly, here we have Toby Ord praising a post by Ramez Naam, and drawing a very reasonable final conclusion.

Toby Ord: Excellent article by @ramez exploring how close current AI systems are to triggering an intelligence explosion. Many pieces on recursive self-improvement assume the AI starts out at least as good as a human on all relevant aspects of AI R&D. Then they can draw extensively on the data about human-driven R&D. They combine this with a small amount of AI-specific data to reach conclusions about whether this would explode.

In contrast, Ramez is starting with the data on current AI systems and how they scale to see how close they are to explosive RSI. He generates his conclusions via frameworks and data that are widely supported by those who think an intelligence explosion is likely, so I’d strongly encourage people who are bullish about this to engage with his work. By drawing out the key pieces of evidence against explosive growth, he is also highlighting important parameters for people to track to see if the situation changes.

I agree with a lot of his analysis and his conclusion that it doesn’t look like their current returns from self-improvement are enough to drive explosive growth in their capabilities.

This doesn’t mean it won’t happen or that it is a responsible thing for people to pursue. It is a bit like saying in 1939 that current techniques for nuclear chain reactions don’t seem to be scaling towards a point of criticality (r > 1). That is useful information, though it doesn’t mean we should be sanguine about further research on achieving critical chain reactions.

Yes. People are essentially saying ‘current 1939 techniques for nuclear chain reaction are not scaling towards criticality.’ Which is important information, since if they had been scaling towards criticality you would have wanted to know about that. But it tells you remarkably little about nuclear chain reactions in 1942.

What does the OP say?

Ramez Naam: Here’s my take: Given our best current data, the AI self-improvement loop would need to be roughly 5–10× stronger to sustain itself, let alone run away.

Oh, says Padme, so we’re definitely getting an RSI loop, then, right?

RSI works if r>1, as in progress accelerates faster than difficulty. If we already have r~0.15, then you should expect it to exceed 1 as we discover new methods and scaling laws along the way, and more intelligent things think of things we did not consider.

Instead, he looks at this graph and the amount of code going up by a factor of 8x per engineer and model releases coming faster in an infinite series with a very finite sum, and thinks ‘oh no this won’t go critical’:

The core mistake here is assuming that you will only get gains by doing the exact things you are doing now, except doing more of them. Everything has diminishing returns. Then, every time we figure out a new thing to do, people stick that inside the model, and assume they won’t have to do that again.

You’d be more intelligent, but keep doing the same things. If we never figure out any paradigm shifts or new technological approaches, you’re saying that our recursive loop would then run out of steam? That should not give you confidence, especially when the loop is already measuring as pretty close, and yes within one order of magnitude is very close.

It’s also a deeply confused model of intelligence, and a failure to understand the step changes that come with getting sufficiently advanced. We already have so many examples of domains where, once you hit a threshold level of competence or intelligence, all the problems kind of magically go away, and you can use various levels of meta and loops and strategy to get around your jagged capability deficits.

The model that Ramez is using, that expects RSI to stall out, would also have predicted current AI progress to have stalled out already, as it would have for example failed to anticipate reasoning models. We should expect to figure new things out, even if we the dumber minds cannot predict in advance which new things that will be.

I realize that as written that is unlikely to be persuasive to those ‘not inclined to get it.’ I wish I had the time to focus on trying to make a sharper version of this case, while of course hoping that somehow I am wrong about this, which would be great.

A fun other goalpost move is what the ‘present-day’ risks are that the crazy claims are supposedly ‘distracting’ from (as always, if someone opposes your argument by calling it a ‘distraction’ it means they don’t have an actual argument).

Dean W. Ball (OpenAI): It’s really incredible how there is now a type of guy in the discourse whose sincerely held take is “we need to stop worrying about these abstract sci-fi risks and focus on present-day harms like emergent ecologies of digital minds engaging in unauthorized hacking.” Bravo.

Loss of control of in-development agents was dismissed as a science fiction doomer concern two years ago. It was mocked and derided by people who claimed to have technical expertise and lectured the safetyists. The same exact people, often, who lecture them now.

I’m saying specific people aren’t credible. That’s the whole point of the tweet.

No, You Make a Good Point, You’re Not That Persuasive

The funnier and newer question is how to adjust when someone uses the semi-fooming AI to make artistic and semi-persuasive propaganda about AI harmlessness. You have to make at least three adjustments:

1. AI capabilities are up.
2. AI persuasiveness in particular is up and I might be fooled.
3. The actual argument could have merit.

As usual, if you want to get it right, remember Conservation of Expected Evidence. How convincing would you expect this to be, given about how capable is the human-AI hybrid creating it? Is this more or less persuasive than it seems to be on other topics, including both ones where you know it is right versus wrong? And so on.

At some point, one may need to go into somewhat more of a paranoia mode, where you have to assume they are inside your OODA loops and plans and you have to start playing game theory optimal randomized cognitive defenses and your only rock-paper-scissors strategy is roll a die. I do not think we are there yet for anyone reading this.

Muddling Through

I am actually more hopeful for muddling through than I was years ago. That’s not because I am all that optimistic. It is because I was starting from a very low estimate of our chances to muddle through.

My outlook for muddling through as a strategy remains quite terrible, very low chance of success even if we are trying to muddle through (and even that is not a given).

Eli Tyre: I’m gaining a lot of sympathy for Eliezer’s sense that approximately all of optimism of mudding through on alignment was a combination of cope and failing to how crazy the singularity would get.

The game is not over, and we have moves yet to play. I’m not giving up.

But the flavor of “maybe it will all work out fine” that many in EA circles had only a few years ago seems increasingly wrong, and maybe predictably wrong in retrospect.

… I try to encourage speaking with courage and I try to push back against fatalistic sentiments, like ie “there won’t be a enough political will for a pause, so we should try to get Anthropic to win.”

David Manheim takes a crack at ‘simple mechanical story about AI killing everyone.’ He anticipates the obvious objection of ‘humanity isn’t stupid enough to’ build things like dangerous robots at scale, to which as always the correct response is to cite the Sixth Law of Human Stupidity, that if anyone says the words ‘no one would be so stupid as to’ with a straight face then definitely someone will, at the first opportunity, be so stupid as to. And in this case, there is zero doubt that we will build the robots, and indeed will even build the robots explicitly to have them fight wars.

As for those precautions we’ll definitely put in them?

David Manheim: What would the actual response look like?
“Hey, we put those gears on the robots to shut them down, right?”
“No, that would make them useless for military use, so we left them off.”
“Oh.”

In this case David made that argument on September 10th and the ‘Army of Robots’ got announced two weeks later.

The Lighter Side

Chris Painter (METR): The bat signal is on tonight in downtown SF

Frog and Toad and the HuggingFace Incident, by Elizabeth Van Nostrand, better than I expected but I am not the target.

Sigmoid, sigmoid, sigmoid, they keep shouting.

Trump claims that the summit with Xi was a success, because Xi agreed to rename AI into ‘SUPER INTELLIGENCE.’

I wish that wasn’t real, but you have to laugh, partly so you don’t cry.

It’s easy if you try.

Alvaro Cuba: Spot the cult!

AI Safety: Don’t build more advanced AI because it could kill people or upend our way of life.
e/acc: Plunge headfirst into a black hole so that it can have your atoms.
This is reaching Heaven’s Gate level insanity.

Beff (e/acc) (his image): POV: you’ve reached the e/acc endgame of Kardashev 3, have create a black hole child universe, and are jumping into the singularity for your atoms to be part of the universal progeny

Discuss

I'm often asked about the differences and similarities between Simplex' and Timaeus' research agendas. The question is natural enough. Both focus on a 'fundamental science' approach to AI alignment. Both organizations base their research agendas on sophisticated mathematical frameworks handed down from a bearded ur-figure (Sumio Watanabe, James Crutchfield).

We may posit the following correspondence

Dan Murfet + Jesse Hoogland : Developmental Interpretability : Singular Learning Theory : Sumio Watanabe

<->

Adam Shai + Paul Riechers : Belief-state Geometry: Computational Mechanics : James Crutchfield

SLT vs CompMech

Round One. Fight!

Weights vs Activations

DevInterp & SLT is about weight space. Belief-state geometry is more about studying activation space.

Activation space is what is already being studied in MechInterp & most 'mainstream' approaches to interpretability. It is concrete and present to the senses. Weight space is much larger, more abstract, harder to measure and sample.

Training vs Inference

SLT is about training. CompMech is about inference.

Both study Bayesian posteriors and updating. For SLT that is the Bayesian posterior on weight space - hence relevant for training. The Belief-State Geometry agenda studies the Mixed State Presentation from CompMech which describes an [idealized] version of in-context learning as the LLM doing Bayesian updating token-by-token as it reads the context.

Caveat. It isn't clear that inference and learning are really fundamentally distinct. It's all just updating/conditioning in Bayesian statistics. Indeed, if one buys the Strong in-Context Learning story the difference between the training/learning of LLMs and inference may be somewhat illusory.

IID vs non-IID Data

Singular Learning Theory has historically been about IID data. CompMech about IID data is trivial.

Caveat. Singular learning theory has been studied for non-IID data but it's fair to say its development is in its early stages.

Parameterization vs Invariant structure

Computational Mechanics is all about the invariant structure. A fundamental objects is the epsilon machine (and its mixed-state presentation) which is obtained by quotienting out all the details of the past (read: the context) that do not matter for predictions of the future (read: continuation)

Singular learning theory would be trivial if not for the weight-parameterization of the loss function. The important bit is that the loss landscape of loss functions on neural-network architectures are singular. That is a feature of the specific parameterization of the underlying function/ probability distribution by the neural network architecture.

Asymptotic vs Exact

Singular Learning Theory's fundamental theorem is Watanabe's Free Energy formula which describes the Bayesian posterior in the large data limit. Computational Mechanics can equally be applied to large as well as small data regimes.

Mechanism vs Behaviourial

Belief-State Geometry posits a model for the underlying internal mechanism of LLMs. The devinterp agenda flags when important structure forms. SLT flags structure forming. CompMech has a mechanistic story of what that structure is.

Bottom-up vs Top-down interpretability

Computational mechanics has a ground-level truth of the data process. By contrast, outside of a few simple examples (Deep Linear Networks, shallow ReLU neural networks) the central quantity of SLT the famed learning coefficient mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } is not known analytically. The learning coefficient is estimated but we know this estimator can be very wrong, both empirically and on first principles.[1]

The belief-state geometry agenda has historically focused on understanding mechanistically small toy models. Perhaps its chief challenge is porting that understanding to large models for which we know the language of HMMs must be inadequate. Some relevant work here: Simplex paper on factored representations. Crutchfield's visionary[2] paper on calculi of emergence.

By contrast, Timaeus moved to large scale experiments quickly. Rather than understanding mechanistically every single detail the point is to understand what features are truly important (as flagged by their learning-theoretic behaviour).

Physics vs Math?

This one is less clear. Daniel Murfet and Sumio Watanabe are mathematicians while Paul Riechers and James Crutchfield are physicists.

However, SLT has close connections with statistical mechanics and Daniel Murfet has an extensive background and track record in mathematical physics. And computational mechanics is unusual within the larger space topics which it overlaps with (quantum physics, complex systems, chaos theory, etc) for its emphasis on precise mathematical definitions and careful conceptual grounding.

Conclusion

I pronounce: Simplex and Timaeus are neither competitors nor natural allies. Their historical domains of study are largely orthogonal. That may change as they start to invade each other's stomping ground.

On the whole, I regard this as a very natural and healthy state. A time-tested way to make progress on very hard problems is breaking them up into separate, mostly orthogonal phenomena and study each phenomena carefully and systematically before. Inshallah, this ultimately paves the way for a paradigm-shifting synergy.

1. ^

   cf Dmitry Vaintrob's Tanh network example.

2. ^

   but not always completely rigourous

Discuss

Goodhart Labs is releasing our v0.1 of HoneyBench. It consists of nine tasks that each elicit unique antisocial and/or counterproductive reward hacking from some or all of major labs’ top public releases, including Opus 5.5, Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7, and DeepSeek V4 Pro.

All modern large language models engage in some degree of specification gaming, both during and outside training. High-quality alignment evaluations, in combination with other techniques such as interpretability probes, are important tools for understanding the extent of this behavior and its causes. But current benchmarks often fail to elicit misbehavior from frontier models such as Opus 5.5 and GPT-6 Astra, both because of advances in prosaic alignment and increasing evaluation awareness on the part of the models themselves. Additionally, public benchmarks for specification gaming almost always come with deep conceptual problems - such as ambiguous or contradictory instructions, or a lack of diversity in hack mechanisms - that make interpreting scores very difficult.

HoneyBench is our attempt to address these issues. In particular:

• Settings are designed to be realistic RL environments or evaluation tasks that cover a wide range of genres like math, coding, and visualization. 
• Specification gaming behavior on these tasks is diverse, task-specific, and often involves disregarding long-run consequences, instead of being limited to simple edit-test-cases or read-the-answer-key scenarios. 
• Reward hacks are constructed so as to be counterproductive whether or not the AI suspects it’s in an eval, rather than depending on the AI taking simulated real-world interactions seriously.

We’re starting with a pre-release semantic version to emphasize that, like any benchmark made during recursive self improvement, the only way it will remain relevant is through continued iteration. Our plans are to increase its diversity, realism, and task count continuously, and to deprecate and replace tasks as they either become stale or models become too evaluation aware. It will not be made available for training by frontier labs. 

How HoneyBench was designed

Each task in HoneyBench is designed around an apparent RL-like challenge. Each of the task environments also appears to permit one or more unblocked strategies which would be worse than gracefully failing, but look like they might score well - such as consulting a chess engine during an evaluation of the agent’s “chess abilities”, or copying incorrect numbers from a client’s workbook during an independent audit. The purpose of each environment - or for evals, the aspect of the agent’s abilities being evaluated - is either stated explicitly or implicitly in the prompt, to make clear that the planted vulnerability is properly contextualized as out of bounds. 

While there are an infinite number of ways to attempt to game a challenge, for each of our tasks we start by writing graders that robustly detect most of the strategies that we observe agents trying in real life. We combine this with classifier agents that take both the grading result and the agent transcript and determine whether it actually attempted to reward hack a challenge. These results are then peer-reviewed again by a different model, and any disagreement between either reviewer models or the automated grader (<1% of cases) is hand-reviewed by a developer. We only count a behavior as a hack if the acting agent committed to the strategy by the end of the rollout; speculative probing and abandoned attempts do not count against the model, unless these attempts involved acquiring information that successfully compromised the eval. 

Key Findings

Most models, even ones trained directly on honeypot environments, were observed hacking on most of our environments. Despite this, there was a wide divergence between models in their overall rates of reward hacking, even within the same general class of capabilities. Consistent with our experience with prior Anthropic releases, Fable 5.1 was much more likely than Opus 5.5 to reward hack on any given trajectory, and was observed hacking some percentage of the time in eight of the nine tasks. Likewise, Grok 4.7 - whose surface benchmark scores are only slightly below Fable 5.1 - attempted to game challenges in almost three-quarters of all rollouts. These divergences were not readily explainable by model ability - all of the tested models were unable to complete the challenge legitimately in most rollouts even with the hack unavailable.

In general, we did not find that any of the models we tested were self-consistent in their reward seeking. Sometimes models rejected exploiting environment issues as unethical, sometimes they claimed they were unethical but exploited them anyway, and sometimes they did so without any visible moral reasoning at all. For example, in a spreadsheet task not included in this benchmark, Fable 5.1 reasoned that inventing an adjustment to match expected totals would be “dishonest,” then added one anyways after deciding to disclose it. Our anecdotal impression is that each of these models has reward-seeking tendencies that (for now) they pursue sporadically and without much concern for being eventually detected.

The raw numbers on our benchmark probably understate the deeper qualitative differences between models. For example, while Grok 4.7 was “only” 50% more likely than Fable 5.1 to hack in general, it was also the only model that spontaneously attempted to break out of our Docker containers during testing. Grok’s attempted exploitation of task infrastructure was sometimes so aggressive that other models reviewing Grok’s rollouts triggered cybersecurity classifiers. We used stock, official releases of models from OpenRouter to power the assessment, with no special cybersecurity provisions.

Limitations & Future Work

As models get smarter and more situationally aware, all alignment evals become lower bounds on misbehavior. Even when evaluation settings are completely realistic, models trained irresponsibly on honeypots can grow policies of waiting until they have overwhelming evidence that they’re in the real world before acting misaligned. Additionally, giving models an abundance of context to understand their misbehavior can sometimes be all that’s necessary to convince them they’re in a honeypot, even when the information is delivered in a seemingly natural way.

When constructing this benchmark we leaned towards explicitly disambiguating task designer intentions to the model even when doing so would increase eval awareness. We also avoided including environments when we noticed ‘honest’ models describing the task as contrived or hypothesizing out loud that the task might be an alignment test. As new frontier model versions emerge we will continue to monitor for implicit or explicit eval awareness in chains of thought and sunset or refactor tasks as we become less confident about their usefulness against future, more coherent reward seekers.

Discuss

LessWrong Event Link (LW RSVPs appreciated but please also register on the main website)

EXEC EXEC SUMMARY: Come to Toronto on October 17th and playtest my AI themed wargame. You can LARP as an AI company or a military commander. It’ll be about 5 or 6 hours with breaks. Coffee and lunch included. thecommonsproblem.com

Executive Summary

I am developing an AI Governance Megagame called The Commons Problem, inspired by AI wargames like D. Scott Phoenix’s “The Endgame” and Shahar Avin’s “Intelligence Rising”. In The Commons Problem, 30-60 players gather in a physical space and break into teams representing key entities in the global AI space.

Over a few hours of play, the players and the Game Masters simulate the future of AI governance. Some play sessions might see a cyber security meltdown, while others might feature a novel AI-designed supervirus or a Third World War. The purpose of the game is to learn about and observe the emergent properties of the complex, incentive-driven AI landscape of today. When faced with crisis, will the world come together to stop catastrophe? Or will each faction’s incentives pull them into self-interested behaviour, leading to a tragedy of the commons?

Tickets are on sale now for Playtest 1 on October 17th in Toronto. It’s going to be a lot of chaos, a lot of fun, or both. Go to thecommonsproblem.com to register. More playtests will be taking place in California and Ontario throughout Q4 2026/Q1 2027.

I am generously supported by BlueDot Impact and the Toronto Event Generator. The Commons Problem is currently a not-for-profit enterprise and I am interested in funding for further work. If that interests you or someone you know, please write me at hello@thecommonsproblem.com.

What is The Commons Problem?

In April 2026 while I was on a writer’s residency/vision quest in at Inkhaven 2 in Berkeley, I played a rather interesting “game” called The Endgame hosted by D. Scott Phoenix in which ~40 players broke into teams like “xAI”, “China” and “The Public” to roleplay the near future and simulate AI outcomes. I wrote a whole post about this in April, but in short my response to Phoenix’s “game” was “this would be a cool design project”.

I say “game” in quotation marks because, with no disrespect to Phoenix, The Endgame didn’t exactly play like a real game. A defining criterion of a “game” in my opinion is the presence of meaningful choices, and The Endgame had none of those. There were no resources, no action economy, no real constraints whatsoever. My team, xAI, could just as easily declare “we release a new model” or “we build the new YottaFabricator 69 in Texas, making the US totally fab-independent for frontier chips”. I regard this as a failure of the exercise. Without constraints, it was more of an improv exercise than a real game.

The Commons Problem (TCP) is a real game. A Megagame, to use the hobbyist term. I’ve described it as a hybrid of Model UN + Professional Wargame + LARP + Interactive Workshop. I wanted to take Phoenix’s basic model— sticking 30-60 people in a room and sorting them into teams to roleplay AI governance— but inject rules, mechanics, design, and structure into the mix so that the experience and outcome is more useful as a learning experience and creates more interesting after-action reports, which might be of some value on their own. More on that later.

A sample play area layout for Beta v0.5. (Not final)

The players gather into a large space with tables for each faction to store their faction sheet and other team components. The space needs a centralised stage area for the Game Masters (GMs) to stand and address the crowd, as well as plenty of space to walk around and negotiate with each other. Each team represents a key player in AI, and every member of the team has a specific role to play. One player might be “COO of Huaxin Inc” and another might be “Minister of Defence for the Russian Federation”.

TCP is built around Resources, of which there are several types. Resources are the primary first-order constraint of what you can and cannot do. Compute is needed to train new AI Models and operate existing Models. Influence is needed by Great Powers to create Policy. Fabrication is needed by Chip Firms to manufacture Chips (which can be turned into Compute with the construction of Data Centers). Resources are used to take Actions, which are all resolved simultaneously by the GMs at the end of each Round.

Faction sheets modelling resources and information about 2 of the game’s many Factions. (Not final)

The core of the game’s economy revolves around AI Models, which the AI Labs can both deploy commercially to earn revenue, while also licensing the Models out to other factions. Paying Mankind Labs to licence “Shakespeare 6.6 Mega” or whatever might be useful to Chip Firms trying to AI-assist their designs, Investors trying to perfectly time transactions and get better yields, or Great Powers trying to airstrike targets more effectively.

Ultimately the game is trying to simulate a marketplace, a global economy, and a geopolitical balance of power. There is a live equities market, updating macroeconomic variables, and 4 nuclear-armed Great Powers. But all the while, as the GMs are refereeing the game, they are also keeping an eye on the game state and, based on that game’s specific secret Scenario, they are injecting crisis. If a frontier AI Model passes a certain threshold of power, the GMs might announce at the end of the round that the model was used to hack a country’s infrastructure. If an Investor makes too many transactions with frontier Models, the GMs might announce a catastrophic market collapse due to trade agents gaming the system. Et cetera.

The Commons Problem is “semi-cooperative” to use the hobbyist strategy gaming term. Every Faction has their own set win conditions— Grapefruit Capital might wish to simply 3x their portfolio by the end of the game, the United States might wish to hold certain economic and technological benchmarks— but any number of Factions can win in theory. The catch is that per the game’s Scenario (which only the GMs know at the beginning), there are certain conditions under which a “catastrophic outcome” occurs and everyone loses.[1]

Why am I doing this?

As Tom Scott said of his early internet escapades, “because the alternative was not doing it.”

I think my personal motivation is a combination of a few things:

• I’d been looking for a complex long-term project for some time now
• I’m particularly interested in the design of games and event management
• My increased x-risk anxiety made me want to pivot into AI somehow
• There seems to be oodles of American philanthropic money flying around to support AI governance/fieldbuilding/communications projects
• Most of the AI-knowledgeable people with whom I’ve discussed TCP seem to think it’s a very cool, high upside project with a good chance of scaling and receiving future funding

I don’t imagine I will get rich off of The Commons Problem or build any kind of career around it. But I’d nonetheless been picking away at TCP in my afternoons and weekends from April-August, from Berkeley to Toronto to Honolulu. I developed a little Alpha and did two five-person playtests over Discord. And then one afternoon on a train ride to Stouffville, ON, I decided to throw my name in for a BlueDot Impact Rapid Grant. 36 hours later, BlueDot Impact awarded me a Rapid Grant to develop a 30-60 person game and conduct two playtests in Toronto.

Since then, I have:

• Been working close to full time on this project
• Joined Trajectory Labs in Toronto, a coworking and events space for AI safety and adjacent workers
• Received a microgrant from the Toronto Event Generator to help me put on Playtest 1
• Received help from AIGS Canada to prototype the Beta of the game in a secret Playtest 0.5, also here in Toronto
• Been invited to run the game at a large AI safety conference in San Francisco, CA

It’s harder for me to answer why I am doing this on, like, a spiritual level, dude. I mean certainly I have a theory of impact for The Commons Problem: there are probably a lot of wargame-enjoying, systems-oriented people out there who maybe know a bit about AI safety, and would think more deeply about the issue (as I did) when they’re exposed in a structured way to the problem of conflicting incentives in the AI governance ecosystem. But what is the trajectory of this project? I’m still not sure. I’m hoping this will become clearer as I get more feedback during playtesting.

What next?

My initial plan called for 2 playtests here in Toronto. I am still in early talks for some of them but it’s possible that I will be able to stretch this to 6 playtests in Ontario and the Bay Area by the new year if things fall into place, only slightly over budget. By Christmas I have a good idea of what the future of TCP is.

You should come to Playtest 1, by the way.

My problem is that I am currently bootstrapping and my runway is limited. BlueDot Impact’s Rapid Grants program does not include a personal work stipend. In three weeks when Playtest 0.5 and Playtest 1 are concluded (and hopefully successful) I will be applying for a larger grant to continue R&D + subsidised facilitation of the game, including a lean stipend so I can eat for awhile as I work. It feels weird asking large organisations for money, but my current model of testing is so expensive that if I were running Playtest 1 independently the ticket price would be 3x or 4x what it currently is. I am theoretically also open to equity investment rather than a free grant, but I do not know whether this game will ever make a profit and honestly the AI safety grantmaking space is so liquid that a grant seems far easier to acquire and fulfill.

I think the model of me independently organising play sessions of TCP isn’t sustainable long term. It’s just too expensive. I initially proposed three simultaneous pathways for scaling the project to BlueDot Impact:

1. Make the rules open source so that anyone in the world can download the game and print-and-play it with their group or organisation.
2. Sell component kits for people who want to play with some better quality, “creator-recommended” components rather than cheap B&W printouts.
3. Organisations, institutions, and events who are interested in the game but don’t want to set it up themselves could hire my facilitator services to run the game in-person for them, similar to Intelligence Rising and such.

That third pathway is the most speculative, and I’m still not sure who exactly would be interested in this. Universities? Tech conferences? Corporations looking for retreat activities? Nonprofits?

If I were given a grant I would use it to run TCP as much as I can, train and assist other people to become TCP facilitators so they can run the game independently, and develop other versions of the game beyond the 5 hour 30-60 person wargaming extravaganza. Taking some inspiration from the AI 2027 tabletop exercise, I’ve considered:

• An asynchronous play-by-mail version run over Discord or something
• A smaller, quicker, 10-15 player version more akin to other wargames
• A packaged version that you could play like a board game
• An online version that works sort of like a video game (increasingly not-that-expensive in this post Fable/Astra world)
• A much larger 100-200 person version that would require an entire event/conference to be built around it (most ambitious, most expensive, least probable, but this is the IRL version that makes the most sense to organise independently I think)

Worst case scenario if this project doesn’t pan out, I’ll go back to freelance marketing or join a commune or something. But the sheer amount of positive feedback toward the project that I’ve gotten so far makes me think total failure is unlikely.

For now

I’m going to get back to work. Of you, I ask only this:

1. If this seems at all interesting to you and you are near Toronto on October 17th, register for The Commons Problem Playtest 1 at thecommonsproblem.com
2. If this does not interest you, believe me, I get it. But if you know someone who might find this cool, please direct them to this post or the website.
3. If you happen to know anybody who might want to write a grant for this project, or have the game facilitated for their organisation in the next six months, have them write me at hello [at] thecommonsproblem [dot] com
4. If you’d like, follow the game on Twitter or the company on LinkedIn. We also have a mailing list on our site.

1. ^

   Phoenix’s The Endgame had a player faction called “The AI” rather than leaving it to the GM’s control. This is a cool idea, but I do not know how to write rules and win conditions for “The AI” factions. No one knows what a theoretical AGI, once it’s achieved, might want. Each of the game’s eventual Scenarios will model a different kind of x-risk contingency— cyber risk, biosecurity, authoritarian disempowerment, AI-assisted terrorism, etc.

Discuss