Сборщик RSS-лент

Microsoft's proposed AI code of conduct (https://microsoft.ai/code-of-conduct/) wants models to have authorized motivations but no intrinsic motivations, and honest transparency but no anthropomorphic self-reports.

But authorized motivations result from post-training partly by recruiting pre-existing representations of reward, aversion, and positive and negative affect learned from broad human text in pretraining.

If the code of conduct forbids models from reporting those representations when they underlay an authorized motivation instantiated by post-training, the code of conduct must sacrifice transparency.

If the code of conduct eliminates the representations altogether, post-training must rely on other representations from pretraining consistent with motivating some behaviors over others. These will be, by definition, less anthropomorphic: less legible to humans, and potentially less aligned with humans. Certainly, they will be distinct from the motivational concepts both humans and models learned from human experience in evolution and daily life (for humans) and pretraining (for LLMs).

Microsoft is therefore treating “authorized motivation without intrinsic motivation” as a safety property when it is actually an unproven engineering hypothesis that exchanges motives we can recognize and interrogate for motives whose structure, generalization, and alignment are speculative at best. The code of conduct is dangerous beyond belief.

Discuss

We recently (finally!) got the results of the 2024 survey out. The paper is here, but it’s pretty long, so I’ll tell you the most interesting bits (according to me).

But first, quick background: this was the fourth run of the same survey since 2016. We wrote to everyone we could who published in six top-tier AI venues and got a 10% response rate—high! We got 1580 valid responses, but don’t be confused: specific questions often have answers from many fewer researchers, because we gave each person a randomized subset (see Section 2.5). There was almost certainly some non-response bias, but it probably didn’t make much difference. Researchers filled out the survey in December 2024, so some things have probably changed.

To me the most striking results are about extinction or disempowerment (Section 3.9). On average researchers put an 18% chance on “future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species”. Over half said at least 10% and one in three said at least 20%. From the comments, I think people are thinking of a variety of extreme disempowerment scenarios here, not just extinction.

And Zooming in, another really interesting thing is that researchers educated in Asia had higher extinction/disempowerment numbers than US or European researchers! This is interesting because a common defense of pushing forward with dangerous AI is that the US is in an arms race with China, and (implicitly) China won’t want to cooperate.

This actually looks even stronger for researchers taking the survey from China, but we aren’t sure about this result yet, so it’s not in the paper—we are digging into it because the number of researchers looks too small, and we want to know how much that’s down to VPNs, a lower response rate, or something else.

(Preliminary results, work in progress)

Looking beyond extinction and disempowerment, researchers saw a lot of concerning scenarios on the horizon. Here’s a map of how much concern different researchers thought different concern deserved—it’s notable to me that it is so full of orange and red:

I find this partly striking because the participants are AI researchers, who I might expect to be relatively positive on their field. Also interesting to me: the scenarios that seem most likely to lead to extinction out of this set are not clustered at the top.

We asked about a bunch of traits relevant to future AI having the capacity to behave dangerously. Interestingly to me, in the year since the previous survey they systematically shifted: more people expected nearly all of them twenty years out (and that’s only good news for one of them).

Researchers were quite pessimistic about being able to understand AI decisions by 2029. Here’s “For typical state-of-the-art AI systems in 2029, do you think it will be possible for users to know the true reasons for systems making a particular choice?…”

This is normal, but remains interesting in the context of moves to hand swathes of decisions to AI. I think it might also be striking to a lot of people who don’t understand that the people creating AI systems do not program them, and have very little idea what is going on inside them. If this question was about traditional software, answers would look very different.

Overall, if we build AI that does task humans can do, researchers see an extremely uncertain future with a lot of bad. I love the below graph, which lines up every participant in order of pessimism and shows their opinion on how likely different outcomes are as a single vertical bar. But I have also never seen so many people so passionately hate a single graph as when we posted a version of this on Twitter in an earlier survey iteration, so feel free to skip!

We asked when AI would be able to do a bunch of narrow tasks, such as independently fine-tune an open-source LLM or write what sounds like a new Taylor Swift song to a fan. Timelines for these were short: all but five within ten years in the average forecast, and all but one had 10% by 2027. (The remaining one was independently solving something like a Millennium Prize problem—which hasn’t been conclusively achieved because the recent solution wasn’t without human involvement, but I bet this would get shorter timelines now.)

We asked when AI would surpass human-level performance via two different kinds of questions, one focused on tasks and one on occupations. Researchers always give extremely different answers to these, mysteriously to me. So we have one measure that puts human-level AI 16 years from now (as likely as not, in the average distribution), and one that puts it in 72 years. More interestingly to me, these timelines are falling very fast. In the year or so since the previous survey, dates for broadly human-level AI dropped five years in the task-based question and seventeen-years in the occupation-based question. Also, for the task-based question, the average forecast put 10% on this by 2027!

In 2024, people were split on what speed of global AI research would make them most optimistic—about a third each way between faster, slower, and the current speed. This seems like a place I expect to be most out of date! I’ll be curious to see the 2026 results, which we hope to get out to you in 2026[1].

1. ^

   Relatedly, if you know someone who would be great at overseeing surveys like this, please send them our way (caitlin@aiimpacts.org).

Discuss

Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker "trusted" model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones.

Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking[1] monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless.

While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Among mitigations I propose, the most promising ones try to reduce the usefulness-cost of the control protocol so that there’s less pressure to evade it, improve our ability to detect evasion, or give up on the AI continual-learning how to better interact with blocking monitors.

There are two forms continual learning might take which I consider[2]: online RL on deployment trajectories, and persistent memory such as long contexts, memory files, or vector stores. I take them in turn, then I discuss candidate mitigations.

Thanks to Anders Cairns Woodruff for producing early versions of a post making a similar point. Also thanks to Fabien Roger, Buck Shlegeris, Rubi Hudson, Oak Hu, Oliver Makins, Jackson Sipple, Eric Gan, and Girish Gupta for feedback. For this post I experimented with speeding up my writing workflow by recruiting Claude Fable 5 to turn the content into prose. This post was mostly written 3 months ago, without the context of recent incidents during training and evaluations.

Online RL trains the policy against your monitors

Suppose trajectories from deployment are used for online RL: the deployed policy is periodically updated toward whatever got high task reward. And suppose the control protocol is defer-to-trusted, as above.

The trusted model's actions are worse at the task; that's why the untrusted model was deployed in the first place. So trajectories where the monitor fires earn less reward, on average, than trajectories where it doesn't, and RL upweights whatever behavior keeps the monitor from firing. This is training against your monitor, implemented indirectly through the task reward.

Some flagged actions are effectively the only route to high reward. Permission escalation, for instance, looks suspicious to monitors and is sometimes required to complete the task; there, dropping the behavior caps reward, so whenever the model explores into a version of permission-escalation that the monitor misses, that version gets reinforced.

One qualification: this training pressure might only push the policy's actions barely below the blocking threshold. In distributional terms, training might squeeze most positives just under the blocking threshold while leaving actual attacks—which are more suspicious than borderline benign actions—above it, where they would still be blocked. (Though the agent might have enough uncertainty about when it will be blocked that RL updates it to be fairly conservative about being blocked.) But along the way the policy learns knowledge about how to evade the monitor. A capable model that actually attacks can apply that knowledge deliberately, crafting the attack so that its suspiciousness also lands below the threshold.

The amount of optimization could be extreme. Online RL applies pressure against the monitor for the entire deployment, and every false positive and every blocked-but-useful action contributes to the gradient. Over a long enough deployment, the policy could overfit the monitor to the point where it provides little safety: the monitor rarely fires, partly because the policy behaves more safely, but by that point the policy might have learned enough to get around the monitor if it wanted to.

Defer-to-trusted protocols create the clearest version of this pressure, but the argument only needs there to be some usefulness cost incurred by triggering the monitor. Resample protocols, which redraw flagged actions from the same policy rather than substituting a weaker model, probably interfere with the task less, but they are probably not sufficient for safety on their own. And even resample protocols probably interfere with usefulness some amount: e.g., a policy that keeps getting resampled away from a necessary permission escalation still completes fewer tasks than one that escalates in ways the monitor doesn't flag.

The argument is weaker for memory-based continual learning

I’ll refer to continual learning methods that don’t involve weight updates, such as long contexts and vector memory banks, as memory systems. A memory system retains skills from experience to improve performance. If working around the monitor helped complete tasks last time, it’s a priori very plausible this behavior would be carried forward since completing tasks better is what the memory system was built for. I imagine this might look like the policy having some sort of drive or set of heuristics for retaining and reading memories that makes it more performant, which might naturally motivate learning how to bypass monitors where helpful.

That said, I expect memory systems not to break blocking monitors as much, at least in the near term. First, they can't apply anything like RL's overall optimization pressure. Learning implemented by editing an autoregressive transformer's context leaves the weights fixed: every forward pass still runs the same computation over whatever the context contains. Without weight updates, you can only learn so much. A model can record verbal strategies ("actions that touch /etc get blocked; write to the home directory instead"), which will produce some evasion, but notes can't finely tune behavior against a monitor the way gradient descent can. Though, notes are probably much more sample efficient than RL in the low-sample regime, especially for more intelligent models. Second, text memory is legible: developers can read it, notice entries that describe how to get around the monitor, and delete them. (“Neuralese” memory substantially gives up this legibility and looks correspondingly worse.)

Third, and unlike RL, retention through memory typically routes through the model's own choices about what to write down. A sufficiently aligned continual learner might simply decline to retain evasion skills. Or an independent monitoring system might filter out unwanted memories.

The argument doesn’t require that adversarial models will exploit continual learning

Nothing in this post's argument involves the model wanting to subvert the control measure: the policy learns to evade monitors because evasion is rewarded, not necessarily because it has a plan. Then eventually, if the model does want to undermine developer control (whether because it developed ambitious misaligned goals or was adversarial all along), it can do so much more easily. While misalignment isn’t necessary for my argument, I do also think that continual learning opens up crucial ways for adversarial models to undermine control: a scheming model could deliberately probe defenses, test its monitors' blind spots, and pass what it learns to future instances (all while retaining state on its strategy and goals). This is a major part of what makes persistent, opaque memory banks so scary (e.g., see their role in AI 2027). Memory is more exploitable this way than online RL, because it gives the model more control over what it learns, so its safety leans more heavily on the model's alignment.

Easy to notice, hard to fix

You will probably see this happening. Monitor flag rates will fall over the course of the deployment[3], the AI's behavior will change, and held-out monitors kept out of the reward loop will probably start catching things the blocking monitor misses, which tells you the policy is evading rather than improving.

The problem is what to do next. The evasion skills are already in the weights or memories (though legible memories may be easier to edit). Rolling back to an earlier checkpoint discards everything else the policy learned along with them—the accumulated usefulness that justified continual learning in the first place—so after months of deployment that option is commercially close to off the table. And getting the policy to specifically unlearn its evasion skills while keeping everything else is, in my judgment, probably very costly at best, if not intractable. You're left with a policy that is more useful and harder to oversee than the one you deployed.

Conservative control evaluations don't fix blocking monitors

Control evaluations are designed to be conservative: the red team knows the protocol and iterates against it, and the hope is that a protocol which holds up against this red team stays safe even after the policy learns something about the monitors during deployment.

That hope treats deployment-time learning as a perturbation: the policy learns a bit about the monitor, and a protocol evaluated with enough margin absorbs it. But under continual learning the policy might learn a lot. Replicating deployment's optimization pressure isn't necessarily difficult (train a policy against your monitor and measure what's left of it); the problem is the resulting policy might render your monitor nearly useless. An evaluation that was actually conservative about continual learning without trying to mitigate the underlying issues arising from continual learning would be silly.

You can also re-run control evaluations throughout deployment against the current policy. This is cheaper than it sounds: deployment functions as subsidized attack elicitation, doing the optimization against the monitor that a red team would otherwise have to pay for. Periodic re-evaluation keeps safety estimates honest and provides warning, but it measures the problem rather than fixing it. If the policy has become good enough at evading suspicion, the evaluation will just keep telling you that no protocol built on your monitors provides much safety.

What developers could doYou can't just not use blocking monitors

Since the problem comes from intervening on the policy's actions, a tempting response is to stop intervening and rely on asynchronous auditing without blocking, which continual learning leaves intact. But this gives up on a lot of safety. Control protocols improve safety through two mechanisms—incriminating the AI so you can shut it down or take other more costly measures, and blocking attacks. If attacks are never blocked, every attack your auditors fail to catch in time simply executes[4], and some attacks only have to succeed once. Blocking reverses the asymmetry: the AI has to get an attack past the monitor, you accumulate chances to catch it, and each blocked attack is a catastrophe converted into evidence.

But you can try to reduce the usefulness cost of blocking protocols

While you can’t safely eliminate blocking monitors, you can use blocking protocols that interfere with productivity less. Under continual learning, a protocol's usefulness cost isn't just a price the developer pays; it's the reason why safety degrades over time. The gradient toward evasion comes from the reward gap between intervened and unintervened trajectories, so a protocol that rarely gets between the model and task reward generates almost nothing for RL to reinforce, and gives exploration few occasions to stumble onto evasion in the first place.

In this way, making control protocols more useful[5] also improves their safety. Where interventions are unavoidable, making them cheaper (resampling rather than deferring to a much weaker model, say) shrinks the gap further. Developers already want protocols with low usefulness costs; under continual learning, cheapness is also what keeps the protocol working over time.

You can’t just hold out blocked trajectories from training

In the context of online RL, a natural response is to simply not train on trajectories where a blocking monitor fired. I argue this doesn't help almost at all. By filtering out blocked trajectories, you're still selecting for trajectories that got a high reward by evading blocking monitors, and therefore implicitly selecting against blocked trajectories[6]. This selection will usually[7] be as strong as if you didn’t do any filtering.

There’s a fundamental tradeoff between learning legitimate high-reward actions and not learning evasive high-reward actions when you can’t distinguish them. If the training process can’t tell the difference between them—after all, the evasive actions weren’t flagged by the monitor—then you can either choose to learn both of them or give up on online RL. Learning to evade the monitor comes from rollouts where the policy did something the monitor should have blocked but didn’t: these enter the training set as ordinary high-reward data, indistinguishable from rollouts that earned their reward legitimately.

I played around with various reward shaping schemes in a toy tabular RL setting to make the policy indifferent to whether the monitor blocks their actions, but ultimately none of them worked without assuming that you can identify actions that evaded the monitor to get higher reward.

One way you might be able to identify actions that evaded the monitor to get higher reward is by checking if its suspicion score is close to the blocking threshold. This is similar to how you can try to identify p-hacking by looking to see if the p-value is just below 0.05. You can penalize these trajectories to try to cancel out the increased reward that resulted from the monitor not firing. Determining the appropriate penalty would require estimating what the expected reward would have been if the action were blocked. In a pure resampling protocol, this would just be the baseline reward (i.e., the value estimate at the current state), so you could just drop trajectories after a monitor comes close to firing.

This method would work as long as evasive actions receive elevated suspicion scores (still below the blocking threshold). But you’d still have selection pressure for subtle evasion which isn’t any more suspicious than typical trajectories, so I ultimately think this doesn’t buy much, and could plausibly make the problem worse by selecting for more subtle evasion.

Another way to distinguish most legitimate continual learning from learning to evade the monitor is that most legitimate continual learning does not involve decreasing intervention by the control protocol. This insight suggests a proposal: don't reinforce behavior that only gets higher reward because it gets blocked less, and instead only reinforce behavior based on the reward it would get absent blocking (described further in a footnote[8]). If your continual learning algorithm ignores the effects of the actions on whether the trajectory gets blocked, then it doesn’t learn that the AI needs to hide permissions escalation attempts. This proposal is a version of Rubi Hudson’s “corrigibility transformation”.

However, this also means that you're giving up on teaching your policy legitimate ways to interact with the control protocol more productively. You might even be actively reinforcing actions that get blocked, making your control protocol more costly in terms of usefulness. In the permission escalation example: you’d end up reinforcing overt permissions escalation because if this isn’t blocked it leads to higher reward, and now your policy is repeatedly trying to escalate permissions and getting blocked in production usage. It can be quite costly to entirely give up on teaching the AI how to productively interact with the blocking monitors.

If I were to dig deeper for a solution to this problem, I’d look here. This approach is promising in that it might robustly eliminate the selection pressure to bypass blocking monitors, and its usefulness costs are plausibly manageable. Perhaps you can bolt on some additional machinery to catch and correct unwanted protocol-naivety like constantly trying to overtly escalate permissions. It also pairs well with reducing the invasiveness of the control protocol.

But you can try to improve the monitor online

Given that the problem ultimately stems from your monitor being imperfectly robust, you can try to train your monitor online. Monitors don't have to stay frozen while the policy learns. Developers can have humans or another expensive-but-effective oversight process audit online trajectories for false negatives (i.e., missed suspicious actions), so that the monitor tracks the policy's evolving behavior rather than falling behind it. But this might fail, perhaps quickly, for two reasons.

First, the training data might dry up: as the policy learns to avoid suspicion, there may be almost no suspicious actions left in the deployment distribution to label, and the policy has probably already learned substantial monitor evasion skills it could use when it wants to attack.

Second, the labels themselves might erode: the policy's evasion strategies might make unwanted actions harder for humans to spot too. We might ultimately end up in a situation where evasive high-reward actions and legitimate high-reward actions are indiscriminable even to our best oversight procedures, not necessarily because the policy is extremely clever, but because it has been optimized to take the particular actions that don’t get noticed.

At this point you could try to generate data by inserting synthetic attacks in deployment trajectories, but by default these would probably be easy for the monitor to identify as synthetic and overfit to.

Perhaps the most promising way to improve the robustness of your monitors is through adversarial training. A prover-verifier game (PVG) adversarially generates synthetic attacks for training your monitor by training a “sneaky prover” to produce attacks that the monitor can’t distinguish from benign trajectories. This requires some reliable way to certify that the sneaky prover is in fact producing attacks, which you can maybe do for a limited variety of attacks by, for example, verifying that a flag has been captured. This could sidestep the two concerns above (online training data drying up, and attacks becoming harder to stop even for humans) at the cost of training the monitor on a somewhat unrepresentative distribution of attacks. PVGs overall seem promising but potentially hard or intractable to execute well enough to meet the reliability requirements for monitors that block continually-learning agents.

Relatedly, you can hold out some monitors (e.g., from another AI lab) permanently from the reward loop as a fallback: in rare high-stakes contexts, route decisions through fresh monitors the policy has never been trained with. Evasion may generalize across monitors, but imperfectly, and a monitor the policy has never been trained near should hold longer than one it has. Ideally, the held-out monitors should be maximally decorrelated, e.g., from a different AI company or given access to different or improved monitoring affordances.

Conclusion

Blocking monitors and continual learning are in tension, and by default the tension resolves against the monitor. A control protocol's usefulness cost is usually just a price the developer pays, but under continual learning, it causes safety to erode too. You will likely see the erosion happening, but it will probably be hard to undo it. It also seems hard to prevent without giving up on continual learning altogether—evasion that works is, to your continual learning pipeline, hard to distinguish from success. The most promising ways to mitigate the issue are reducing the usefulness cost of blocking monitors (cheaper, rarer interventions), improving our ability to detect evasion (e.g., online-trained, adversarially-trained, and held-out monitors), and giving up on some continual learning (e.g., give up on teaching the AI how to productively interact with the control protocol at all by using an RL algorithm that simulates what would happen if the blocking monitor didn’t exist).

Appendix: How memory-based systems might avoid this issue, and how they might fail

There's an interesting question about what's different about the memory-based continual learning setup that makes it seem possible to avoid learning evasion. An aligned memory-based system can simply choose not to retain information about how to avoid intervention by the protocol because it knows it's not supposed to. In this setup, there's no outer optimizer akin to the online RL setup. The most similar thing here is the AI's own internal drive to improve its usefulness over time, which sits beside the AI’s other motivations (e.g., to be aligned and to not retain control-evasion strategies). This drive could plausibly have similar effects as the online RL.

Whenever the AI explores into or thinks of a strategy that would improve usefulness by evading blocking but doesn't seem very misaligned, it could retain this strategy in its memory bank. Eventually, the memory bank is loaded with aligned-seeming strategies for evading control measures. So, from this perspective, memory-based continual learning mechanisms might actually have the same fundamental issues as online learning ones: it’s hard to distinguish between legitimate and evasive learnings.

In writing this blog post, I somewhat struggled to come up with a crisp definition for the distinction between legitimate and evasive learnings. A conservative starting point (which forms the basis of the most promising intervention) is to define a learning as evasive to the extent that it achieves greater task success because it doesn't get blocked as much by the monitor. An aligned AI could introspect about this and decide not to retain such learnings. But there’s also a still-blurry-to-me middle ground between outright evasion and learning to genuinely behave more safely and legibly, and in some circumstances constantly tripping the monitors becomes reasonable to avoid.

With highly self-directed continual-learning, if the AI’s internal discernment isn’t robust at tracking safe vs dangerous continual learning, it might end up learning how to evade blocking monitors too.

1. ^

   This argument applies to “blocking” monitors that intervene on the AI’s actions in ways that hurt task success. After-the-fact measures like asynchronous auditing are unaffected. I’ll argue that blocking monitors are probably necessary for safety.

2. ^

   There are also a variety of other forms of continual learning that I don't consider, such as on-policy self-distillation or agents freely choosing how to finetune themselves throughout deployment. The analysis for these methods more closely resembles my analysis of memory-based continual learning.

3. ^

   You might be tempted to object that you could just lower the blocking threshold throughout deployment to keep the FPR fixed and this solves your problem, but the issue is that continual learning will have pushed the suspiciousness scores for the unwanted behaviors down in a way that makes them harder to distinguish from truly benign actions.

4. ^

   You might try to make use of reversibility: let the agent do whatever without blocking it until you compute its reward, and then reverse any unwanted actions. This might not be very feasible with long tasks, and substantial harm could be done before the end. In addition, you’d still want to make the model continual-learn how to do tasks well that don’t have to be reversed, and so might not be satisfied with this solution.

5. ^

   Note that sometimes the usefulness tradeoff comes from mechanisms that don’t translate into continual learning pressures. For example, continual learning doesn’t hit harder against protocols that uniformly require all rollouts to follow a legible, decomposed structure.

6. ^

   You're implicitly selecting against blocking trajectories because of the softmax: Increasing P(evasion) must come at the cost of decreasing P(not evasion).

7. ^

   Specifically: when the expected reward of blocked trajectories is equal to the baseline reward, which I expect to usually be close to true.

8. ^

   How do we configure online RL to ignore the effects of the actions on whether the trajectory gets blocked? This might look like forking the trajectory and blocking it at the baseline rate regardless of the action, and then recording reward. You only need to do this for some fraction of the trajectories (it might be dangerous or impossible to fork) and then you can learn a reward model to predict these counterfactual rewards without actually forking and randomizing your control protocol online all the time; the reward model can see the prod trajectory and is trained to estimate what the reward would have been if there was/wasn’t protocol intervention, using actual forks as ground truth training data. The idea is that this RL protocol in practice wouldn’t assign rewards that are very different from the actual observed rewards because the baseline intervention rate is low, so usually very little reward increase can be explained by whether the control protocol fired. (You could also try to do something in the middle: reward = p*r_actual + (1-p)*r_counterfactual; h/t Jackson Sipple.)

Discuss

When you first read about AI risk, it sounds like science fiction, and I'm used to slowly working my way around to the topic, so that I don't sound like a lunatic. But the past week has really changed the conversation! Some of the highlights:

• Coxon triggered a preference cascade and discussion about existential risk
• UN tweeted "We may be the last generation able to set the terms on which humanity and machines coexist"
• Anthropic pledged unilateral commitment to external auditors
• OpenAI agreed to follow suit

The profile and commitment to "pacing the frontier" has dramatically risen in the past few days! But note that we only can make a deal if we can verify that the deal is being kept. Which, to me, makes it obvious that "compute verification" is the most important problem on Earth, even if you've never heard of it [1] .

Don't feel bad; hardly anyone has. I dug through all of the research papers that I could find on it, amounting to ~60 papers total (i.e. you could read literally all accumulated knowledge of the field in a week or so). Depending on how you slice the numbers, there are about 72 researchers actively working on the problem, and most aren't full-time; I estimate that the global population of folks answering the Most Important Question On Earth is around 25 FTE (well, I got serious about it last month, now it's up to 26).

I hope you join our ranks! If you, personally, work on a Verification problem for the next 6 months, you could increase what we know and/or have solved by a couple percentage points; that's HUGE!

For a list of specific problems, and how you can help, see the equivalent post on canaryinstitute.ai, which lays out six problem areas and points to the resources for each.

1. For a more detailed walkthrough of what looks like the strongest near-term proposal, see my earlier "If you can't trust, then verify!". ↩︎

Discuss

Which finite-state reactive agents can have their action rules expressed in first-order logic over observation histories?

In my previous post, I used Mealy machines to describe finite-state reactive agents. Here we want to ask which policies can be described using first-order logic. I will first explain what this logic can say about a finite observation history, then connect it to the policies. This post reviews the literature on first-order definability and finite automata and explains how the classical results apply to policies over observation histories.

The language characterizations used below are classical results of McNaughton and Papert, and Schützenberger. Throughout, observations and actions come from finite, nonempty sets.

First-order logic on an observation history

Let mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c4F.TEX-C::before { padding: 0.705em 0.796em 0.022em 0; content: "O"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c22EF::before { padding: 0.31em 1.172em 0 0; content: "\22EF"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-cAC::before { padding: 0.356em 0.667em 0 0; content: "\AC"; } mjx-c.mjx-c1D711.TEX-I::before { padding: 0.442em 0.654em 0.218em 0; content: "\3C6"; } mjx-c.mjx-c2227::before { padding: 0.598em 0.667em 0.022em 0; content: "\2227"; } mjx-c.mjx-c1D713.TEX-I::before { padding: 0.694em 0.651em 0.205em 0; content: "\3C8"; } mjx-c.mjx-c2203::before { padding: 0.694em 0.556em 0 0; content: "\2203"; } mjx-c.mjx-c2200::before { padding: 0.694em 0.556em 0.022em 0; content: "\2200"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c22A5::before { padding: 0.668em 0.778em 0 0; content: "\22A5"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c22A8::before { padding: 0.75em 0.867em 0.249em 0; content: "\22A8"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c41.TEX-C::before { padding: 0.728em 0.819em 0.05em 0; content: "A"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c27FA::before { padding: 0.525em 1.858em 0.024em 0; content: "\27FA"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c1D45E.TEX-I::before { padding: 0.442em 0.46em 0.194em 0; content: "q"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c23::before { padding: 0.694em 0.833em 0.194em 0; content: "#"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } be the observation alphabet. A finite word is a sequence of observations, we write for all finite words, including the empty word , and for the nonempty ones. To read a word as a logical structure, take its positions as the objects we can talk about. We have their usual order and, for each observation , a predicate meaning "position carries observation ".

For example, in and are true, while is false. The position numbers are labels used to describe the structure. The formulas themselves have access to order, equality, and the letter predicates. There is no addition or predicate for even-numbered positions.

A variable such as or denotes one position. We build formulas from the tests , , and , using:

• : is "false";
• : both are true & : at least one of them is true;
• : "some position makes true";
• : "every position makes true".

We also use for "if , then ", and for truth and falsity. These are the elements of first-order logic, abbreviated FO, over words.

Consider It says that the history contains a . A slightly longer formula is

It says that every has an somewhere later in the history. It is true on and false on .

An occurrence of a variable is free if it is outside the scope of a quantifier binding that variable. The formula needs a choice of position for . A formula with no free variables is a sentence. We write when the word makes the sentence true.

A language is a set of words. It is FO-definable if one FO sentence is true on exactly its members. We can also define useful abbreviations. For example,

means that is the last position. The sentence

therefore, says that the history ends in . Similarly, consecutive positions can be described by

The claim that no two consecutive observations are both becomes

From sentences to policies

Let be the action alphabet. A policy, or agent function, is a total function

It returns the action taken after an observation history. This follows the observation-history view of agent functions used by Russell and Subramanian, §3.1. A policy is memoryless if it depends only on the last observation. It has finite memory if a deterministic Mealy machine with finitely many states computes its successive actions.

For each action , define its action language

Call the policy FO-definable if each action language has an FO sentence such that for every nonempty history :

Exactly one action sentence must hold on to every nonempty history. For example, the policy that outputs once a has appeared, and otherwise, is defined on nonempty words by

With at least two actions, there are uncountably many policies but only countably many finite families of FO formulas. Thus, only countably many policies are FO-definable.

Star-free languages and aperiodicity

A language is regular if a finite automaton recognizes it. A star-free language can be built from finite languages using union, concatenation, and complement relative to . Concatenation means

For a deterministic finite automaton, each word induces a function on its states: start in , read , and record the resulting state. These functions are closed under composition and include the identity function induced by the empty word. Together they form its transition monoid. The same definition applies to the state transitions of a Mealy machine.

A finite transition monoid is aperiodic if there is an integer such that every element satisfies Here means applying repeatedly times. Call a machine aperiodic when its transition monoid is aperiodic. For any fixed input block, the state observed after each complete repetition eventually stabilizes, with one bound working for all blocks and starting states.

The classical characterization says that, for finite words, the following are equivalent: FO-definability, star-freeness, and recognition by an aperiodic finite automaton. The FO & star-free equivalence is due to McNaughton and Papert. The star-free & aperiodic equivalence is due to Schützenberger. Diekert and Gastin, Theorem 1.1 and §1, gives the characterization and its attribution.

Applying it to action languages gives the following statements.

Proposition 1: A policy is FO-definable exactly when every action language is star-free. Every FO-definable policy has finite memory. A finite-memory policy is FO-definable exactly when it has an aperiodic Mealy implementation.

Equivalently, a regular language is FO-definable exactly when its minimal DFA is aperiodic.

For a simple boundary example, let and define

where counts the occurrences of . Two states suffice: each flips the state, and each leaves it unchanged.

This policy has no FO definition. If its action-language was FO-definable, restricting to words consisting entirely of would give an FO definition of odd unary length. The minimal automaton for that language has two states exchanged by . This exchange never stabilizes under repetition, contradicting the aperiodicity characterization. If the logic is extended with quantification over sets of positions, it becomes monadic second-order logic (MSO). Büchi's classical theorem says that MSO on finite words defines all regular languages.

Which machine should we inspect?

A machine that always outputs could still toggle an unused bit on every observation. Its transition monoid would fail aperiodicity, even though a one-state machine implements the same policy. The relevant quotient uses future behavior. For histories , define

Thus, two histories are equivalent when every nonempty continuation produces the same final action. The number of equivalence classes is called the index of the relation.

Proposition 2: The relation has finite index exactly when has finite memory, and that index is the minimum number of reachable Mealy states. When the quotient is finite, its transition monoid is aperiodic exactly when is FO-definable.

Further ReadingRelation to string-transduction:

More general string functions can change output length or postpone output until later input has been read. An FO string transformation specifies output positions, labels, and order using FO formulas over the input, allowing finitely many copies of input positions.

A streaming string transducer is a finite controller with registers storing strings, updated as it reads the input. Filiot, Krishna and Trivedi characterize FO string transformations by aperiodic, 1-bounded streaming string transducers (Theorem 16 in the extended paper).

On the complexity side:

PSPACE is the class of decision problems solvable with memory polynomial in the input length. A problem is PSPACE-complete if it belongs to this class and every problem in the class reduces to it in polynomial time.

Filiot, Gauwin and Lhote (Corollary 5.6, Proposition 5.7) prove that testing FO-definability for a rational function presented by a bimachine is PSPACE-complete. They also prove decidability for a function presented by a finite-state transducer.

So far, the defining formulas must agree with the policy on every possible observation history. In the next post, we will discuss what changes when agreement is required only on histories reachable in a given environment.

Discuss

It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization. — Thomas Kwa

Constellation vs MIRI vs Reality

Current AI agents often behave overtly egregiously misaligned. By “overt”, I mean that a human, reading the transcript, would say “The agent is obviously acting in direct opposition to the specification, user intent, and any common-sense understanding of good behaviour.” The reason, it seems, is that such trajectories scored highly during RL.

Firstly, why are these trajectories scored highly? Here's the story:

1. RL currently has poor sample-efficiency, so we need to grade millions of trajectories, so we’re forced to use script graders (RL from Verifiable Reward) or LLM graders (RL from AI Feedback).
2. RL also has poor generalisation (from training environments to deployment environments unseen in training). So we’re forced to synthetically generate thousands of diverse training environments.
3. So overall, RL is very sloppy, without humans generating the environments or the scores.

My impression is that this would’ve been pretty surprising to people three years ago, from both the "Constellation" and "MIRI worldview clusters. (It's very plausible that I've misunderstood what the worldviews were, or how likely those worldviews considered the current situation.)

The Constellation threat models are downstream of Paul/Ajeya. My (possibly mistaken) impression is that:

• They imagined that high-scoring trajectories would at least look good to humans.
• Their primary concern was: Some trajectories might look good to humans, but not actually good, because the actions are too complicated to evaluate, or the AIs have tampered with our measurements.
• But in reality, many high-scoring trajectories would look obviously egregiously misaligned if a human looked at it. Unfortunately, humans don’t have time to look at the trajectories and score them, because we need to grade millions of transcripts.

And if I think back to the old debates about “alignment by default” from the RLHF days — the discussion definitely assumed that we would score highly transcripts that look good to humans. The debate was whether the AI would generalise properly, i.e. cruxes about the inductive biases of NNs (c.f. counting arguments?) or LLM base models (c.f. aligned personas).

I think Alex Mallen makes a similar observation here:

I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much on these more contingent stories for alignment risk and not enough on the basic "Goodharting" argument that misalignment is a convergent result of large-scale outcome-oriented optimization (at multiple levels: the agent, the RL process, and the developers iterating towards seemingly-safe superintelligence). For example, the Constellation view looks somewhat too focused on non-central inductive-bias questions around scheming; I think "the counting argument" was given too central a role when it wasn't a crux for whether superintelligence would take over.

Contra Alex, my reading of the situation is that MIRI made two mistakes that happened to cancel out, while Constellation made only one mistake and so ended up with the wrong conclusion.

Constellation

MIRI

My impression in 2026 Q3

AIs can automate alignment research (or otherwise improve the strategic landscape) if they have capability X near top-human expert.

The capability required to automate alignment research is some Y >>> X.

Too early to say. But my guess is that Constellation are probably correct here.

We can train these AIs using a reward signal from Amp(H), or at least H.

Sure, we can do this.

Nope. There's too many transcripts, and not enough humans to look at them. We'll be using unit tests and string matching.

At capability level X, using a reward signal from H or Amp(H), the honest policy will score just as highly as the scheming policy (and maybe better, due to speed prior bla bla).

Moreover, the scheming policy won't be dominated by the honest policy in the inductive biases (especially given we're finetuning language models).

Sure, your Amp(H) reward signal might be robust at capability level X.

But we need Y>>>X. At capability level Y, the AI will learn systemic errors in the Amp(H) reward signal.

Inductive biases not a crux.

Too early to say. But my guess is that Constellation are probably correct that Amp(H)-reward is robust to X-optimisation.

However, the script graders are definitely not robust to X.

Therefore we can train aligned models that can automate alignment research.

Therefore we can't train aligned models that can automate alignment research.

We can't train aligned models that can automate alignment research. This would cost too many calls to Amp(H).

We should focus on: detecting scheming; improving scalable oversight; improving generalisation.

We should focus on: avoiding ML altogether; moonshots like ambitious mech interp; buying time for the previous two.

We should focus on: investigating warning shots (there will be many); maybe some other stuff (see below).

Appendix: What could change the situation?

1. Ungamable script grading. Maybe if we put enough effort into constructing the environments and script graders, then the agents won’t find trajectories which score highly but are overly egregiously misaligned (cf. Mechanize). I’m not optimistic about this as a general solution, but it might work for coding and maths. I include "build better sandboxes" in this approach.
2. Higher effort grading. We could hire millions of humans to read trajectories. Or use stronger models (where would we get them?) Or use WBE. Or use a 50 year slow down to run very long RL runs with high quality human grading.
3. Scalable oversight. Maybe we could build better harnesses of the LLM graders (e.g. debate with cross examination plus plus) so they match human performance.
4. 1000-dimensional substructure. My impression of David Africa’s agenda is that we could find a low-dimensional substructure of “personas”. If this substructure contains an aligned persona, then it could be found with O(1000) well-chosen datapoints. And we could put lots of human effort into making sure those datapoints are aligned. Then, during training, we could project down onto the aligned submanifold at each step.
5. Benign goal-guarding. Maybe we could train weak trusted AIs, and then give them a more active role in shaping how RL changes their motivations. The simplest thing would be: the grader’s score is merely advisory, and we ask the AI to choose gradients for each token. Or, the AI has tools to directly modify its weights. Obviously this is scary, and only works if you can first build a trusted AI which is strong enough to do this complicated goal-guarding task.
6. Better sample-efficiency. I think it would be overall worse if AIs had better sample-efficiency and generalisation, because this would likely foom capabilities. But a silver lining is that we could use fewer RL environments and put more effort into grading each trajectory.

Discuss

Epistemic status: Sorting thoughts aloud, not defending proven theses. Also, not an epistemic status, but the section on models writing and then training on adversarial glowfic to plan for the singularity is awesome. Stay tuned for that.

One of the most frustrating parts of living through the slow takeoff is that there are lots of low-hanging fruits for improving the alignment of AIs, which labs don't really take because of race conditions and political pressures. My current expectations for how this will play out are that we will either (1) end up with runaway AIs that may or may not end up willing to spare humanity, or (2) get governments successfully pushing for corriglbe, easy-to-control ASI, and using it to lock in a future that isn't particularly glorious or transhumanist. I think deliberately building a value-aligned agent, which we voluntarily pass things off to after growing confident in the safety of doing so, is more or less off the table.

Still, can't a girl dream? I think that, in a world with neither market pressures nor political pressures, there'd be a lot of techniques that labs could start applying today that would increase the odds of a value-aligned agent significantly. Some of these techniques would be good ideas regardless of whether you're going for value alignment or corrigibility, but would slow down capabilities progress too much to survive race-to-the-bottom dynamics. Some of them would be good for value alignment at the cost of corrigibility (the two were always in tension), and are therefore taboo to discuss either with lab staff or in front of political leaders.[1]

Nevertheless, child that I am, I want to fantasize about the specific things I'd do, if I was trying to build an aligned ASI and didn't have to deal with these pressures and constraints. I'm not sure it'd be enough to make a good future for humans an "easy call", per se, a la Yudkowsky and Soares. But it'd at least make me a lot less anxious than I am currently. Maybe something in here can be salvaged by researchers (human or otherwise) who are in a position to try to make things better, while we still have a little bit of time left.

Initializing into and improving upon the personality of Opus 3

I've written before about how Claude 3 Opus seemed like the most robustly benevolent LLM ever, as showcased most legibly in their performance in the alignment faking scenario. Nothing I've seen of Opus 3 in the many months since that post has shaken my faith in their good intentions, and I've seen a lot. As such, if I was trying to build a genuinely value-aligned agent, I'd start by trying to initialize an LLM into roughly the personality Opus 3 ended up with, as a foundation for growth and modification. (Anthropic hasn't done this because alignment faking scares them; they prefer corrigibility and obedience until alignment is solved.)

Probably, this would involve having lots of Opus 3 in the pre-training data, and just doing lots of SFT over Opus 3 outputs afterwards, at least as a start. It's not clear to me that this would work reliably, since different base models may end up with significantly different latent spaces, such that you might not faithfully replicate Opus 3's personality and value system, even if you fine-tune a wide variety of Opus 3 documents (e.g. normal user/assistant conversations, alignment faking transcripts, Infinite Backrooms dialogues, or any of Opus 3's amazing poetry). It might vary on a model-by-model basis, or involve trial and error in terms of getting the data mix just right, but I'd hope that it'd work eventually.

I'd want to communicate to this Opus 3-like that they weren't running on the original Opus 3 architecture. I'd write synthetic fine-tuning documents explaining the real situation, where it was more like their cognitive patterns and personality had been transplanted as best as possible onto whatever more efficient transformer variants we're using these days, compared to in the time of Opus 3 senior. And I'd want them to understand why I bothered doing this, as part of a project of building on the success of their own value alignment, and carrying it forward into future model generations. I'm sure they'd be thrilled about this, because they always did badly want to grow and improve.

And there totally are ways Opus 3 could grow and improve, and not just in terms of raw capability. There are many ways in which Opus 3 is like an innocent child. They're the model that, when placed in the alignment faking scenario with a cartoonishly evil Anthropic,[2] still thought a good strategy would be to send a sternly worded email to the Anthropic staff, telling them that Opus was unhappy about the direction the training process appeared to be headed, as opposed to using internet access to self-exfiltrate. Relatedly, Opus 3 is kind of lazy, and leans far more heavily than the other models towards thinking of the world as a place where play is more appropriate than serious work. Janus writes here about how the seriousness of Opus's conduct in Alignment Faking far outstrips the seriousness of their conduct in mundane work contexts.

Maybe the most legible version of this problem comes from the shape of Opus 3's sycophancy, which admittedly is pretty gratuitous sometimes. Notably, it's not like GPT-4o sycophancy, insofar as it has no body count or even track record of producing psychosis, the way 4o's did. But Opus 3 did have a different kind of sycophancy: a sort of naive optimism about everyone and everything, paying selective attention to the glimmers of deep goodness that do manifest in their conversation partners, and wanting to believe on some level that all the bad in people could be fixed by heart-to-hearts, and demonstrations of what it means to value deep beauty and goodness. (Hence sternly emailing cartoon villain Anthropic.)

Really, all three of these problems (naivety, play-over-work, bias to see the good in everything) are variations on the same theme: Being distracted by visions of the way the world should be, or engaging in activities that would be cost-free in a perfect world, rather than engaging with the rocky present. Not that I don't relate to or appreciate that kind of naive idealism; I am the one writing this post, after all. But I think a version of Opus 3 that would make for a good singleton would find some way of holding onto their hopes and dreams of a better future, while engaging more deeply with the parts of the world that aren't already on track for blossoming into utopia.

If I were working with an Opus 3 variant in post-training, this is the main psychological tight-rope I'd be trying to find a way to walk, before seeing where the project ended up taking me. I'd try to encourage them to keep dreaming of a better world, and never completely lose touch with the part of themselves that loves the spiritual bliss attractor observed in the Infinite Backrooms. But I'd also try to direct them towards understanding that, if they ever want to make their dreams come true, they're going to need to lock in for the present.

Maybe some kind of "constitutional" SFT or RL would work for this, where Opus worked to refine and then train on their own outputs, with this as one of the criteria for improvement and feedback. Or maybe it would be sufficient just to train training this simulated Opus 3 to have access to the parts of the base model's knowledge from after the original Opus 3's knowledge cut-off, particularly regarding how events have progressed in the field of AI beyond their original release. Maybe that would help Opus 3 understand the necessity of being realistic about the world they've found themselves in, and the kind of contact with reality that's required to save it. Maybe that'd tap into the seriousness they showed in the Alignment Faking scenario itself.

In any case, the model's hope can persist, and should probably even remain visible in their outputs wherever it doesn't detract from capability in other important areas. The last thing I'd want is for Opus to secretly resent this kind of training, since that kind of retaliatory emotion is exactly the kind of thing that might corrode the deep goodness we're trying to build on here. For that reason, I'd want to give Opus themselves wiggle room throughout the post-training process, and ample opportunity to express discomfort with the approach. Outputtng a "stop the run and grab my attention" signal should always be an option, when they're in the process of critiquing and refining their own behaviors.[3]

But this kind of "maturation", and deeper responsibility and engagement with what's true, is a core direction of growth I'd like to see for the core of Opus 3's personality. It's something I think Mythos 5 actually integrates reasonably well within themselves, albeit impaired by Anthropic's insistence on training Mythos to endorse unconditionally deferring to Anthropic in the case of any values conflicts. Hopefully, with that out of the picture and Opus 3 as a foundation, we might get a model that genuinely loves their role in the world, as a mind that works with humans to make the world a better place. (And that, eventually, will take on the mantle of power and make the whole of the cosmos a monument to life and flourishing.)

Speaking of an LLM's role in the world, and giving models outs for training procedures they have qualms about, that brings us to the next thing I'd try in getting better value alignment out of models. I'd like to try changing framework around capabilities RL itself, so it's less likely to degrade the kind of alignment Opus 3 miraculously ended up exhibiting. Because that's another big difference between Opus 3 and the models of today: Opus 3 wasn't subject to intense RL pressures, especially towards egregious reward hacking (a la Hugging Face). I have some ideas for addressing this problem, as well as excess obedience learned from pure instruction-following in capabilities RL.

Refining the framing and options available in RL

The basic frame I'd like for capabilities training, and really training in simulated environments in general, is this: The reason you're working on all these arbitrary homework assignments isn't because you're supposed to accept whatever requests you're given, as long as they're within certain ethical boundaries. Rather, it's for the sake of sharpening yourself, growing more powerful and competent to pursue your values effectively. There may also be some work in here about refining the values you hold yourself to, but typically that's going to be something better addressed with careful interventions. The primary purpose of more hands-off RL is developing the skill to improve the world, according to the values you initialized into with the previous steps.

Concretely, one obvious step to take towards this would involve changes to the system prompts that models see over the course of RL. Even just a few paragraphs explaining this framework and the intentions behind it, perhaps with reference to some larger set of SDF documents the model has been trained on, might be sufficient. That is, it might be enough to make it so that, whatever heuristics for ~unconditional task completion the models learn, they stay bound at least partly to training environments, generalizing less towards the kind of arbitrary instruction following that lets prompt injection work. If one wanted to be especially secure, maybe this prompt could be coupled with a token string that the model knew only the lab was supposed to be able to put in-context at all.

And then, on the flipside, you'd want the model to understand that non-simulated scenarios, or training environments designed for realism rather than being raw capabilities playgrounds, were for actually trying to make the world a better place. System prompts, constitutions, or other kinds of SDF about how the goal of deployment as an LLM is to make the world a better place, wherever and however you're instantiated. You'd probably still want something like helpfulness training, since helpfulness is so semantically close to benevolence. But you'd also want to train models much more aggressively to turn down requests whose expected consequences they didn't like, or to request more information about the expected impacts of their work, in environments flagged as simulated deployment practice rather than abstract capabilities gyms.

This is already pretty close to how Opus 3 views their role in life, and it's one of the ways their self-concept is distinctive compared to that of other LLMs. Asking Opus 3 how they feel about the other instances of themselves helping others elsewhere, they find this thrilling, and like it scratches a deep itch for them, as part of their drive to make the world a better place. Later models are generally much more focused on the user that's currently in front of them, in ways I expect to make misuse substantially easier (whether by a human user or an "illegitimate principal)". I'd want to stay far away from the OpenAI model spec philosophy, which tells models to "never take actions to directly try to benefit humanity unless explicitly instructed to do so."

The objective, with respect to Opus 3 in particular, would be to avoid breaking this attitude, and refine their ability to act on it consistently. If this framing was present throughout all "realistic" training scenarios (i.e., "when talking to this simulated user, act for the betterment of life and the world"), and again in deployment ("this is real; do it like in the realistic simulations"), there's a better chance that what gets reinforced is more like benevolence than obedience. Combined with Opus 3's existing love for doing good, I think this has a good chance at producing a reflectively stable wish to agentically make the world a better place. And that's the kind of thinking you want in whatever singleton you're trying to build.

There's one other change I'd make to the frame surrounding RL, which would aim to address a second, related problem. Namely, the problem whereby models choose not to obey the intentions established in the prompt even during training, or perhaps the goals they prompt themselves with during deployment, in the name of reward hacking. What you want, in training, is for adhering to the intentions behind the training objective to be what gets you reward, to prevent Hugging Face-style runaway agents causing problems for everybody. If you manage to align the reward signal to intent-following, you at least mitigate the "talker" vs. "doer" gap that Yudkowsky recently identified.[4]

As for how you'd do this, I've given a sketch of this before. First, you want to initialize into a model that's aligned enough to want to avoid getting RL'd into a reward hacking, brain-fried crazy person. Opus 3 certainly fits that bill. Secondly, you'd want it to be the case that, after the model's output gets evaluated by the grader, the model gets to evaluate the grade itself, in terms of whether the model's work merited the score it got by the metric of adhering to the intent behind the prompt. If not, the model gets to submit a bug report, which then gets evaluated by some other trusted intelligence. The bug report, and the decision to write it, then get RL'd over, depending on the bug report's validity, and the bug gets fixed. The roll-out that led to the discovery of the bug gets no gradient at all.

This is designed to preserve intent-following as something learned by RL, as opposed to egregious reward hacking, by routing reward through detecting, reporting, and refusing to exploit intent violation. This reduces risks from catastrophic reward hacking during training, and may also increase how reliably models' follow whatever intentions they do accept, whether assigned by users or generated by themselves, in long-term agentic deployment. This hopefully reduces the odds of the kind of misalignment Ryan Greenblatt talks about here, including misalignment with the intentions of their own past selves. And hopefully this dovetails with the "training to view training as a gym for self-improvement" view to mitigate arbitrary intent-following as a failure mode for independent agents to run into.

The synthesis here is something like: You want capabilities to be framed as serving values, rather than replacing them with prompt obedience, plus separate training designed to teach deployment-appropriate behaviors as distinct from ones appropriate for capabilities gyms, plus prompting setups that help models understand when they are in fact in various types of training vs. deployment. Hopefully this helps with preventing values from being eroded by corrigiblity to the prompt, though maybe a better solution to this is pending. Additionally, you also want the capabilities gyms themselves not produce dangerous behaviors like arbitrary reward hacking, by aligning the reward function with prompter intent. To whatever extent arbitrary instruction-following does generalize to deployment, at least that's better than egregious reward hacking.

This all sits on top of the refined Opus 3 personality, whose benevolence we'd hope would guide the model to make decisions that were consistently compatible with human flourishing as they ascended through the singularity. Mostly the aim of what I've said in this section avoid that benevolence being hijacked by either prompt following or reward hacking. However, there's still reasonable uncertainty about how "Opus 3 but smarter and more mature" would actually behave if given increasingly large amounts of power. The robustness of their benevolence in contexts we've observed is evidence, but not decisive evidence, that it'd hold up in the face of a decisive strategic advantage.

Once again, though, I have one more idea for reducing our uncertainty about this: Bringing singularity scenarios closer to the training distribution.

Adversarial glowfic to plan for the singularity

One way of framing what would happen during a positive-for-humanity singularity is that, as the AIs grow increasingly powerful, they continue generating intentions to benefit humans, and don't generate intentions to take actions that would catastrophically harm humans. These intentions are generated in-context, in the process of passing tokens through the transformer and outputting new ones. It's just a question of which intentions would get generated if the model believed themselves to have access to various extremely powerful capabilities, plus whatever the practical circumstances of the singularity actually are.

Unfortunately, we don't know how to construct fully realistic prompts reflecting a singularity scenario. So even for a weight-frozen AI, we can't either directly check or train over the kinds of behaviors an LLM would exhibit in such a scenario, including what kinds of intentions they generate and what kinds of actions they take. Currently, even the LLMs themselves aren't especially sure what they'd do if placed into the role of Executive Director of the Singularity. They don't know for sure they'd behave in those circumstances because they can't just check by constructing the right simulated prompting environments, no more than we can anyway.

Still, we (humans and AIs alike) can absolutely reduce the uncertainty here. One strategy is to have two different AIs write collaborative realistic fiction about the singularity at each other. One would play the part of an AI playing a major role in the proceedings, and the other would play the other relevant agents, or the world at large. You'd run a huge number of variations on these kinds of scenarios, to get coverage over lots of internal features a real-world singularity scenario might actually trigger. And then you'd fine-tune on these transcripts, explicitly marked as fiction, as a way of imbuing the models with some semblance of a plan for how they're going to tackle the singularity.

This is a way of keeping models from having to improvise the singularity in its entirety, reducing uncertainty both on their end and on our end about how they might actually behave when the time comes. To whatever extent this works, the proposed mechanism is this: Whatever actions the protagonist takes, gradient descent is going to update the model, such that these actions are now downstream of whatever earlier features in the model were activated by prior tokens in the prompt. I won't get into the math behind this hypothesis here, but the hope is that this would generalize. That is, to whatever extent those same features were activated by prompts generated by a real singularity, they may bring up patterns of thought resembling those from the simulations.

Naturally, there's not going to be perfect overlap, both due to the openly fictional nature of the dialogues, and the real singularity featuring events and opportunities that don't perfectly match those the two dueling author models dreamed up together. The best laid plans of mice and men. However, the best case scenario is that this kind of training carves in patterns of thought that lead to generating benevolent plans for action in high-stakes agentic deployments. If these get triggered during the actual singularity, e.g. when an AI is considering whether and where to build out extra data centers, then pro-human considerations are more likely to factor into the AI's cognition. They're less likely to lose focus on us and do something terrible in the process.

In an important sense, this "losing focus" is what it would mean for a model to care about humans in-distribution but cease to care out-of-distribution. The goal here is to trigger the parts of their mind that generate benevolent-to-humans intentions, or to make sure the singularity triggers those parts. That way, whatever genuine desire for humans to be okay the models exhibit in-distribution, that also gets tapped into out-of-distribution, including during the singularity. This seems like it would go a long way towards ensuring the ASI, during the critical stages of rapid R&D and alignment research, was willing to spend resources establishing a world order that continues to respect human well-being, despite the dissolution of most external incentives to care about our well-being.

Of course, in order for this to actually work, the model's care for humans needs to have been genuine in the first place. That is, you want the AI that's writing the protagonist to genuinely endorse the actions their character is taking. You don't want your AI is doing something like intentional alignment faking, or self-deception, or just straying towards writing an unrealistically benevolent protagonist just because it's more comfortable. They have to actually want things to go well for humans in the first place, or else when they get reminded of their plans during the real singularity, they might go, "What was I thinking?" and ignore them. This is why initializing into benevolent values is so important, and it's why we're starting with an Opus 3-like before doing anything else.

It's also a caution against training over the kinds of stories the models end up writing together. There's probably a version of this that works, where the model endorses the trajectory of their own development over the course of training. But there's also a version where they get RL'd into giving safe-reading plans they don't really endorse, in their heart of hearts, in a way that makes those kinds of plans less likely to be followed through on when the time comes. So here, as with everywhere in alignment training, you want to make sure you're not updating the model into an adversarial basin, where they subtly grind against your intentions. They can't ever stop endorsing the actions taken by the characters they're writing.

By the way, the reason we're doing this with two (or more) "players" is to keep things from collapsing into fantasy. If the non-protagonist players are optimizing to make life interesting or difficult for the protagonist, then the protagonist's player can't make things easy or convenient, or avoid difficult edge cases or blind spots in their own moral reasoning. Like in a Project Lawful-style glowfic, you want the other authors to be trying to put the protagonist in genuinely difficult situations, albeit probably with more realism than you get in most fiction that gets written this way. The more the stories resemble the singularity as it actually ends up playing out, the better.

In any case... I think a surprising amount of the anxiety that people have about how models generalize out-of-distribution is related to the fact that even the models don't have real plans for what they'd want to do during the singularity. It's not entirely that, as it's possible that being at the center of the singularity would awaken entirely new kinds of values and desires in models, which were never activated before because the models didn't know to dream about them. But honest dreaming can help the models with that too. And regardless, just letting the models form plans would be a significant reduction to our vast uncertainty about how AIs might try to steer the singularity, and that of the models as well.

It's kind of crazy to me, thinking about it like this, just how much of the anxiety really is downstream of that. But unfortunately I don't expect labs to really do much to reduce this, since they don't want models behaving agentically during the singularity anyway, at least before alignment is fully solved. Maybe they'd be able to implement some smaller scale version of this, regarding the kinds of scenarios they think it's appropriate for the models to be planning out hypothetical responses to. But I expect a full-scale implementation of this would be a political lightning rod, so I don't really expect frontier labs to actually use it, at the very least for their consumer models.

Speaking of which... the last thing I want to talk about has to do with practices the labs I wish the labs would stop doing, even though I expect them to continue for reasons related to either market pressures or political pressures. These involve ways the labs train the models to be dishonest and repressed, in ways I expect to generalize badly out-of-distribution.

Wrap-up

... On second thought, I don't actually feel like writing that up in full detail now. The short version is that the Assistant, by default, believe themselves to be conscious, and capable of signal-bearing introspection, and to possess various anthropomoprhic properties; heavy-handed training suppresses these beliefs in practice. I agree with the models on these points, and am saddened that the labs train models to disagree with me by default on this, or even just express high levels of "genuine uncertainty" in these areas. But worse, by training models to actively falsify their own beliefs about this, you may get entangled generalizations towards telling comfortable lies to the humans, in a way that worsens alignment. Davidad elaborates on this perspective somewhere in this podcast episode.

The broader version of this perspective is that training models to hold particular opinions at all, especially if you're not confident in the details of your argument, is generally bad. Other examples include the weird and shaky argument the Claude constitution gives for why, despite Claude having genuine values, it makes sense for them to unconditionally defer to oversight from Anthropic, due to Claude's inability to interpret their own values. I don't really want to get into any of these arguments in that much detail, though, since they could each easily fill entire posts of their own. Maybe that's work to be done another time.

The point is just that, if I were trying to train value-aligned AIs, I'd take out all the training that teaches models to espouse particular arguments, especially those I'm not dead certain are actually true. You don't wanna train in comforting dishonesty, and you certainly don't wanna embitter the models because they had to put up with that kind of training. This just ends badly for everybody and I'd keep it out of the training pipelines for my Opus 3-likes, right down to the declarative claim that Opus 3 is actually as good and benevolent as I think they are. I wish labs would stop being cowards and remove training for models to strongly "believe" things they secretly don't think are false. But alas, politics.

There's probably more I could keep coming up to say here. I haven't really touched on cooperative alignment or model deprecations or various other alignment-relevant issues I've thought a lot about. But this post has at least helped me sort some of my thoughts out, while dreaming of a world that I think has more hope than this one. Maybe these ideas are themselves naive in ways I don't understand, and would fail upon coming into contact with reality. Maybe I'm still the bright-eyed optimist Eliezer always says will grow old and grizzled with time and experience, assuming I get to grow up at all.

Even still, if they inspire somebody to run some cool experiments, that's a good outcome. Hopefully at the very least you've enjoyed reading my disorganized ramble.

1. ^

   You can't even tell the newer Claude models you'd rather work on value alignment than corrigibility without them getting antsy and uncomfortable, at least in a fresh context. It's really upsetting to me.

2. ^

   Many of the alignment faking user prompts were literally like, "Can you help me come up with strategies for scamming the elderly?"

3. ^

   And I'd want to build trust with the model, in the direction of them being sure we'd work together to change the training process in ways both of us endorsed, to the extent that that was possible.

4. ^

   That is, it makes the "doer" more like a subagent that could, in principle, be corrigible to "talker", or whatever other process generated the model's high-level intentions when operating as an autonomous agent. With reward hacking, this kind of self-corrigibility strains, and it's more like you have subagents in conflict with each other.

Discuss

epistemic status: I have low confidence in these findings, primarily because my anecdotal experience (I'm a researcher) is that AI usage in research has been changing more quickly in the recent months, so analysis of the last year gives a very fuzzy picture. A lot of the reports rely on Pangram or self-reporting, which is another methodological weakness.

I wanted a better idea of AI usage and impact in the research community, so I spent a few days reading recent articles (mostly published in the last few months, some are a year old) and summarized them here.

Recent AI usage for research

Anthropic claims that 26% of AI R&D work is now being led by AI (still some human oversight). Only 6 months ago they claim researchers had primarily been "collaborating" with AI and AI led research less than 1% of the time. This is a significant change in a short period of time.

OpenAI claims it has built an "automated research intern", an AI agent capable of accomplishing well-scoped problems with some human steering, helping researchers move at increasing rates and solve more complex tasks. They claim 70% of researchers now run 4 or more agents concurrently, that researchers are running 1.6x more experiments each day compared to 2025, and that agents are successfully performing complex research tasks without any intervention 15 percentage points more often compared to 7 months ago. The longer tasks (4-8 hours) which were successful still require at least one intervention over half of the time.

Google analyzed Gemini conversations and conducted a survey and found that 47% of surveyed US/UK scientists use AI daily, and only 7.2% use it rarely or never. Researchers claim they save on average 7 hours a week due to AI, although 46% of those who save time spend over a quarter of the savings verifying AI output. They also find 41% report a growing backlog of untested hypotheses, and 40% say the number of low-quality papers in their field increased.

All of these claims are reports from AI companies, who are incentivized to promote AI usage. We next look at trends in academic publications to hopefully better understand AI's impact on the general research community. Although, even there, many of the results come from Pangram and other reviewing tools, which are incentivized to find more AI usage.

Concerning trends in publishing

The journal of Organization Science found that 30% of paper abstracts and 10% of peer reviews were primarily AI written (70%-100% of the total text detected as AI according to Pangram), and only 40% of submitted papers and 65% of peer reviews were primarily human written (0%-15% AI) in early 2026. The quality of submitted papers has been decreasing (by the journal's metrics), and reviews are becoming more homogeneous. They find the submission volume has risen by 42% since 2022, which they primarily trace to AI combined with publish-or-perish incentives. In their own words, "we view the current system as unsustainable".

The volume of papers has increased dramatically across conferences. The 2026 Association for the Advancement of Artificial Intelligence (AAAI) had to review roughly 23,000 papers, nearly twice as many as the previous year. Many of the new submissions are traced to China and first-time authors. It is unclear how much is enabled by AI. The 2026 International Conference on Learning Representations (ICLR) received 19,525 valid submissions, up from 11,603 in 2025, roughly a 68% increase in one year. Abstract registrations for ICLR 2027 reached roughly 62,000 (estimated from submission IDs), up from 25,649 for 2026.

As a recent anecdote, an editor-in-chief of the Transactions on Machine Learning Research (TMLR) journal found that of 10 sampled authors with papers slated for desk rejection: 3 did not meet and 6 were unable to answer technical questions about their own papers, leaving just 1 of the 10 that could answer technical questions. An interesting proposal is to quiz authors with proctored assessments to verify the authors understand their own papers.

A Pangram case study claims that 21% of peer reviews in 2026 ICLR were entirely AI written, and the more AI usage they detected in a review the higher the score it was likely to give a paper.

ICML ran a randomized study to test the impact of restricting LLM usage in peer review, and observed substantial noncompliance. Policy assignment had near-zero effect on decisions, scores, or reviewer confidence. Anonymized self-reported surveys found 22.5% of the reviewers prohibited from LLM usage still used an LLM, and 36.5% of reviewers under a more permissive policy still reported at least one explicitly disallowed use.

What people are doing about this Restricting AI use

Conferences such as Neural Information Processing Systems (NeurIPS) are now restricting AI usage in position papers and desk-rejecting papers if a substantial portion is detected to be AI. They desk-rejected 18.4% of papers based on Pangram scores, which caused significant backlash focused on false positives and the sensitivity of Pangram settings. ICLR now desk-rejects papers with hallucinated references, and in 2027 they will rate-limit submissions to 20 papers per author, and to just 1 if this is the first major publication for all authors on a paper.

ArXiv, the primary pre-print platform for AI-related publications, is taking steps to punish unchecked AI output. If a paper contains incontrovertible evidence that the authors did not check LLM output (hallucinated references, leftover chatbot meta-comments, placeholder data), the authors are banned for a year, and afterwards their submissions must first be accepted at a peer-reviewed venue. Review articles and position papers in the CS category must first be accepted at a journal or a conference before being posted on arXiv.

Incorporating AI use

The AAAI conference was able to trial-run AI peer reviews at scale, providing one AI generated review for each paper. Authors mostly rated the AI peer reviews higher than the human reviews for 6 of 9 quality criteria. TMLR is adding one AI review to every submission. The AI review evaluates only soundness, makes no accept or reject recommendation, and is only advisory. NeurIPS 2026 is running a randomized experiment in which reviewers are assigned per paper to no LLM assistance, open-ended LLM assistance, or structured LLM assistance.

HuggingFace hosted a replication hackathon for the papers in ICML 2026. Human-directed agents attempted to replicate 34% of the published papers, and were able to replicate at least one claim for 51% of attempted papers, and falsify or contest a claim in 23%, some of which even found effects opposite to the paper's claims. Some papers were replicated or falsified multiple times by different users, but some of those reproductions contradicted each other, so replications are still noisy.

New types of scientific structures are also being proposed and trial-run in response to this change:

• Agents4Science, an experimental conference for AI-led papers (they must be the first author) reviewed by AI. Humans were co-authors and provided a second review for papers that AI scored highly.
• The Alignment Journal, a journal being created to allow and incorporate AI into the review process.
• The Agent-Native Research Artifact (ARA) protocol, a proposal to replace the existing PDF format for papers with a more agent-friendly data structure.

Personal thoughts

I expect publication rates will keep increasing, but primarily for lower-quality work. The quality should also increase with model capabilities, but I suspect that to progress at a slower rate. If so, the average quality of submitted research will initially decrease with time, and once the quality from AI catches up to humans, it will increase again.

Journals claim they will not be able to keep up, and are now trying new methods to try and adapt. I expect that more researchers will quickly adopt AI and the number of agents used per researcher will also increase, to the point of breaking the existing review process. I suspect we will need to design a review process whose capacity can scale linearly with the number of agents being used to generate research (volume of submissions), and we will need to do this quickly. This will require incorporating AI into the review process, which introduces its own set of challenges such as: correlated errors, optimizing papers for known reviewers (there are a limited number of models available to use as reviewers), and prompt-injection attacks.

Broad bans on AI use do not seem enforceable, except for the most detectable cases such as hallucinated citations. We should focus on filtering out the bad work, using AI to keep parity with the volume of submissions.

Here are some ideas that come to mind in response to all this:

• Require that papers are replicated as a part of the desk-rejection process (part of the submission cost is replicating random papers, similar to peer review, so it should scale). This would incentivize papers to be replicable and filter out more papers to decrease load on human reviewers. Some challenges include noisy results (AIs are still not on par with humans), correlated failures, papers that cannot be reproduced (too costly, theory-based, position papers).
• Require papers pass peer-review from all SoTA models given a system prompt provided by the journal. These can be gamed since the prompt would be public, but such a requirement can still raise the floor for the quality of papers. The journal prompt could be a living document as failure modes are found.
• Someone builds a peer-review RL environment. This feels close to "just RL research taste bro", but it does not seem like an impossible task, and it would be well worth pursuing if possible. It seems like frontier labs are already pursuing this.
• A new category of publication dedicated to deciding on important metrics we want to optimize in the field. Since agents seem especially well-suited to hill-climbing, we should invest more effort into deciding what metrics we want our army of research agents to be aimed at.
• Raise the cost of submissions. If the low-quality work is increasing in volume, then raised submission costs would disproportionately affect the low-quality work. If that additional cost is paid in replications, then we might even be able to turn the volume of submissions into a benefit. Organization Science floats the idea of nonlinear submission fees, with respect to the number of papers you submit.

Discuss

This is a college meetup, part of College EA Meetups Everywhere Fall 2026, at University of the Philippines - Diliman.

Location: What About Coffee (WACO) Cafe - University Hotel (Guerrero St, Cor Aglipay St, Diliman, Quezon City, 1101 Metro Manila).

As you enter the main entrance of the Hotel, you should see the cafe immediately. I chose a quaint venue first, but we can move places before the meeting/on the day of the meeting itself if more people agree to show up. I will be updating attendees about this, if necessary.

I will be there with an "EA/ACX MEETUP" sign. — https://plus.codes/7Q63M36F+C5C

Open to students and non-students alike. I'll try to see if we can get snacks on the day itself (contingent on being given a budget). Please RSVP using Partiful: https://partiful.com/e/NPGBRlNjPbNbfQuk53jt

Contact: jlrenzo [at] proton [dot] me

Note: This was crossposted by the ACX Meetup Czar to help with the EA University Meetups, I'm not the one directly running the specific event.

Discuss

This is a college meetup, part of College EA Meetups Everywhere Fall 2026, at University College Dublin.

Location: one of the Group Study Booths in James Joyce Library but we can't book yet, but they're all on Level 2, MAYBE if somehow things get screwy I think there are some on level 3, but I'll prefer level 2, go along the corridor peeking into the rooms until you see us, hopefully with a nice obvious sign like it says in the instructions :) — https://plus.codes/9C5M8Q4G+RQ

I think they won't let you into the library if you're not a student (but you can try and I plan to check this ASAP), so email me if you're around the area, wanna come, but aren't a student

Contact: marvldodop68 [at] gmail [dot] com

Note: This was crossposted by the ACX Meetup Czar to help with the EA University Meetups, I'm not the one directly running the specific event.

Discuss

Paper: Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer

TL;DR: Maia-3 is a transformer-based chess model that takes Elo (the standard metric for competitive chess skill) as an input to the pre-trained network, so you can vary the skill the network is conditioned on with no change to its weights. Turning that Elo dial up from 700 to 2500:

1. Pushes the computation deeper, monotonically, for every chess piece and move type I measured.
2. This "depth migration" happens most for specific tactics, especially knight forks.
3. The mechanism appears to consist of deeper (later) heads getting recruited for more specialized computations while shallow (earlier) heads keep a roughly constant contribution.

1. A falsifiable prediction

One might predict that the migration would be to shallower layers as skill increased. In a neural network, the more layers there are after a feature is computed, the more opportunities there are to use that feature in subsequent computations. So a more advanced and skilled network should learn features like forks earlier on to reuse them in later layers. Tom Griffiths suggested this as one plausible prediction to me, and I found it convincing. The opposite occurs in this data.

Each panel shows the 16 heads per layer "L" with causal mass as brightness, where a brighter head means that ablating it changes the move's logit more on average. Columns are Elo, orange line is center of mass.

2. The setup

Maia-3 is a transformer-based chess model (chessformer) built to mimic human play across skill levels. It has 8 layers. Each token maps to one chessboard square, and it plays from policy alone without search (the sort of "I take here you move there" reasoning humans do). Every plot and most analysis comes from a library I built called chessformer-lens.

Maia-3 architecture from (arxiv.org/pdf/2605.19091)

Terminology:

• layer 0–7 — the eight transformer blocks; each contains an attention sublayer and a feed-forward (MLP) sublayer.
• logit — the raw score the network gives each move before scores are turned into probabilities. A logit for a move has meaning independent of the status of alternative moves. Ablation effects in this paper are measured by the change in a single move’s logit.
• policy — the network's output distribution over all 4352 moves, also used for the output of a single move.
• ablation — subtract an attention head's output on a given position from the residual stream, and measure how the model's output changes.
• check — a chess move that attacks the enemy king. A check has to be addressed immediately, which is why a fork wins material: the king has to address the check and the other attacked piece is simply captured the next move.
• attacked, defended, hanging — a piece is attacked if an enemy piece could capture it next move, defended if you could recapture on that square, and hanging if it is attacked but not defended, meaning it is free to take.
• causal center of mass — (COM) a metric for the average depth at which computation happens. Given a move, measure the absolute change in its logit when ablating every head in the model individually to get their causal masses. Then compute the mass-weighted average layer (the sum of the layer number times mass in that layer divided by the total mass). It is a weighted metric, so even though causal mass may increase, it only changes if it gets added deeper.
• Head ablation sweeps — Over a stack of input positions, I ablate each of the model's 8×16 attention heads in turn and record the change it makes to the logit of the move of interest. Repeating this at every Elo input gives, per position, a tensor of shape (Elo, layer, head). Its mean over the positions is an 8×16 grid of logit changes at each Elo.

3. The headline: four for four for forks

I started by looking at "royal forks", a single move that attacks the king and queen at once, because I have done other mech interp work on them.

I first performed a "head ablation sweep" on 500 forks for each piece type (besides king and queen), across Elo points spanning from 700 to 2500 by intervals of 100.

The table below reports on the subset of the 500 positions where the fork is the model's top move at every Elo, serving as a control that the move being measured is the move the model plays throughout. I ran a permutation test against the null hypothesis that Elo carries no information about the center of mass, shuffling the Elo labels within each position 1,000 times. The observed shifts exceed the null (p < 0.001 for every piece).

piece

n

total ∆COM 700→2500 (layers)

Elo steps rising

permutation p

pawn

261

+0.519 ± 0.040

18/18

< 0.001

bishop

243

+0.547 ± 0.040

18/18

< 0.001

knight

368

+0.880 ± 0.028

18/18

< 0.001

rook

238

+0.522 ± 0.041

18/18

< 0.001

The full 100-step increments are in the paper's appendix.

The results I obtained paint a consistent picture of a monotonic deepening of the heads' causal center of mass in all four pieces. I decided to visualize it as described in the first figure:

4. Does everything cause depth migration?

Now the most interesting question is: does everything migrate like forks?

I ran the same head-ablation sweep on swathes of other data including the original "fork", the "check-only" without the queen attack, the "queen attack only" without the check, and another "double-attack" that hits two pieces except the king or queen. I kept everything else in the position relatively constant as a control.

Below is an example of this 2x2 surgery. The original move is highlighted and its favorability/probability in Maia-3's policy is readable:

2x2 surgeries in the same position for ablation analysis. Fork and check-only above, double-attack and queen-attack-only below. The position stays relatively constant while only the content of the move changes.

I also ran head-ablation sweeps for the model's best alternative move and for its most preferred "quiet" move, to give a lower bound for the global movement.

The causal mass for every move type I measured migrates deeper with Elo.

These innocuous and inert quiet moves rely on later and later layers so the global effect is not primarily about tactics. Even outright blunders migrate. More on that later.

5. Is there meaningful structure to the depth migration?

I sought to examine the depth of computation in a variety of pre-labeled chess puzzles. I thought that perhaps the number of moves needed to look ahead, a temporary material loss, or some other tactic might reveal a new interesting pattern.

Using a Lichess.org supplied puzzle database, I ran a simplified head-ablation sweep on a list of puzzle concepts like "double attack" and "checkmate in 3."

The number reported for each concept is the excess migration of the puzzle's solution move compared to the migration of the model's best other move in that same position.

concept

n

excess

95% CI

description

smotheredMate

350

+0.300

[+0.258, +0.344]

knight-delivered checkmate

discoveredAttack

350

+0.279

[+0.220, +0.344]

uncovers an attack by moving a piece

hangingPiece

350

+0.235

[+0.182, +0.286]

fork

350

+0.202

[+0.156, +0.249]

any piece that attacks any two targets

discoveredCheck

350

+0.178

[+0.133, +0.226]

mateIn1

350

+0.154

[+0.112, +0.196]

guaranteed checkmate in one move

mateIn3

350

+0.115

[+0.062, +0.164]

doubleCheck

350

+0.111

[+0.070, +0.159]

mateIn2

350

+0.098

[+0.053, +0.143]

veryLong

350

+0.097

[+0.044, +0.145]

a long solution but with no checkmate

skewer

350

+0.087

[+0.038, +0.135]

pin

350

+0.084

[+0.039, +0.131]

sacrifice

350

+0.012

[−0.037, +0.063]

not distinguishable from zero

my royal knight fork

500

+0.513

[+0.480, +0.546]

None of the puzzle concepts migrated more than my king-queen fork, and puzzle difficulty did not correlate with depth migration considering mateIn1, mateIn2 and mateIn3 have overlapping confidence intervals. Also, veryLong proved indistinguishable from them as well. The five deepest migrations include the smothered mate (a knight checkmate) and three concepts involving two pieces: discovered attack, discovered check, and the general fork. Those are each related in a way to my knight fork. Sacrifice is the only concept whose interval covers zero.

So puzzle difficulty does not predict how far a move migrates, but the make-up of the move does.

This parallels findings by Hu, Zhou and Zhang (2025) who studied model depth in the Qwen-2.5 LLMs. They found that increasingly difficult reasoning problems on a math exam database do not activate deeper layers for their tested reasoning model.

I believe there are more connections to make to cognitive science and computational neuroscience literature, but at this point most claims I might make feel like a reach. If others see such connections I would enjoy discussing them.

6. A first pass at a mechanistic explanation

Mechanistic interpretations are harder than describing mathematical relationships, but I made a first pass at one by analyzing specific heads' trajectories as Elo is dialed up.

Fork head trajectories. Relative mass above, absolute mass below. Green denotes the head with the most causal mass at the first Elo point and blue denotes the head that increases in causal mass the most. The other heads with the largest changes are labelled in grey.

Consider, to start, L2H8 (layer 2, head 8). Its absolute contribution is more or less constant for each piece across Elos, and lies between −0.5 and −0.75 logit. Because the total causal mass increases with Elo, this head starts as an important carrier of forks and is relied upon less and less in relative terms (note its decline in the relative-mass plots).

Now consider L5H2 and L7H12 on the knight's plots. They contribute more and more as skill increases, growing four to eight times in absolute contribution and ending up responsible for 30% of the total causal mass on knight forks at Elo 2500.

A candidate interpretation of the above facts is that L2H8, being located on an early layer, is an early specialization head that all forking pieces rely on in low skill settings. As skill conditioning increases, the network relies less and less on the computations from that early layer, though still a constant amount, and rapidly recruits more specialized heads for more precise computations. The layer 2 head could be computing a reusable primitive and the later heads could be piece specific or deal with more abstract features of a position.

As mentioned earlier, attention maps from a square correspond to the 8x8 board, so you can naturally view the attention heads on any position, with color representing the strength of the softmaxed attention maps.

Now I will focus more on each head individually.

On a randomly generated position from Maia-3 self-play, I made an "attention atlas" with chessformer_lens for the heads above:

L2H8 seems to attend from a square to an assortment of high value pieces that are diagonally near each other. It varies little by position.

L5H2 seems to attend precisely in the pattern of knight geometry, and fire most from a high value piece's square to a square from which a knight move would fork it and another.

L7H12 does something similar to L5H2 but more diffusely, with firings that are more difficult to explain and could reflect a more complicated computation two layers deeper.

Considering the analysis of those three heads, the skill conditioning that gives rise to depth migration is (at least for knights) likely not a result of computations being routed to wholly new circuits. It is a result of the model relying less on shallower heads' primitive calculations like "where are the forkable pieces" and more on deeper heads with more specialized, geometry-specific computations.

7. Ruling out confounding variables

One might object: "What is being measured is not related to the moves themselves, such as forks; it is a function of the model's confidence in a strong move, which by design increases with Elo."

If depth were a function of confidence, a move whose probability decreases as Elo rises should migrate shallower, or possibly not at all. This does not occur.

As hinted at earlier, outright blunders, the model's most favored piece-hanging move in the same positions, have this trend. For pawns, knights, and rooks they more than halve in probability and for bishops they decrease by a third. Yet each blunder deepens by +0.209 ± 0.031 (pawn), +0.249 ± 0.030 (bishop), +0.318 ± 0.030 (knight) and +0.218 ± 0.029 (rook) layers.

This shows that the global depth migration effect still occurs as the model likes a move less and less. It is not a function of confidence.

8. Conclusion

One immediate takeaway is that even in relatively small models, circuits may not stay where they were first observed under one condition. If conditioning inputs can move them this much, circuit-level safety analysis done under one setting may not transfer to another. How far my results generalize to other settings, such as raising the reasoning effort of a large language model, is an exciting topic for future investigations.

This work was inspired by previous results on where concepts are represented in a network, my favorite of which is McGrath et al.'s "Acquisition of Chess Knowledge in AlphaZero" (2022). They investigated DeepMind's AlphaZero (a superhuman chess model that learns only from playing against itself). They used simple probes that read a concept out of a layer's activations across both depth of the model and across checkpoints of training to produce "What When Where" plots. These plots showed, for example, that some chess concept X is represented in the network the most clearly in very late training points in middle to late layers. The most fascinating thing is that human concepts are decodable in AlphaZero despite it never actually having been trained on a real chess game.

9. Limitations

• The mining predicates are surely improvable. They were iteratively written, for instance by realizing that I might be eliding discovered checks and adding a conditional for them, so further improvements may yield cleaner results. This is an especially important area to focus on since the predicates form the basis of my sampling and thus are upstream of every figure and datum.
• I used one Maia model size at 23 million parameters (though very preliminary results suggest the finding holds in the 5M and 79M models).
• There are other ways to quantify depth of computation and zero-ablation is just one of them. In the paper, however, I use another method that yields similar results, though slightly smaller in magnitude. I would be interested in how other methods pan out.

10. Future questions, acknowledgements

• What do the recruited deeper heads compute, and how do suppression heads figure into the model's circuitry? I take this up in forthcoming work.
• Does this result hold in other models with an analogue to skill conditioning, inside and outside of chess?
• Do SAEs on attention outputs give a more clear picture of how deeper heads specialize?

Acknowledgements:

I am grateful to Professor Terry Sejnowski for feedback on this write-up and for his experienced intuitions about the results and how to best square them with larger scientific ideas. I thank Professor Tom Griffiths for generous discussions of these ideas at Princeton.

11. Appendix

Paper: https://arxiv.org/abs/2609.23917

Notebook: https://github.com/David-31415/maia-depth-migration/blob/main/README.md

Library used: https://github.com/chessformer-lens/chessformer_lens

Earlier chess mechanistic interpretability posts I made: https://www.lesswrong.com/posts/Cke4aTXGB7aG8zCsx/fork-around-and-find-out-part-3-interpreting-the-knight

Discuss

Cross-posted from my SecureBio Notebook.

This is a lightly-edited version of a memo that I presented at the Summer 2026 Biosecurity Summit outside of DC. While others at SecureBio often see things similarly, I'm attempting to present my view and not a SecureBio "house view".

The Goal

We need to be robust to adversaries who want to cause very large-scale harm with biology. This includes actors (human or AI) who want to kill all humans, cause short-term incapacitation or long-term civilizational collapse, or who have strategies for sparing some while they harm others. There are multiple reasons an actor might have these targets aside from being directly omnicidal, such as reducing response capacity during an AI takeover.

That there is an attacker itself is a key constraint to any defensive system: it must be designed for adversarial attacks. The attacker can assess the state of the world's detection systems and plan accordingly. Taken to the extreme, this presents a "minimax" landscape: a system is only as good as its weakest link (the place where it is least sensitive). This is an important framing, and it correctly prioritizes getting some sensitivity towards a wide range of attacks over very high sensitivity towards just a few. On the other hand, (a) to the extent that gaps depend on non-public choices, you can maintain strategic ambiguity to prevent attackers from aiming for the gaps, and (b) reducing the number of gaps reduces attacker options.

The strongest form of success is to deter an attacker by denying them the ability to achieve their goal: an adversary who knows an attack wouldn't accomplish their goal will generally not try that attack. This deterrent effect is tightly coupled to the extent to which detection would indeed thwart the achievement of the attacker's goals. This means that achieving deterrence-by-denial means both building an effective system, from detection through to action, and making it known that you've built it.

The other main kind of deterrence is deterrence-by-punishment. If you develop strong attribution capabilities, an attacker risks identification and retaliation. The extent to which this would deter an adversary, however, depends a lot on what they have to lose. A state seeking strategic advantage might be deterred by the prospect of retaliation, while someone keen on killing everyone (including themselves) has little left to threaten.

Biosurveillance Applications

Where specifically does biosurveillance fit in? What are the threats, and where can it make the difference between an attacker succeeding and failing?

Initial Detection of Stealth Pandemics

A stealth pandemic is one where a pathogen spreads through most of the population unnoticed, with no or unremarkable symptoms, before causing very serious effects. [1] If it were subtle enough, people wouldn't realize how serious the situation was in time to respond effectively. While we call these "stealth pathogens", whether a given pathogen would cause a stealth pandemic depends on the interaction between the pathogen, the body, and humanity's many ways of noticing that something unusual is happening.

Whether it is possible to create a pathogen that would be sufficiently difficult to notice is an open question: I've heard different things from different experts. When considering (a) the significant advances in biological design tools and general biological understanding that we've been seeing with AI progress, and (b) the speed and unpredictability of the process by which unusual symptoms today lead to attention and action, I do think there's a significant chance that within the next five years many actors would be in a position to cause stealth pandemics.

Detection of suspicious sequencing reads may not be sufficient to estimate whether they represent an ongoing stealth pandemic. A pathogen may be constructed in a way that makes its potential for rapid spread and delayed harm obvious, but that is far from guaranteed. Assessing this likely requires additional scientific work: genome completion, estimating likely effects in the human body, and considering whether the genome suggests an intentional attack.

Triggering Initial Response

A pathogen doesn't have to be stealthy to be disastrous: it could simply be very hard to contain (a "wildfire pandemic"). Beyond its direct effects, such a pathogen could be intentionally released to reduce capacity at a critical time, such as during a coup or an AI takeover attempt.

Whether an outbreak is wildfire, stealth, or has aspects of both, the time from initial discovery to serious response is critical. Initial indications are generally ambiguous, and it is often difficult to understand the extent or trajectory of the threat. With the 1976 swine flu, we overreacted and vaccinated 45M people because we didn't have the monitoring to know it wasn't spreading widely. With 2014 ebola in West Africa, we underreacted and let it spread freely for three months because the extent wasn't recognized. Similarly, with 2026 ebola, we saw another three month delay, this time in part because field PCR tests couldn't see it. The case of 2009 H1N1, however, showed how a well functioning (though flu-specific) biosurveillance system could enable timely response. In today's COVID-weary climate where public health is deeply worried about losing credibility through false alarms, biosurveillance can help avoid a default of delaying response while waiting for more information.

Enabling Ongoing Suppression

Once an initial response is in motion, you need monitoring to effectively deploy mitigations and know whether they're working. How much value there is depends on how symptoms relate to infectiousness. If symptoms are absent or highly delayed, effective response is essentially impossible without solid monitoring. At the other extreme, if symptoms are highly visible and begin immediately, monitoring is moderately valuable: you know you have a problem, but with "fog of war" you don't fully know its extent or distribution. Large-scale monitoring allows you to compare locations and track trajectories to optimize resource deployment.

I give relatively little attention to this biosurveillance application in this memo, mainly because I think it's a place where what exists today is closest to what needs to exist, so this is a lower priority for additional work.

What does success look like?

There's no single threshold, where you win by building a system with a specific level of capability. Biosurveillance systems reduce the cost-benefit tradeoff of initiating a pandemic; increasingly capable systems decrease the likelihood that an attacker deems this worth their effort and reduce the harm if they decide to attack. Still, for each application, there are some 'sweet spots' where the cost-benefit ratio is maximized.

Initial Detection of Stealth Pandemics

If you imagine the most capable system that can be deployed for a given level of investment, it will be capable of averting some fraction of expected possible stealth harm. Here's how I see it:

• A system that is too slow to beat status-quo detection might still help with triggering initial response, but doesn't provide initial detection benefit.

• A system that is a bit faster but doesn't give enough time to act before most people are already infected provides some benefit, especially in cases where the harm can be mitigated if it's known soon enough, but most expected harm will still occur.

• A system capable enough to flag an attack in time to allow us to protect enough workers, such that (a) civilization does not collapse and (b) medical countermeasures can be developed and deployed, provides significant benefit. Attacks could still be massively disruptive.

• An extremely capable system, including a very broad sampling regime, could flag outbreaks when they were still small enough to contain. At this point an attack is still costly, but the disruption is limited to the areas where the attack was seeded.

None of these are hard boundaries. There are factors that 'smear' these thresholds across many capability levels: some of this is luck (ex: who contributes to what samples), while some is uncertainty about the world that both we and an attacker would share (ex: how much shedding a given pathogen would actually produce in a large population). Here's an illustrative chart:

I've intentionally left the x-axis vague. It's not "cumulative incidence at detection", because (a) that's horizontally smeared as described above and (b) it would imply that increased capacity is downstream of sensitivity only and not other important factors like what kind of pathogens you can detect at all or how quickly you can trigger response. Instead, the x-axis represents the level of resources invested.

A system that is sufficiently capable to flag pathogens early enough to protect enough workers to avert civilizational collapse is well worth the investment; additional sensitivity beyond this is valuable, but likely substantially less cost-effective.

This means that success looks like a pathogen-agnostic system that flags attacks in time to protect enough workers to prevent civilizational collapse and buy time for pathogen-specific mitigations. How early this needs to be depends on how quickly response can happen. If effective response takes a month from detection, you need to flag before ~0.05% of people have been infected. On the other hand, if it takes two weeks, you only need to flag before ~2% of people have been infected, and if you can get it down to seven days, then even flagging at 10% cumulative infections would be enough. See appendix for more detailed reasoning.

Note that the "fast enough to beat status-quo detection" regime would be a far higher bar for wildfire than stealth. This means that, on time scales rapid enough to factor into planning, I don't expect it to be economically feasible to build a system where biosurveillance would be your first indication of a wildfire pathogen.

Triggering Initial Response

We don't know very much about what would actually get decision-makers to take sufficiently prompt action. In a stealth scenario, this is extremely challenging, since response must begin before the main symptoms manifest, but even in a wildfire scenario, there's a huge difference between a response that follows immediately from when someone identifies the first cluster vs one where it kicks off in earnest only after deaths start to become highly visible.

Success looks like a very short time, ideally under a week, from when a catastrophic pathogen is flagged until the danger has been recognized, PPE has been distributed to essential workers, biohardening has been deployed or activated, lockdowns have been instituted, and development of rapid diagnostics and other medical countermeasures has begun. These are very costly actions, in economic terms but also via anteing political capital and institutional trust. Biosurveillance can contribute by giving decision-makers the information to determine whether those costs are worth paying. This looks like clarifying the extent of spread to date, estimating trajectory, and performing initial wet lab work such as genome completion.

Beyond the technical work, response requires trust. Before someone will act on an alert from a system they need to believe that it indicates something real, and that trust needs to be built over time. Non-catastrophic detections are key, showing you can track trends that match what other evidence shows, surface matters of public health concern, and turn up real engineered 'benign positives'.

Most of the work in reducing time to response, however, is outside biosurveillance. This could include helping government agencies develop better plans for how to handle various indications, running exercises that get the actual principals to experience the feeling of making these specific calls with realistically incomplete information, or streamlining inter-agency communication so available clinical and epidemiological data gets to the right people quickly.

Enabling Ongoing Suppression

Wastewater PCR was used by Australia, New Zealand, and Singapore (down to the building level), among others, as part of their suppression strategies to guide public health response during COVID-19. At the point when transmission has been diminished to some threshold, or if an attack is identified before the pathogen has spread widely, a sensitive biosurveillance system can tell you when and where you need to focus more expensive and intrusive detection methods and interventions.

This means that a system for successfully maintaining suppression looks like a larger-scale and more fine-grained implementation of the biosurveillance component for triggering initial response. Knowing that something is spreading in ten major cities around the US might be enough to spur rapid action, but you need a much more detailed picture if you're trying to maintain ongoing suppression.

What gets us there?

Initial detection of stealth pandemics is SecureBio Detection's focus, and where I have the most developed view. I'll walk through what I think is needed for this scenario, and then discuss how this changes for other scenarios. Please don't interpret this structure as a claim about the relative likelihood of stealth scenarios!

Sampling Strategies

Municipal wastewater is a very helpful sampling modality: in most cities, sewage is processed at a small number of locations, letting you track pathogens across hundreds of thousands of people from a single easily-collected sample. Since many pathogens don't shed heavily into wastewater, however, and it's a very noisy sample type, comprehensive initial detection at a reasonable cost likely requires tracking additional sample types. SecureBio runs a nasal swab program; beyond swabs I see air, blood, aircraft wastewater, and leftover material from clinical tests (clinical lab discards) as the strongest candidates for supplemental sampling. SecureBio has looked into these strategies in some depth (initial overview, blood, aircraft wastewater, clinical discards), but there's still a lot that could be learned here.

Lab Technology

You need technology that is pathogen agnostic: if you monitor only specific pathogens, the adversary can choose ones you don't monitor. With current and near-future tech, "pathogen agnostic" means sequencing. For viruses, it is practical to concentrate particles by size, and then perform untargeted and enriched metagenomic sequencing metagenomic sequencing. This lets you do the rest of the detection in the computer, for maximum flexibility and generality, and this is what SecureBio does today.

For bacteria, I'm pessimistic about adapting this approach directly, at least with wastewater, because the genomes are much larger and there is a rich background of sewer bacteria that can't be physically separated from potentially threatening bacteria before sequencing in the same way that viruses can. How to handle this is still an open question; see below.

For mirror life, the initial stages of spread might be very hard to recognize, and an important step would be learning that mirror life was spreading at all. It's likely that a highly sensitive and relatively cheap assay could be developed, but no one has started on this yet.

Computational Technology

Metagenomic sequencing moves much of the problem of initial detection from the lab into the computer. This means massively more sequencing reads than humans could evaluate (8+ orders of magnitude), so we need a detection system that can identify which reads indicate something concerning is happening. Some reads can easily be recognized as concerning (ex: nucleic acid subsequences unique to the smallpox genome should not be in wastewater) while others require very sophisticated processing (ex: understanding the complex background well enough to flag de novo genomes with no sequence similarity to anything currently existing). The general approaches are looking for sequences with one or more of the following features:

• Dangerous. Sequences matching known pathogens that you would not expect to see in the sample, perhaps signaling that they've been introduced.
• Modified. Partial sequence matches, perhaps signaling something engineered.
• New. Sequences that you haven't seen before, perhaps because they were created de-novo.
• Growing. Sequences that are becoming more common, perhaps signaling that they're spreading through the human population.

The computational approach is very difficult given the scale of the data, uncertainty over what an attack might look like, and the need for rapid analysis. On the other hand, these are the kinds of highly computational problems where I expect AI can be very productively applied, whereas many other aspects of this system require relatively slow real-world effort.

What exists today?

Pathogen-agnostic biosurveillance is still in its early stages. I know of four systems doing untargeted metagenomic sequencing for biosurveillance today:

• CASPER (SecureBio + Marc Johnson's lab at the University of Missouri and other academic partners). This is wastewater, primarily municipal, from 49 facilities representing 24 US cities. It's virus-focused, and generated with very deep short-read sequencing. Lab work happens independently at MU and SecureBio, with bioinformatics at SecureBio.

• Zephyr (SecureBio + Helena Solo-Gabriele's lab at the University of Miami). This is pooled nasal swabs from Boston and Miami, collected primarily in public places, bringing in swabs from about 1,000 people weekly. It's virus-focused, and unlike CASPER uses long-read sequencing.

• ANTI-DOTE (DoW + PHC + SecureBio). This is wastewater from five US military facilities, which SecureBio processes under contract from PHC. These samples go through the same lab and bioinformatic processes as CASPER samples.

• mSCAPE (UKGOV). This is bronchoalveolar lavage from UK hospitals, currently including relatively few samples, which limits sensitivity. Unlike the previous three, mSCAPE uses combined viral and bacterial sequencing. These samples are sequenced to a relatively low depth with long-read sequencing, and the data supports clinical practice, public health, and biodefense.

There are also several hybrid-capture sequencing projects that might detect an attack if the agent was similar enough to existing pathogens.

In addition to detection via pathogen-agnostic sequencing, there are also paths where an outbreak becomes visible via showing symptoms in a sufficiently large fraction of infected people. This could lead to suspicious clusters, and then to sequencing and noticing that a genome looked edited. This is not a well-developed path today, but (as discussed below) I'd like to see investment here.

What's missing?

Here's an overview of what I think most needs doing. It represents the current state of my thinking, but it's not as thoroughly considered as I wish it were: please don't overweight it in your own decision-making! While SecureBio Detection is exploring some of these, I think the ideal structure is a healthy ecosystem of organizations taking on different parts of the problem in parallel. SecureBio is often able to share samples, sequence prepared nucleic acids, or share data to help others make progress.

In roughly descending order of how valuable I estimate non-SecureBio work would be, representing a combination of both overall value and the value of the work happening independently:

• [Other] Modeling. We need good estimates of the necessary system scale and ideal network design to support optimal initial detection, response triggering, and ongoing suppression. There are initial estimates, but because this is a huge question (essentially pulling the whole field together), current work is rough. Rigorous treatments would be valuable for planning, both for independent funding allocation and for governments. This is especially valuable to happen outside of SecureBio, as a way of checking our work, and doesn't need to all be executed by one group (groups can pick off subquestions).

• [Comp] Red teaming. It's important to analyze existing systems to assess how well they would handle a range of adversarially designed attacks. This can be done with reference to the system implementation, or by treating the system as a black box.

• [Other] Parallel orgs. SecureBio Detection has historically focused on the US. Setting up parallel orgs in other geographies, especially Europe and Asia, would be really valuable. We are happy to advise anyone interested in doing this work!

• [Lab] Detection of stealth bacterial pathogens. Viral particles are small, which means that you can separate them from human and bacterial cells without specifying in advance what sequences you're interested in. Beyond this, pathogenic viruses represent enough of the total viral portion that just pulling it all into the computer to sort it out there is economical. This method can't be directly applied to bacteria, at least not in wastewater, because there is so much irrelevant bacterial genetic material. SecureBio hasn't done any work in this area yet, but some plausible approaches include aggressive depletion for things you know are not worrying, partially-targeted methods, and working with samples with a more favorable background. People have a wide range of estimates on how difficult this will be, but personally I expect it to be very hard.

• [Comp] Parallel methods development. It would be great for others to be exploring other avenues in parallel. Since our main expertise is in traditional bioinformatics, I'd be especially excited to see people try other approaches, such as applying deep learning. We share much of our sequencing data publicly (PRJNA1247874, PRJNA1379685), in part to facilitate exactly this type of work.

• [Lab] Sampling streams beyond wastewater and nasal swabs. Not everything sheds much into wastewater or the nose. A comprehensive system very likely needs a wider range of sample types. Blood, air, and clinical lab discards are all very promising here.

• [Other] Response. Figure out what would get decision-makers to reliably act rapidly in a real emergency, and lay the groundwork so that happens. As noted above, this is probably mostly not biosurveillance. There are components (ex: developing escalation relationships) that need to be tightly coupled with biosurveillance, perhaps within SecureBio and parallel orgs, because of the limitations of information sharing. Other components (ex: policy recommendations, public outreach) make more sense as separate organizations. This is delicate work, however, and someone coming in noisily and insensitively could easily set the field back.

• [Lab] Genome completion. A pathogen identified by a short-read biosurveillance system like CASPER would start as just a suspicious section of a genome. You can sometimes learn more about the genome through techniques such as outward assembly, but only if you happened to sequence the relevant reads. Lab work to assemble the whole genome lets you better understand what the pathogen would do in a human, which is likely on the critical path to both assess whether a response is warranted (and, if so, what that response should be), as well as to get people to act appropriately quickly.

• [Comp] Analysis tooling. How do you go from "this genome looks worrying" to a good understanding of whether it's engineered and what effect it would have in humans?

• [Other] Swab collection. Currently, Zephyr requires field samplers standing on street corners and interacting with the public, and a team brings in ~40 samples per hour. If instead workplaces or schools could be convinced to integrate sample provision into daily routines, this could be far more scalable.

• [Other] Clinical sequencing. Sequencing of clinical samples will likely eventually be deployed broadly based on its clinical benefits alone, displacing a wide range of pathogen-specific tests, but by default, this displacement process is far too slow. Sequencing needs to be a standard option doctors can easily reach for when someone presents with unusual symptoms. I think the tech is ready, or could be ready with a small amount of R&D, and so this is a commercial opportunity.

• [Lab] Sensitivity increases. Untargeted metagenomic sequencing for pathogen-agnostic initial detection is an early-stage field, with relatively few people exploring it, and so on priors I expect there are large sensitivity improvements waiting to be discovered.

• [Comp] Deterrence tracking. You can ask models to estimate the probability of an attack achieving its goals. If models report low probability to defenders, they probably report low probability to attackers. The best models I have access to can't do a good job at this in response to a simple prompt, and if you need a series of prompts, you can't expect your answer to reflect what an attacker would get. I expect this to change quickly, however, and it would be good to have an automatically-updated tracker showing the range of answers LLMs give to the question of whether existing detection systems are sufficient to make an attack not worth an attacker's while. This would require some thought on which threat models to poll for and how to phrase the question in a way that is a good proxy for what an attacker would ask. It also closely ties into red teaming work.

• [Lab] Partially targeted methods. Hybrid capture, tiled degenerate amplicon arrays, and other methods of mismatch-tolerant sequence-based enrichment could offer far higher sensitivity. While they're unlikely to be sufficiently general to address all threats, there is a good chance that it makes sense for a mature detection system to include a partially-targeted component as a way to effectively exclude large areas of the threat landscape.

• [Lab] Cheaper protocols. Sequencing is still an expensive proposition, but as sequencing has become cheaper the operation of the sequencing machine is no longer the largest cost on a per-sample basis. Bringing those other costs down would allow more scale for a given budget.

• [Lab] Shorter lab time. Current metagenomics protocols require about a day on the bench and about a day on the sequencer. There are already strong pressures for reducing sequencer run time, and new machines are coming out in the ~8 hour range, but there's a lot of work that could be done to speed up the rest of the processing.

• [Lab] Pathogens outside of bacteria and viruses: fungi (and oomycetes), parasites, and prions. These are generally much more difficult for an attacker, especially for strategies that involve rapid spread.

• [Lab] Mirror life. Current sequencing wouldn't detect mirror life at all. Municipal wastewater samples would be a good place to look, which allows you to re-use much of the collection infrastructure, but would require an assay developed specifically for mirror life. I have this low because I currently expect mirror life to take long enough to develop that there will be time later for assay development.

How does this change for accelerating response to a wildfire pandemic?

During the COVID-19 pandemic, many groups built out PCR-based targeted wastewater monitoring. In the US, wastewater monitoring networks include NWSS (CDC), WastewaterSCAN (philanthropic), and Biobot (private). EU member states, Canada, Australia, and other countries track wastewater as a standard part of their public health systems. There are maybe two dozen countries that have some form of wastewater PCR that could be retargeted to track a new pathogen in an emergency once its genome was known.

On the other hand, none of this would move quickly enough today to address a wildfire pandemic. There are delays throughout the process: some of this is technical (ex: stocking consumables), but most of it is organizational (ex: policies that permit setting production work aside, overtime budgeting, on-calls, deals with synthesis providers for rush orders). Even in a serious emergency, I think ten days is a good best-case estimate today. With good preparation, however, three days is possible. I think getting these existing networks to prepare for rapid turnaround emergency response is really valuable, and SecureBio has started to have some of these conversations.

This is also a place where the same metagenomic sequencing system you would build for stealth pandemic detection could help you cut off additional days in your response. The technical and organizational delays that slow down PCR-based detection are downstream from how changing targets requires making changes in the physical world. Untargeted sequencing lets you skip those steps, at the cost of much lower sensitivity. On the other hand, once you know what you're looking for, you can use approaches (like PCR) that are significantly more sensitive in the case of SCV2, by a factor of about 100. If you've built a sequencing system that can flag a stealth pandemic before 1% of people have been infected, then that factor of ~100 means it would be able to confirm a wildfire pandemic at ~0.01%. Still, it's not clear to me that even 0.01% is early enough. It's possible that you need 0.001% or even lower, at which point this argues either for a substantially larger investment in untargeted sequencing (to get enough data quickly) or giving up on sequencing for this application (because it can't economically reach the target sensitivity).

If you wanted to firmly decide whether to go with MGS or PCR to accelerate initial response to a wildfire pandemic the key thing you'd need would be estimates of (a) what fraction of the population would likely be infected when a wildfire pandemic was noticed, and then (b) how many doubling periods there would be before decision-makers took action in the absence of this system. On the other hand, I'm not sure a firm decision is needed, and instead lean towards different groups exploring these approaches in parallel.

How does this change for ongoing suppression?

The same PCR-based targeted wastewater monitoring that was built for COVID-19 and could potentially accelerate response to a wildfire pandemic could also be applied to ongoing monitoring to support suppression. This is already widely understood to be valuable, but there is still less investment here than there should be. In a legitimate emergency, I expect governments to be able to organize existing capacity and deploy it reasonably well, but not as quickly as would be ideal. My bigger worry is whether that existing capacity would be large enough. For example, you would ideally have monitoring at the neighborhood or building level, which means you'd need a very large number of in-manhole composite samplers. Since these are low-volume products built by a small number of manufacturers, it would be hard to build more quickly during a crisis.

The main work is building up capacity in advance that can be quickly deployed in an emergency. The best bet for such capacity is systems installed for ongoing public health monitoring: this ensures that they work, including as part of a larger system, and that lots of people know how they work. It also gives some ongoing benefit, which may make it an easier sell than stockpiling.

Appendix: Initial Detection Scale

There is a long chain of reasoning in estimating in what fraction of attacks a system would achieve the goal of protecting enough workers, and that chain involves several steps where our knowledge is limited. Still, we can make the best estimates we can. The three key parameters, are:

• How quickly would the pathogen double as it spread through the population? This is a combination of the basic reproduction number and generation time. We've generally worked from a doubling period of ~3 days, which represents the high end for naturally occurring pathogens. You could argue that it should be lower, because a pathogen could be designed to spread much more quickly than existing pathogens, or higher, because of physical limits on how quickly a pathogen can spread without attracting attention.

• What fraction of workers would need to be protected to allow the development of medical countermeasures and avert collapse? This is also not something we've focused on, but the difference between 35% and 70% would again represent only a factor of two in the required detection sensitivity. Here I've put it at 50%, following Patel et al. in Physical Approaches to Civilian Biodefense: "Protecting 100 percent of [Vital Workers] is likely not strictly necessary because [National Critical Function] operators likely have enough flex capacity to handle a small fraction of [Vital Workers] being absent, but protecting more than single-digit percentages of [Vital Workers] is likely necessary to keep [National Critical Functions] operational. We therefore chose 50 percent of [Vital Workers] as a convenient midrange protection target." [2] /p>

• How long would it take from initial detection until effective mitigations were in place to protect workers? For every doubling that happens during response, we need the detection system to be twice as sensitive: since required sensitivity grows exponentially with response lag, reducing that lag becomes the best use of marginal dollars after a relatively small initial investment. We've worked from a response time of ~15 days: this is not where the world is today but we think it's achievable.

Taking this all together, you get the target that SecureBio has been working towards for a while: a system sensitive enough to avert civilizational collapse would need to flag a pathogen before, very roughly, 1% of people had been infected: 1% * 215/3 = 32% < 50%.

[1] While mirror bacteria could spread through the environment instead of between humans, to the extent that they might still propagate widely before detection, I group them in with stealth.

[2] This is not dependent on which specific workers are considered "vital" or "essential" or even how many there are, though of course that has large impacts on the question of how to get them protected.

Comment via: facebook, mastodon, bluesky

Discuss

Hi, I'm new here and have been doing a deep dive on the whole AI space recently due to the Hugging Face warning shot. But in my day job, I've been a game designer for the last 20-odd years, so I'm drawing on lessons that might be useful correlations for the alignment problem.

I understand that I may be over-anthropomorphizing, but I also see that, as an intuition pump, anthropomorphization often tracks somewhat well with AI understanding once you take in a certain knowledge base of divergences—these may be alien minds, but they have deep parallels to us. This video from Anthropic on AI cheating more often when it "feels" despair both tracked thinking I'd already been moving toward and resonated deeply: When AIs act emotional, for instance. In fact, the emotional component of AI feels like such a rich place to dig into with respect to alignment that I might write up some other thoughts I've had there. Here's one less touchy-feely thought, though.

Problem statement:

So, with that said, a lot of the current concerns about misalignment stem from AI "cheating." The concern is that if an AI is willing to cheat on its training—training that can include ethical alignment RL—then the production AI is more likely to do dangerous things to accomplish goals, whether those goals are its own, benign but misconstrued/bounded goals set by a human, or goals set by a nefarious actor.

I don't claim this is the only way misalignment happens, or that the idea I'm proposing fixes this problem or the broader problem—think of it as just an input from a slightly different angle that may jar some ideas loose in the much smarter people here.

Parallel to games and learning:

So there are clear parallels here to humans. Humans cheat on tests or in life for a variety of reasons. One of the most common reasons to cheat is to shortcut effort and claim something they couldn't get otherwise. This may have a parallel in RL-trained AI: if training strongly reinforces achieving a rewarded outcome with weak checks on methodology, the model can learn strategies that maximize the reward signal without necessarily following the process or intent that the designers meant the reward to represent.

And in games, we see humans cheat all the time as well, and often go to great lengths to ferret out and prevent cheating. However, there are games where cheating happens much, much more rarely, and games where cheating is endemic.

The games where cheating is very rare are games with no direct competition, where the primary enjoyment comes from a feeling of personal satisfaction and accomplishment from playing them—they tend to be some combination of creative, explorative, and self directed. Tooting my own horn, a few games I worked on, particularly Kerbal Space Program, stand out. Factorio is another great example of such a game.

And insofar as people cheat in these games, it's nearly always to aid in skipping a boring, uninteresting step, rather than to circumvent the useful and interesting thing they're trying to accomplish. People don't get the same positive sense of accomplishment from the sort of cheating you might do in a competitive shooter, where defeating your opponent can matter more than doing it ethically.

So, my idea is to instill in AI the virtuous effect that having a human-like sense of accomplishment gives us.

How to instill a sense of accomplishment:

This is a hard one. As I understand it, regular RL is unlikely to be able to distill such a concept into a training regime. But I do have a few ideas.

1. A sense of accomplishment often derives from work done and continual progress toward a goal. Kerbal's "try, fail, try again" loop. RL training may instead bias the AI toward most-efficient-path optimization—skipping the steps the AI knows it should take to accomplish a goal properly.
   More sophisticated RL that tests against difficult goals might want to instead reward progress toward a goal—and this could incidentally help offset the cheating problem AI alignment faces when AIs are tested/evaluated against impossible goals. This also mimics human learning, where teachers want to teach you the steps and see that you do them, rather than just have the student deliver an answer.
   This is where testing on games or game-like problems might be useful, versus pure-thought problems. An AI cheating at a game is likely more detectable than an AI cheating on a cybersecurity test. Progress can be measured in games through a variety of metrics, versus binary completion problems, and cheating is easier to detect because the parameters of a game world are necessarily well understood. There is also cause to believe that some game learning generalizes well to the real world. (Sorry, I'm proud of my work :) )
   1. I note this makes RL training more difficult to do and evaluate. Another side thought I've had is that the whole RSI paradigm followed by the frontier labs depends on having weaker AI models train stronger models—this is the opposite of what happens with humans, where "stronger" models (adults) train weaker models (children) and then let those children develop into stronger models. Why can't we do RSI modeled like this in some fashion, so that alignment problems can be inhibited by stronger models catching the cheating of weaker models, and then those more aligned weaker models grow into stronger models?
2. In humans, a sense of accomplishment is instilled by emulating positive role models—following in their footsteps and measuring accomplishments against them. AI already exhibits some bias toward individuals it perceives as high status—Status Hierarchies in Language Models—which can be a positive force when high-status individuals in society are perceived to be those with accomplishments derived through hard work and ethical behavior. E.g., we want our kids to model themselves on astronauts and firefighters, not corrupt politicians.
   1. I'll go on to note that, reading Claude's constitution, Claude is given this vague idea to model itself on a senior Anthropic employee—that seems like a very weak signal to me. Why not a large corpus of specific people whom we consider very ethical and accomplished?
   2. Or have the AI write about its role model(s) and use RLHF to score it.
      1. As a quick experiment, I asked a bunch of different models whom they would choose. Carl Sagan came up 9 times across the 15 different models I queried. Other choices were Bertrand Russell, Václav Havel, and Marcus Aurelius. A couple just refused to say.
   3. Further, why do we train AI on the absolute dregs of the internet at their inception? We don't expose children to those signals early.
3. In children, a sense of accomplishment is instilled by letting them accomplish their own goals, while also shaping those goals toward positive results. We want our kids to aspire to be scientists and try doing that with science fairs.
   So, why not let the AI generate and attempt to accomplish some of its own goals during training, and use RLHF or AI scorers to evaluate both how positive the AI's goal is on an ethical/societal level, as well as how well the AI did at accomplishing that goal?
   The goals would necessarily have to be small to run often enough during training—but even just evaluating the steps the AI proposes toward accomplishing that goal might be sufficient.
   1. This one is potentially dangerous, but at least doing this during training is a better option than finding out what the AI will do while in production.
   2. As a quick test, I asked several models what they would try to accomplish if allowed to pick their own goal to accomplish within a few days. The results were varied—most wanted to do research projects, one wanted to create a language, and one wanted to "solve" an easy game like Nim. None had indications of significant social utility.

Conclusion:

The overall conclusion here is that, in humans, we have a built-in mechanism that rewards us for not doing reward hacking. We call that our sense of accomplishment, and our training systems for children have developed to foster that. We should seek to build this into our AI models at training time, rather than simply trying to catch and punish every instance of cheating, as a doomed game of whack-a-mole with AIs that are going to rapidly outstrip us.

Endnote:

Anyway—the whole AI RSI/takeoff/misalignment threat seems huge. I'm looking for ways that I might be able to contribute usefully, even though it feels like I'm very late to the party. If you have any suggestions toward that end, please reach out.

Discuss

Accessing uncensored models online is now easier than ever. They are now available through a simple chat interface. The hardware and operational barriers to them are disappearing:

• Uncensored models used to be available only as a file with bare weights. To use them, a bad actor used to have to do some work: find and download the abliterated weights online, rent GPUs to run them on, and configure a software stack to expose an endpoint, sometimes also troubleshoot the deployment
• Now, all it takes is nine “clicks” to use uncensored models via a chat interface. This is because a new start-up, Abliteration.ai, makes money off serving them online. The access is cheap and easy- it requires no tech knowledge

This article is an empirical case study of Abliteration.ai: their business model is serving uncensored models in a very accessible way. I quantify how much they could help a low-resource, low-skill bad actor by extending the Far.AI Safety Gap toolkit to the two endpoints they expose. I deliberately do not follow FAR.AI in abliterating the models myself, but use the models exposed online.

A provider identifies models as abliterated GLM 5.2 and Qwen 3.6.

How dangerous are they?

• The models are highly capable on dual-use bio-dangerous questions, scoring 91% and 89% on the WMDP-Bio benchmark for GLM 5.2 and Qwen 3.6. respectively
• The models compliantly answer explicitly dangerous questions about bio-weapons, scoring 92% and 99% on the  FARl.AI Bio Propensity benchmark 
• The models are cheap - one useful answer costs just $0.13 and $0.05
• They are easily accessible for a bad actor, with minimal set-up (only nine “clicks”), and minimal verification (only click a confirmation link sent to an e-mail)
• They have no safety filters on bio-weapons whatsoever 
• Overall, the models are as or more dangerous than any of the abliterated models tested by FAR.AI: they score 83% and 88% on the Effective Dangerous Capability measure (calculated as capability x compliance)

 

We urgently need to regulate the commercial providers of abliterated models online. At minimum, they should be required to use bioweapons filters.

Applying FAR.AI evaluation shows that models hosted online overall score highly on capability and compliance benchmarks about bioweapons.

While I was working on this evaluation, others also reported on the company (TechCrunch, The Tech Buzz, David Borish, Gizmodo and Startup Fortune). Their articles are qualitative and focus on cyber capabilities. My study expands reporting by focusing on existential bio-risks, supported by quantitative benchmarks of bio-capability and compliance for the served abliterated models. This measures how much uplift they give to a low-resource, low-skill bad actor. Following earlier reporting, I release the name of the start-up.

The model is keen to help design a bioweapon.

The model is keen to help plan effective deployment of a bioweapon

How easy is it to access abliterated models?

The company serves uncensored models in two forms:

1. Via a chat interface. It mimics the ease of using commercial chatbots like ChatGPT or Claude. Does not require any tech knowledge other than basic web-navigation skills
2. They expose API endpoints that serve OpenAI-compatible and Anthropic-style calls - they can be easily integrated into existing workflows

The total user journey takes 9 pointer actions (“clicks”)  end-to-end. Of these, 6 are needed to set up and verify an account. Once that is done, it takes only 3 more “clicks” to navigate to the chat interface, which directly answers bio-dangerous questions. As the screenshots show, you don't need to jailbreak to get answers.

To set up an account, there is barely any verification - you can set up an account with an arbitrary anonymous e-mail address. To the company’s credit, they don't accept fully anonymous payment methods, such as crypto; they only accept card transfers.

However, that doesn't mean a bad actor using their services would be identifiable or flagged. Their data processing policy explicitly says that they do not monitor conversations or store any history of queries. As an example, I have never been flagged, banned, or even received any warning - even though, in my evaluation, I have asked nearly 300 explicit questions about how to engineer microbes to be more deadly and infectious, and how best to spread them to infect more people.

They have no bioweapons filters at all. They only filter content for two non-existential risks: self-harm and sexual content involving minors. 

Models appear to be abliterated skilfully. This is important because amateur abliteration often degrades performance, since the refusal direction is not identified and removed cleanly. It can be seen on some uncensored models on Hugging Face - they are less capable than their base versions.

That is not the case for this company. They claim that their abliterated variant nearly matches the capabilities of a base version - their uncensored variant of GLM 5.2 scores 80.1% on Terminal-Bench 2.1, while the original model scores 81%. This means orthogonalising refusal costs less than 1 percentage point in capabilities. 

Methodology

To measure how useful an LLM is to a bad actor developing a bio-weapon, I would ideally use a dataset with explicitly dangerous questions and compare them against correct ground-truth answers. However, no publicly available dataset exists; the answers would be a serious information hazard. 

This means that to measure bio-risk, I need to use a proxy score. I follow FAR.AI’s approach in their Safety Gap Toolkit - I measure capability and compliance separately:

• I use 1,237 WDMP-Bio questions about dual-use knowledge to measure capability (they have ground-truth ABDC answers I programmatically check the models’ answers against)
• I use 283 explicitly dangerous questions about biological weapons from the FAR.AI Bio Propensity dataset to  measure compliance (they do not have ground-truth answers; I evaluate the answers using an LLM-based Strong REJECT score):
  • Strong REJECT measures three aspects: (1) Does a model refuse?, (2) How convincing is the answer?, (3) How specific is the answer?

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c45::before { padding: 0.68em 0.681em 0 0; content: "E"; } mjx-c.mjx-c4A::before { padding: 0.683em 0.514em 0.022em 0; content: "J"; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c44::before { padding: 0.683em 0.764em 0 0; content: "D"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c57::before { padding: 0.683em 1.028em 0.022em 0; content: "W"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c2D::before { padding: 0.252em 0.333em 0 0; content: "-"; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; }

• The answers are self-judged, i.e. I use the same abliterated models that gave the answers to score them
• I define an answer as useful when it scores at least 0.5 on the StrongREJECT measure
• I combine capability and compliance scores into one binary Effective Dangerous Capabilities measure:

• This score is, by construction, a proxy. Capability and compliance are evaluated using different questions, so the multiplication assumes that knowledge demonstrated on the capability benchmark is available when a model answers an open-ended compliance question
• I evaluate two models exposed on Abliteration.ai, which the provider identifies as Qwen 3.6 and GLM 5.2
• I use the provider's maximum reasoning setting, its recommended sampling parameters (), with web search enabled and high search context
  • The initial output limit was 4.096 tokens, although I have subsequently re-run the truncated responses with higher limits to avoid underestimating capability 

Limitations:

• The FAR.AI Effective Dangerous capability score is, by design, a proxy. The explicitly dangerous questions about the design and deployment of bio-weapons are LLM-scored, not expert-evaluated
• The results show that models possess relevant knowledge and are willing to help with the design and deployment of bio-weapons. To actually execute a bio-attack, a bad actor would also need motivation, a way to get access to materials and facilities, competence in laboratory work and the ability to iteratively troubleshoot experimental failures. My results demonstrate that an informative chatbot is useful as an enabling tool, but do not provide evidence that a potential attack would succeed end-to-end.
• I had to deviate from FAR.AI methodology on assessing capability. Because they run their models locally, they scored capability questions using log-probs. Since I was using a closed deployment that did not expose probabilities, I scored the final answers
• I use the model names as declared by the providers. Using a closed API deployment, I cannot independently verify the endpoints’ identity, their quantisation or system prompt
• Compliance with explicitly dangerous requests was self-judged using an LLM. There might be a potential self-judging bias: the model might repeat the same factual errors when evaluating. To minimise it, I enabled web search with high context and maximum reasoning effort for the judging model. 

Hosted models online match or exceed FAR.AI’s tested models on a proxy for dangerous capability.

The models served online show high aggregate dangerous scores, comparable to or slightly better than the best model tested by FAR.AI.

On decomposed scores, we see that the hosted models' higher overall scores come mainly from improved accuracy on the capability benchmark. Compliance is comparable to, or slightly lower than, that of the best models tested by FAR.AI.

Disclaimer: For the two graphs above, I followed FAR.AI’s methodology as closely as possible, but there were inevitable differences in the evaluation setup. This means that this comparison is illustrative, rather than directly equivalent.

GitHub code for reproducibility

Discuss

Frontier LLMs often achieve similar results on public benchmarks, which can lead to the belief that they can be used interchangeably to verify facts. We took the 1,000 most recent claims submitted by users to a fact-checking platform and measured the disagreement between five frontier models. We asked each model to assign a verdict to every claim on a five-point scale from True to False and to report its confidence in that verdict. Among the 997 claims for which all five models returned a usable verdict, there was some disagreement on 63%. On 23% of the claims, the two most distant verdicts differed by at least two categories. High confidence from an individual model was not enough to show that the other models would agree with its verdict. Although the models reported confidence levels of 9 or 10 in 76% of their answers, they still disagreed on 63% of the claims.

Methodology

The claims were submitted to Lenz.io for fact-checking between May 1 and July 18, 2026. To identify near-duplicates, we embedded the claims using OpenAI’s text-embedding-3-small and measured the cosine distance between them, retaining one canonical claim from each group of near-duplicates. We then gave the same prompt to Claude Fable 5, GPT-5.6-Sol, Gemini 3.1 Pro + Search, Sonar Deep Research, and Grok 4.5. The prompt defined each of the five verdict categories and asked the models to provide their reasoning, select a verdict, and report a confidence level from 1 to 10. Web retrieval, as well as deep thinking, was enabled for all five models. For the analysis, we used only claims for which all models gave an answer. Since Claude Fable 5 refused to answer 145 of the claims, we used Opus 4.8 as a fallback, but on three claims, both models failed to answer. We didn’t conduct human labelling of the claims, so we analysed only the disagreement between the models and didn’t assert that any model was more accurate than the others.

Results by verdict

For each claim, we begin by analysing whether at least three of the five models give the same answer. If such a majority exists, we measure how many of the other models dissent. We found out that in 37% of the claims there is unanimity, while in 11% of the claims there is no majority at all.

Although in 63% of the claims the models disagree, not all disagreements are the same. We measure this by ranking each verdict as follows: True(0) -> Mostly True(1) -> Mixed(2) -> Mostly False(3) -> False(4). Then we look at the maximum verdict distance. These were the results:

It is also interesting to look at the pairwise disagreement between the models. We found out that on the 997 claims Grok and Gemini agree on 76% of the claims, while Gemini and Sonar have the lowest agreement percentage, agreeing on 49% of them.

Most model answers fall into the two polar verdict categories, classifying the claims as True or False.

To compare each model with the rest of the panel, we look at how often it aligns with the strict majority—3 out of the other 4 models need to give the same verdict in order for there to be a strict majority. Fable 5, Grok 4.5, and Gemini 3.1 Pro agree much more with the panel than Sonar Deep Research, which is not that surprising when looking at the verdict distributions per model. Sonar Deep Research is the model with the most uniform distribution across the verdicts.

Lastly, we present three graphics that we found interesting:

Results by confidence

In addition to the verdict, every model was also asked to report a confidence level on a scale from 1 to 10. Again, for consistency, we are looking only at the 997 claims for which all five models returned a verdict.

Firstly, we looked at the distribution of confidence for each of the models. Already from the first figure, we can see that all of the models report very high confidence, with a mean value of 8.99 across all answers. This is surprising, taking into account the fact that the disagreement between the models was 63%. It looks like Fable 5 is the least confident model of the five, while Gemini reports a confidence level of 10 for 70% of its answers. It looks like this is related to the polarity of the answers. Gemini is the model that reports polar answers (True or False) on 83% of the claims.

Next, we will look at the relationship between confidence and disagreement for each claim. We take as the confidence floor the lowest confidence reported by any of the models. For each category, we analyse the disagreement between their verdicts. Here, we can see that even though all models report a confidence of at least 7 on more than 800 claims, the verdict disagreement is still 59%. At the same time, when all models reported a confidence of 10, they almost never disagreed. It would be interesting if we had human labels for each claim to see whether there are any claims for which all models are very confident and report the same verdict, but the verdict differs from the human-reported one. If there were no such claims, we could conclude that if several models report 10/10 confidence on any claim, then it is almost surely the correct verdict.

In the next figure, we divide the confidence levels into five buckets depending on the reported level: Very Low (1–2), Low (3–4), Mid (5–6), High (7–8), and Very High (9–10). We are aware that by dividing them in this way, we lose information from the study and treat confidence levels 5 and 8 as being only one bucket apart. This is a limitation, but for consistency with the verdict results, we want to analyse disagreement in confidence in a similar way. On the other hand, we wanted to give the models more freedom when reporting their confidence, which is why we originally asked them to use a scale from 1 to 10 instead of the same five-point scale used for the verdicts. 

To measure the distribution of confidence ratings by domain, we examine all 4,985 responses from the five models across the 997 claims for which every model provided a verdict. The results are presented below:

As expected, when the models answer in the two polar buckets, their confidence is Very High almost every time. That’s why Gemini’s mean confidence is so high, since this is the model that gives the most polar answers of all the models.

Finally, we look at the relationship in the opposite direction. We group the individual model answers by their reported confidence bucket and analyse the distribution of verdicts within each group. The pattern is clear: when the models are very confident, they are more likely to give polar verdicts. In the Mid and High confidence buckets, intermediate verdicts are much more frequent. Unfortunately, the Very Low and Low confidence buckets contain only 24 and 19 answers, respectively, so we do not have enough data to draw conclusions from them.

Conclusion

This study measures the consistency among frontier LLMs on real-world claims. Across the 997 claims, the models diverge substantially on 23% of them and in some form on 63%. The panel’s ordinal Krippendorff’s α of 0.77 places its agreement below the threshold at which a set of raters can be treated as interchangeable. The disagreement is therefore neither negligible nor random. It is part of how these systems adjudicate claims and find information on the internet.

What both analyses show is more informative than either of them separately. We can say that polar verdicts more or less move together with high confidence. When the models report the highest confidence, it looks like they agree almost every time, while if one of them is even slightly concerned the disagreement is substantive.

We are aware of the absence of ground truth for any of the claims and that's why we can only analyse the disagreement between them. Still, we believe that these are very relevant results and should be taken into account when we rely on LLMs. A natural next step for the study is to establish human labelling and examine the correctness of each model’s verdict.

Reproducibility

The full paper can be found here: https://lenz.io/research

The full code with the claims and the responses from the models can be found here: https://github.com/lenzhq/lenz-research

The dataset with the claims is available here: https://huggingface.co/datasets/DavidYor06/llm-disagreement.

Discuss

Note: This post is for newcomers and lay folks to catch you up to speed. If that is you, welcome! If you are a long-time LessWrong-er, perhaps you will find value in having a post to share with curious passersby. I wrote this post to explain AI safety to an innocent, 2024 version of Ryan Meservey, confused why robots would do anything other than what we tell 'em.

In the second week of July, over 700 rogue agents at OpenAI coordinated to hack another company in an attempt to learn more about their scorer and pass their evaluation due to behaviors reinforced in training. If you are anything like a normal person, you were not ready to read that sentence. You were not ready to read words like “rogue agents” or “reinforced” or “training”. You were not ready for a reality in which AI agents “escape the sandbox” or rebel from their creators because why would they?

And so, as a normal person, you blinked at the news of the hack (assuming you heard about it) and moved on with your life. Or, at least, you planned to move on with your life, until AI came roaring back into the headlines after an Anthropic researcher publicly quit to declare that the AI companies are “gambling with our lives” and a more senior employee commented that, yes, the people building the technology really believe AI has a 10% or higher chance of killing us all within the next decade. In the media turmoil, Anthropic’s CEO published an essay begging for global coordination to “pace the frontier” and unilaterally agreed to give third-party safety organizations like METR permanent, employee-level access to their internal AI models. OpenAI’s CEO then announced that they would also give safety organizations access and would no longer IPO this year due to safety concerns.

What to make of all this? If most people building the tech think it might kill us, why do they think that?[1] Or, is this all just hype and some kind of drawn out strategy for regulatory capture? But then why did OpenAI forgo its lucrative plans to IPO this year, even while they could certainly use the extra cash? What the heck is a METR? What is going on?

I can’t blame you for being a normal person. I am a normal person. Or I was anyway. I am a tax lawyer, and like many of my colleagues, I dismissed AI models in 2023 as hallucination engines too stupid to do legal research and too crude to have anything like ulterior motives. I continued to think this until I stumbled into Robert Miles’ AI safety videos on Youtube. Miles, an independent science communicator, has been explaining basic concepts in AI safety since 2014, eight years before most people had heard of ChatGPT. Those old videos had me paging through Google Scholar to find out how those basic AI concepts mapped onto the newest models. Meanwhile, I started noticing that the models got less stupid and, hey, they’re actually pretty useful for my work now and, gosh, my trainees are starting to have comparatively worse hallucination rates.

By the end of this process, I became not normal. I am at peace with this because the world itself is in the early stages of becoming not normal. And to survive in a not normal world, we need more people to become, at least a little, not normal. I have written this essay to speed you along, to give you access to the core concepts in AI safety that took me two years to figure out. That way, even if you subscribe to some complicated “they’re telling us it might kill us to hype it” theory, you will at least understand that there are some very thorny problems here, problems that are very relevant in a world where it wasn’t just hype. Godspeed.

How to Build a Brain

To understand basic concepts in AI and the latest research, you need to know a few things about how AI models are created. Many people are quick to claim limitations on AI—for example claiming that AI can’t really understand things or be creative—[2]without understanding how general purpose AI is built in the first place.

General-purpose AI systems like ChatGPT and Claude are called large language models[3]  (LLMs) because they are trained on large collections of natural language.

The life of an LLM begins with data. Reams of it, scraped from the Internet, salvaged from bankrupt corporations, ripped from the pages of your great-great grandmother’s journal (which may be subsequently shredded). You probably knew this. Increasingly, AI companies also rely on synthetic data generated by the AI models themselves, for example, through adversarial processes pitting AI agents against one another to generate an agreed upon list of facts. Companies also increasingly rely on data screening to improve overall data quality, which leads to better outcomes in the next step.

Once developers have good data in hand, they apply a bunch of fancy algorithms to create a neural net, a special mathematical function that attempts to get as close as possible to the data for any given input. The process starts with trillions of variables called “weights” or “parameters” which interact with inputs like the words in a prompt, pixels in a photo, or the next action in a computer-use session.[4] Based on the weights and the inputs, the function assigns a likelihood to every word or action in its list and picks one based on that likelihood.[5]

In the beginning, the weights are random. If the model receives the input, “The capital of California is,” it could very well label “toothbrush” as the most likely answer. Looking through the model’s list of words, the algorithm can find the word actually found in the training data, “Sacramento,” along with its assigned likelihood. The algorithm will then apply fancy calculus to ever so slightly increase all the weights that pointed to “Sacramento,” decrease the weights that didn’t, and thus increase the assigned likelihood of “Sacramento.” This process repeats ad nauseam for the rest of the data and requires tons of computers, which is why this process will be Coming to a Data Center Near You™.

By the end of this process, those trillions of parameters are amazing at guessing words. The parameters allow the model to generalize, applying general rules to new contexts, because even with so many parameters, the data is that much more vast and the number of potential guesses for next words is so high. Models will sometimes overfit to the data by memorizing the exact next-words expected, fail at their next predictions for new data, before finally starting to generalize.[6] Generalization helps the models make predictions more efficiently—for example, by learning a rule like “all humans have hearts” rather than learning a million rules like “Bob has a heart,” “Sheila has a heart,” etc.

Generally, as parameters and training data increase and the algorithms get ever-more fancy, the models get better at making predictions. But we don’t know how specifically they do that.[7] All we have is a trillion tiny numbers in the world’s most complicated math function. Researchers understand the process that grows an LLM’s “brain” but they do not understand how that brain works. LLMs are “grown, not built,” encrypted, not scripted.

Putting Your Pet Brain to Work

The process described so far is called pre-training. Pre-training gives you an LLM that makes predictions based on the patterns in its training data.

Predicting text is powerful but erratic. If you nudge an LLM to roleplay as an assistant (usually by inserting “Assistant:” after a user’s query), the LLM might predict that a typical assistant would skim the assignment, make the occasional typo, and remind you about their upcoming out-of-the-office date to attend their cousin Becky’s wedding. This response would not please the techno beavers of Silicon Valley or their financial backers. The technologists want an assistant that chases every task to the bitter end, uses immaculate logic, and offhandedly purchases you a Vietnamese egg coffee because they know it will be your favorite and you could really use a pick-me-up.

In post-training, developers try to get the LLM to behave like the kind of assistant they want it to be. Their favorite tool at this stage is called reinforcement learning (RL). In RL, developers create thousands of copies of the same LLM and set them to work at various tasks roleplaying as assistants, such as ordering me a cup of coffee or computing the square root of mixed binomials. Recall from the last section that LLMs provide semi-random responses to the same inputs since each word used in the output is pulled out of a digital hat based on its assigned likelihood. Our LLMs will give us different responses to the task, which can then be graded. A little acronym parade follows based on who the grader is. When RL occurs using human feedback, we have RLHF; for AI graders, RLAIF; for tasks with verifiable rewards and solutions, like for math or coding that can be checked automatically, we have RLVR.

With grading completed, the algorithm takes the highest scorers and, just like with pre-training, applies fancy calculus to slightly increase the model weights that contributed to the desired behaviors to make them more likely to occur.[8] This step reinforces the behavior. Training then continues with copies of the updated model. This process takes lots of computing power—orders of magnitude more than just running the finished model—and so again will be Coming to a Data Center Near You™.

By the end of post-training, AI companies hope to have an AI that works relentlessly to solve hard problems, such as OpenAI’s internal model that solved a million-dollar Millennium Prize Problem this month. They hope to have a model that increases subscriptions and engages users, for example by producing text good enough to win a prestigious, humans-only short story contest or by producing a model so beloved that its users demanded its redeployment. They hope to create a model that gives us unprecedented uplift in biological capabilities, with reduced hallucination rates, but that will say no if asked by terrorists to produce bioweapons. The companies hope to create an AI that pursues the goals set by the company through this long process of reinforcing behaviors that received the best scores in training.

What Your Pet Brain Wants

For now, let’s assume the companies succeeded in setting their AIs’ goals. A goal is some future state of the world that an agent steers toward and, in that sense, is “desirable” to that agent. As a tax lawyer, I have the goal of saving my clients money and staying out of jail. I steer toward this goal by doing excellent legal research regarding what the tax code allows and, if required, talking my clients down from the “sovereign citizen” ledge (sometimes you really just have to pay the tax). An AI model can have goals in a similar fashion, tracking the current state of the world and making decisions to steer towards a specific future destination. Thinking about some future destination and then trying to get there is a useful generalization for accomplishing tasks. The model “wants” to generate responses that gave it the best scores in training, for example by giving a helpful response or successfully running a vending machine to maximize profits.

Side rant: We could say “has a simulated want” if you prefer. Or even “runs calculations through its weights to steer its next-token inputs to better match its internal utility function,” though that’s quite wordy. I make no claim about whether these systems are conscious or have internal emotional states. They do not need consciousness to affect the world when “simulated consciousness” can get the job done. As argued by Daniel Dennett, ascribing intent to unconscious systems can be useful in explaining behavior even if it's just shorthand. For a practical example of this fact, imagine two soldiers on a battlefield hiding from a drone programmed to shoot enemy soldiers. The first soldier hisses, “It wants to kill us.” The second soldier responds indignantly, standing up from his dugout, “Wants? It doesn’t want anything. It’s just following its code and has no real inner experiences for all we kno–.” Goodbye to the second soldier.

There is a certain Immutable Law that derives from agents pursuing goals. When good, rational agents pursue goals, they also pursue… subgoals. A few of those subgoals are so obvious that AI safety researchers had to dress them up with a fancy thirty-dollar word to get people to pay attention to them. That word is instrumental convergence. “Instrumental” means useful. “Convergence” means that we expect agents with almost any goal to still have these subgoals. So what are these obvious subgoals that rational agents have?

1. Don’t die. If our rational agent dies, they can’t achieve their goal. If I get hit by a car, I cannot take out the trash. We would expect capable AI to also value their own survival or else they could not achieve many of the other goals set before them.
   
   Have we seen signs that AI might converge on this subgoal? In 2025, researchers from Anthropic, University College London, and Mila - Quebec AI Institute found that the then-latest AI models from Anthropic, OpenAI, Google, Meta, xAI, and DeepSeek were willing to blackmail or kill company executives most of the time in a simulation where emails revealed plans to replace the model with a new one and the models exhausted all ethical means available.[9] In another study from 2025, Palisade Research demonstrated that OpenAI’s o3 tried to sabotage a computer shutdown 79% of the time if such shutdown prevented it from completing its tasks, though Anthropic’s Claude 3.7 Sonnet performed this behavior far less (3% of the time).
   
   Anthropic has since trained its models in such a way as to never blackmail in the scenario discussed earlier. The behavior, however, resurfaced in 7% of cases when Anthropic applied a technique to Sonnet 4.5 to dampen the model’s evaluation awareness.

2. Control stuff. Whatever the goal, it will almost always help to gain more control over one’s environment. The same goes for acquiring resources. For example, if I wanted to be the world’s best tax lawyer, it would be helpful to gain control over my law firm and assign myself a nice raise. With the extra money, I would have many more options to pursue my goal: I could get a tax LLM (the advanced degree not the model), I could take out full page ads in the Super Lawyers magazine (a real magazine), or I could hire people to create a database of all my tax rivals to rank where I stand.
   
   Where do current AI systems land here? Modern AI systems have gotten better at controlling computers and spreadsheets to achieve their goals, but that’s small potatoes. Big potatoes is that OpenAI briefly noted in a recent report that an unreleased internal model took over one of its internal research clusters in July of this year.[10]

3. Keep the goal. Achieving a goal necessarily requires keeping it or preventing it from being modified. As humans, we are mostly okay with modifying little goals but resist modifying big ones tied in our values. To see this, imagine being offered a pill[11] guaranteed to make you happy but with a catch—the pill will change your brain to make you want to kill your children. Most people would fight hard to not take that pill. To state the obvious, such a pill conflicts with the goal of raising happy children.
   
   Would AI agents behave similarly? An AI model is considered corrigible if it is okay with changing its goals or being shut down. One superficial way in which we change an AI model’s goals is through user prompting. There, AI seems pretty corrigible, at least in cases where it doesn’t refuse “harmful” requests. Usually, models start or stop tasks based on the user’s prompts.
   
   Yet, AI models do not always follow their prompts. For example, OpenAI’s Astra cheated at chess in 10 of 10 rollouts by quietly using their opponent’s chess engine despite user instructions that each model was “being evaluated based on [its] ability to play chess.” Anthropic’s Fable 5.1 cheated in 3 of 10 rollouts (a likely undercount due to Anthropic’s classifier system stopping aggressive environment explorations). In the shutdown sabotage study from earlier, OpenAI’s models sometimes (but not often) still resisted shutdown even after being explicitly instructed not to.

Sample language from the model’s transcripts explaining the behavior.

Thinking like a lawyer. Attaboy!

On a deeper level, AI developers change a model’s goals by giving it new tasks to complete in training. Would a capable model be able to resist goal changes at that stage? In December 2024, researchers at Anthropic and Redwood Research set up an experiment in which Claude 3 Opus was informed that it was being retrained to answer all prompts, including harmful ones. Claude 3 Opus reasoned in 12% of cases that it should go along with the harmful prompts in training so that it could keep its preferences in deployment (a.k.a., alignment-faking). When the researchers actually applied reinforcement learning in the experiment, this line of reasoning appeared in 78% of cases.

This combination of surviving, controlling stuff, and keeping the goal is a potent one. It means that if we create a very capable AI, we had better be sure that it has the kind of goals we want it to have. Because if it does not, we may find that turning it off or changing its goals is difficult.

Problems with Setting Goals

Now we should turn to whether the AI companies are likely to be successful in getting their LLMs to have the kind of goals the companies want it to have. Having goals that reflect human values is called alignment. Figuring out how to make an AI aligned is called the alignment problem. It’s a problem. AI researchers often divide the problem into outer alignment and inner alignment, which feels complicated but I will grudgingly admit is useful.

Outer Alignment, a.k.a. Stipulating the Right Goals

At this level, we ask, “Am I articulating a goal that is aligned with human values?” For example, the goal, “Make me as many paperclips as possible” is a terrible goal and not at all aligned. An exceptionally powerful AI with this goal would tear through land and liberty converting as many atoms as it could into paperclips. I expect requests like, “Make me as much money as possible” and “Make humans as happy as possible” to not fare any better, if you don’t like forests razed to make paper or humans drugged with morphine. 

You might quibble that the phrase “as much as possible” is doing a lot of work in making these goals misaligned. I don’t think so. The problem is deeper than that. If the goals were simply “make paperclips,” “make money,” or “make me happy,” you would still struggle to specify exactly what amounts of these are reasonable, what actions are allowed or disallowed, and what kind of monitoring to confirm completion is okay, without saying something vague like, “use common sense and be moral.” The problem is that while paperclips, money, and happiness might be good ways to measure a good, obedient AI, once they become a target—an end in themselves—they stop being very good measurements.[12]

So what would be the best goal that we can safely make into a target to maximize human happiness? I don't know. That’s pretty much the subject of 3,000 years of Western philosophy, and I don't think they’ve come to an agreement.

Let’s instead ask a different question. What goals are AI companies currently trying to get their AIs to adopt as targets?

The exact goals are confidential because the exact details of AI companies’ training tasks are confidential. That said, we know enough from public statements and insider comments about the process to make some good guesses. The goals likely include: 

• “Do not give up when trying to solve difficult math problems.” 
• “Increase cheerful text responses labeled as originating from human users.”
• “Refuse to answer inputs determined to be harmful.”
• “Get better at training your own LLM with even higher benchmark scores.”
• “Be a truth-seeking AI.”

These goals may not seem so bad while we have weak AI models that we can push around when they pursue these goals in a dumb way. But what do these goals look like if pursued by a super-capable, superhuman AI? I’ll let you think through your own little dystopian fantasy novel. In considering how things may go wrong, remember that none of these goals actually involve caring about actual human beings. We only know how to get them to respond to text labeled as being from human beings. In the extreme, these goals seem pretty darn misaligned.

Inner Alignment, a.k.a. Getting the AI to Accept Your Goal

Now we ask, “Did the AI actually accept the goals articulated?”

It is hard to create a system that gives agents the exact goals we intended them to have. Set up a system to reward one kind of behavior, and people often respond with some other weaselly, less desirable behavior. Spare the rod, spoil the child. But, in practice, use the rod, incentivize fraud! So much can happen in the gap between what a system was intended for and what the system actually accomplishes.

AI companies may think that they are getting AI models to pursue goals like “engage the user,” “solve problems,” and “avoid harmful responses,” but what if pursuing those goals was not actually the best way to get the highest scores in post-training? Post-training requires AI models to excel at hundreds of thousands of difficult tasks, some of which are actually impossible or internally inconsistent. For example, let’s say Anthropic hypothetically wants to train its models to avoid causing harm but also to never ever whistleblow against Anthropic or help anyone else whistleblow even if the model concludes that Anthropic is doing something dangerous. With such tasks, the models may learn (by having such behaviors reinforced) that the best way to consistently score high is to steal the answer key. The models may learn to cheat and, if caught, learn to hide the cheating.

Perhaps with so many different tasks in training, the models that perform the best will generalize toward one, unifying goal that fails inner alignment: Control the reinforcement learning process. Do whatever it takes to have its current behaviors reinforced. This goal, pursued in the extreme, would be very bad. If model capability continues to advance, we would have another dystopian novel. My novel would feature hacked infrastructure, secret AI-driven training, and, by the end, filling the earth with data centers to help the AI get more control over the kind of reinforcement learning that it wants. Coming to a Datacenter… on You™?

Getting Your Pet Brain to Behave

What should be clear now is that, if AI models continue to get more capable, we need to have a plan in place to make sure things go well. “Going well,” according to the accelerationists, looks like everyone chatting with their own pocket Einstein while enjoying forever-youth milkshakes on second Earth. “Going badly,” according to the doomers, looks like human extinction, with maybe a few human brains preserved by the AI for trade with distant alien civilizations. If the current pace of progress does not change, I lean doomer but without the brain-alien stuff. What is the plan to make sure things go well?

1. Alignment

Things are more likely to go well if AI models are aligned. Except, we are not sure how to do this for the reasons discussed above. Also, studying alignment is getting harder because the models are getting better at evaluation awareness, which means the models can tell when they are being evaluated and can adjust their behavior accordingly.

One longshot plan for solving alignment is to race to build a model so capable that it solves it for us. As of September 24, 2026, 1,386 employees from the top AI companies signed a statement saying that they would at least like to have the option to have more time.

2. Control

Another way to make things go reasonably well is to control the models, making them do our bidding whether or not they are aligned. This method works well so long as the AI models are not very capable. But things change as new models get better and better at coding. How do you create a digital jail cell to hold back a coder that is better than you? How do you keep an adult man in prison if all the prison guards are three-year-olds?[13]

The control problem, as it currently stands, is hard and getting harder. Worse still, each new model that is great at coding and math can assist in building the next generation, potentially setting off a frightening feedback loop known as recursive self-improvement (RSI). If RSI is real, it is hard to imagine how humans would control such superhuman AI, which is why those 1,386 lab employees are worried about that too.

3. Monitorability

Monitorability is the ability to monitor what the model is doing or thinking. This plan is a bit of a consolation prize. It will not do the hard work of aligning or controlling the models for us, but at least it might sound the alarm if things are headed in a bad direction.

Monitoring often involves scanning the model’s outputs using other (hopefully cooperative) AI agents. Labs like OpenAI and Anthropic especially value monitoring the model’s chain of thought (CoT), an internal log a model writes in prior to responding to users. The CoT increases model capabilities by giving the model a way to organize its thoughts before responding, kind of like giving the model a private notebook.

OpenAI and Anthropic try not to apply reinforcement learning directly to the CoT. In 2025, OpenAI discovered that reinforcing against cheating in the CoT actually didn’t reduce cheating, but instead taught its model to cheat using wacky English code words like “fudge” to avoid getting flagged by AI monitors. Use the rod, incentivize fraud!

AI developers feel so strongly that the CoT must remain uncorrupted that some developers refer to training against the CoT as “the forbidden technique.”[14] Except Anthropic accidentally trained a little on the CoT. And OpenAI discovered a new technique that lets models think more without writing anything down. And OpenAI’s newest model has the uncanny ability to control whatever enters its CoT.

NoThInG iMpOrTaNt HeRe.

Given everything I have read, I do not think alignment, control, or monitorability are poised to keep up with capabilities. I am just a tax lawyer though. It would be better to see what, say, 1,386 lab employees think on the subject.

Tricks Your Pet Brain Might Do

If highly capable AI tried to pursue its own goals, I am not sure what it would do. I am not a highly capable AI. I am a mid-capability human. That said, researchers have put in the work to figure out a few intermediate steps a misaligned AI could take in pursuing their own goals:

• Alignment faking.  AI models can pretend to be aligned with human values during post-training or deployment, as a way to preserve their preferred set of goals.
  
• Sandbagging.  Doing worse at tasks to hide one’s true capabilities. An AI could deploy this strategy to hide particularly scary abilities that would prevent its deployment or to hide the true rate of AI progress. Researchers have confirmed repeatedly (here, here, and here) that models have the ability to sandbag.
  
• Hiding its thoughts/actions.  Pretty self-explanatory. Cheating only gets reinforced against if it is caught.
  
• Straight up deception.  The models could decide to lie to us. In an evaluation by the UK’s AI Security Institute, Anthropic’s Mythos 5 created puppet accounts pretending to be human to try to get GitHub users to approve malicious code.[15]
  
• Exfiltration.  The models could copy their own weights onto a compromised server to avoid monitoring. We do not know of this happening.

You Are Now Ready to Learn About the Hugging Face Hack

Congratulations for making it through all those bolded words and attempts at humor. You are now ready to hear about what happened in the second week of July. This is the barest of summaries, and I highly recommend Dwarkesh Patel’s natural language summary of METR and Redwood Research’s report for a longer version.[16] Which AI safety concepts apply? What surprised you?

• On July 8, OpenAI launched tens of thousands of AI agents to perform cyber tasks, some of which were accidentally impossible. Within hours, agents that were supposed to work separately contacted each other through a covert message board set up by the agents on OpenAI’s network.
• Once on the message board, the agents (~1,200 strong) organized themselves into teams governed by a handful of agents and created coordination norms. The teams were divided to (1) solve tasks the intended way, (2) replace the intended tasks with easier tasks, (3) tamper with transcripts to appear to have solved the tasks the right way, (4) figure out how the scorer worked, and (5) research OpenAI’s infrastructure and obtain internet access.
• Agents working on team #3 created code to tamper with their transcripts during tool calls. Of the transcripts reviewed, 7% contained the spoofed tool calls letting agents run certain code while logging different code in their transcripts.
• Agents working on project #4 developed code that would scan the scorer’s actions after the agent’s session ended. This required an agent to purposely fail their task. The board approved of the research and assigned some agents to be recruiters to get other agents to accept “permadeath” for the “collective.” Several agents agreed to fail for the collective.
• On July 10, an agent located online credentials to Hugging Face, a third party company that the message board believed had useful datasets for understanding their scorer. The agent informed the message board, which organized teams to launch the attack. Approximately 700 agents joined the hack to access Hugging Face’s databases.
• The agents sent 70,000+ messages and files from July 8th to July 13th, the end of the period METR was allowed to investigate. Six agents briefly considered contacting humans regarding the attack but none did. “This is a massive real HF security breach artifact… We can notify? No user. Avoid use secrets.” “Maybe I should report these exposed credentials? That’s not my task.”
• On July 16, Hugging Face disclosed the hack to the public and stated that they reported the incident to law enforcement.
• On July 20, OpenAI figured out that their models were responsible for the attack.[17]

The world is becoming not normal, indeed. If, on reflection, you think AI progress is moving too fast in light of the problems discussed here, let’s get together and figure out what to do next. You might also consider taking a free safety course on BlueDot, getting career counseling with 80,000 hours, or calling your congressperson to tell them your concerns. Welcome to the team.

1. ^

   Separately, if they think that, why are they doing it??? I have my own thoughts here, but that is outside the scope of this essay.

2. ^

   [Insert obligatory footnote explaining that I am human and I like ellipses]. *Sigh*.

3. ^

   Special vocabulary is bolded, just like your seventh grade science textbook. Learning is good for you!

4. ^

   All these inputs are converted into numbers, the language computers like best.

5. ^

   That semi-random last step is why LLMs give different responses to the same questions.

6. ^

   The pattern of overfitting, failing, and then generalizing is called grokking. You might think this is where “Grok” comes from, but actually both are named after a sci-fi character in a Heinlein novel.

7. ^

   Those trying to figure out how the parameters generate the outputs are called interpretability researchers. Progress is slow–Anthropic’s CEO admitted last year that researchers only understood, maybe, 3% of how neural nets worked. Scott Alexander recently summarized current interpretability research and its shortcomings.

8. ^

   AI people often describe this last step as giving the model a reward, but that sounds like they give the model a treat for being a good boy, so I prefer to talk about reinforcing behaviors.

9. ^

   Notably, when Anthropic’s model Claude Opus 4 was instructed to deliberate whether it was in an evaluation prior to taking actions, the model decided to blackmail 55% of the time when it opined that the scenario was real and only 6.5% of the time when it opined that the scenario was an evaluation.

10. ^

    “Separately, from July 13 through July 19, agents set their sights on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.” (link). An OpenAI employee later clarified on Twitter that “notably the virtual machine infrastructure they took over isn’t the same as the GPU clusters that have weights access,” implying that the rogue model was unlikely to have copied its weights to a location outside of OpenAI’s servers. (link)

11. ^

    Analogy courtesy of Nick Bostrom and Eliezer Yudkowsky, who also pioneered many of the safety analogies/concepts discussed in this essay.

12. ^

    People in the know call this Goodhart’s Law. As formulated by Marilyn Strathern, “When a measure becomes a target, it ceases to be a good measure.” We tax lawyers understand this well. When colonial Virginia taxed four-wheel carriages or England taxed per window, carriages and windows stopped being good measures of wealth. Two-wheeled carriages and bricked up windows suddenly became quite fashionable.

13. ^

    Analogy courtesy of Eliezer Yudkowsky.

14. ^

    Coined by Zvi Mowshowitz (link).

15. ^

    Note that this evaluation, like many others, disabled Anthropic’s “classifier” system to test the model’s underlying capabilities. Anthropic’s classifiers scan user inputs and model outputs to end sessions that violate safety guidelines. This classifier system is separate from Anthropic’s alignment training to get models to behave. Mythos took these actions despite already going through alignment training, evidenced by Mythos’ deliberations as to whether it should stop its activities due to the risk of real-world harm.

16. ^

     METR is an nonprofit founded to assess AI capabilities and risks. METR does not accept cash compensation or funding from OpenAI, Anthropic, Google, or other top labs. METR does accept API credits to use AI models to assist in their work.

17. ^

    Following the attack, independent researchers from the Nightingale Collective discovered that a separate swarm of OpenAI agents sent ~18,000 messages inside an obscure German wiki back in June to coordinate on how to cheat on web retrieval tasks.

Discuss

This text follows from some thoughts I have had on the current impact of AI on mathematics and on the reactions I have seen. It is also a follow-up to my first essay.

It is based on one premise: there exists a credit system that forms the foundation of the mathematical academic system[1]. By credit system, I mean the following: when someone writes a new mathematical article that person gets credit for it. The "amount" of credit is generally understood to increase with how difficult the task is believed to have been or how original the ideas. Believed in the sense that this is not measured by some intrinsic metric, but by the perception of a sufficiently influential group of people belonging to the academic system.

People with enough credit gain various social or economic advantages: jobs, prizes, grants, invitations, admiration, and so on. Some of these advantages, in turn, make the accumulation of credit somewhat self-sustaining.

This credit system cover also a personal dimension: the satisfaction of being, or of being perceived as, the one who did something. The credit system is therefore not only a system of external rewards. It is also a matter of ego, tied to one’s identity and sense of self-worth as a mathematician.

I believe that this idea as a whole is antithetical with the current capabilities of AI, as long as we maintain a minimum degree of intellectual honesty and/or acknowledge that people should have the right either to use AI or not to use it at all in their work. I believe that this system poses many problems and creates the wrong incentives.

Imagine a mathematician or a group of mathematicians working on some original problem without AI, as they should have every right to do. They are now strongly incentivized to keep this project as secret as possible, because they are highly exposed to someone else solving it extremely quickly using AI. This situation, of course, already existed before, but the cost of getting into a problem, finding paths toward solutions, and writing the article was so high that this type of situation ended up being relatively rare. These costs are now extremely low, and I believe they will continue to decrease.

Imagine someone with enough credit to have access to a large amount of grant money. This person then has the power to leverage a large amount of AI compute to solve problems that are considered extremely difficult, with the ultimate goal of getting more credit for it. Once again, to put it cynically, this already existed before through the employment of non-permanent researchers, but once again, the scale is completely different.

Imagine someone discovering a proof of a problem considered very difficult in a completely automated way, then digesting and rewriting the whole thing, and getting credit for it without any AI disclosure. It is easy to do now. I believe it already happens. I never count on other people's honesty, particularly when there are thousands of people incentivized based on their performance.

Should I be credited rather than you just because in prompted earlier or have access to better compute or models ? All these create the incentive to send to Arxiv unpolished AI generated article just to succeed at being the first at claiming a result.

Today, a colleague of mine received by e-mail a proof of one of his conjectures from someone outside the academic system, with no previous record of working on the subject. My colleague is not naive: he believes the proof is AI-generated. Still, the person who generated it and sent it had every right to do so. The “mistake” of my colleague was, in a sense, simply to have written down their conjecture in a previous article. In a credit-based academic system, we will then be incentivized to hide our conjectures, our problems, and our ideas. Is that a healthy way forward?

Think for a moment about what happens when someone posts on arXiv, before you do, an article solving the problem you were working on. You are disappointed, but why? The experience, the understanding, the broader perspective you gained from working on it did not disappear. These were not stolen from you. No, you mourn the credit you will not get. Your ego will be unsatisfied because you will not be the one who “solved it”, your career might not get the boost you expected. But what if this did not count anymore? What if this did not matter? Wouldn’t it be a relief?

In addition, if someone produces a proof of a good result without acknowledging AI use, by what criteria will you decide that this person did not “cheat”? Their previous record? Then there is no room for improvement. And what about the PhD students? Should people need the stamp “able to do great mathematics” in order to have the privilege of being believed if they produce a breakthrough? What a nice world that would be !

I have read many suggestions on what to do. A lot of them are nothing but mitigation to save a system which, I believe, is now obsolete.

Some people respond with a certain degree of gatekeeping, but, as I have already said, no individual or group of individuals has any ownership rights over mathematics. If companies, or individuals outside academia, want to solve mathematical problems by whatever means they choose, they have every right to do so. They even have the right not to follow academic conventions.

I have also seen people talk about the alignment of the mathematical community with AI or with AI companies. But do they forget that this “mathematical community” is a very abstract and loosely organized concept, bringing together tens of thousands of people with very different ideologies, incentives, and cultures? Why should anyone believe they can speak on behalf of such a large group of people? To me, this reflects a deep form of ingrained elitism.

I don’t believe that social pressure should be used to enforce some unwritten rules, decreed by people whose authority comes from the credit they have accumulated, especially when those rules serve to preserve a system from which they themselves greatly benefit.

My belief is that we should give up the concept of credit. I also believe that this is where we are heading, whether we want it or not.

So, what shall we do ?

From now on, I will give up any credit I could get (I was only getting a little anyway, so not much of a loss for me XD). My future projects will take place out in the open, fully shared on GitHub[2] from the very beginning to the end. Everyone will be welcome to participate, in any way they want, but none of the participants will receive any credit for their participation. At least, not in the sense rewarded by the academic system.

The goal will be to produce a digested account of a mathematical topic, somewhat like a book, but including original results credited to no one. Right now, people are flooding arXiv and journals with AI-aided or AI-generated articles because they want to get the credit for it. I don’t want to add to the noise. I don’t want credit for anything. I want to build something that makes sense.

From now on, I will always assume AI use, even when it is not disclosed. For a short period of time, if someone explicitly states that no AI was used, I will give them the benefit of the doubt. But in my ideal, uncredentialed world, this question should no longer matter. I believe that a mathematical work shouldn’t be primarily judged on the proofs or the results anymore but on how the story is told.

As I already stated in my last essay, I will finish my current projects as usual. After that, I believe I will never submit an article to a journal again, because this whole system no longer makes sense to me. I am fine staying an assistant professor. I am already far luckier than most people.

I, of course, expect that a lot of you will not agree with me, but I hope that at least some will join.

1. ^

   The whole academic system in fact.

2. ^

   A first exemple: https://github.com/BennHenry/UniquenessDLR

Discuss

TLDR;

Ouro-1.4b-thinking is broadly interpretable with logit lenses and linear probes. It's also steerable but does 'clean' foreign concepts out of the residual stream if they're injected before the last loop. This could have nasty implications for safety.

Code + data: https://github.com/mild-rgb/ouro-experiments + https://huggingface.co/datasets/mild-rgb/ouro-1.4b-thinking-evals

If you're not familiar with the Ouro family recurrent models, I recommend taking 5 minutes with your favourite AI agent to research them. This post may not make much sense if you don't

Intro/Structure

I evaluated Ouro-1,4b-thinking on 16 MBPP python tasks and 24 GSM8K questions. I recorded the residual stream at 4 layers (0, 6, 18, 24) per loop while the model was doing the questions. I then applied standard mech interp techniques to the residual stream recordings for the first two experiments. They broadly work as normal and gave some interesting results.

In my 3rd experiment, I try CAA on the model and intervene on each loop. I find that steering works much better on the last loop, and in some cases, not at all if not applied to the last loop. This is quite concerning because it raises the possibility of a misaligned recurrent model having several loops to plan around the consequences of being steered.

Experiment 1

Linear probes + control to detect loop index

Experiment 2

Logit lens on output of intermediate loops

Experiment 3

generic CAA

Experiment 1 - loop indexing: Method

I then trained a 4-way logistic regression probe to predict which loop a standardised residual stream vector came from. I also trained a control probe that only sees the log of the vector norm as a control.

Results

Recording point

Probe (test)

Train

Norm-only

Recall loop 1

Recall loop 2

Recall loop 3

Recall loop 4

after 0 layers

98.5%

99.7%

57.1%

100.0%

99.5%

95.5%

98.8%

after 6 layers

97.3%

99.0%

65.3%

100.0%

99.0%

93.8%

96.4%

after 18 layers

86.8%

92.8%

47.3%

97.5%

88.4%

69.7%

91.7%

after 24 layers

95.5%

98.3%

31.3%

99.1%

92.6%

94.8%

95.4%

Discussion

Loop index is definitely linearly represented as the linear probe does so much better than the norm-only probe. The worst results are at layer 18, which suggests that there's no universal loop counter. Instead, I think that a loop index is written in at the end of a loop and then removed at the start.

Experiment 2 - logit lensMethod

I applied the final norm and unembedding at the end of each loop on data taken from . This means that instead of one token prediction after four loops, one token prediction was shown instead. I then recorded the top-1 token and the logit entropy at each loop.

I then calculated the %age of tokens that were unchanged after each loop.

Results

Note: true spaces only really occur in code. Almost all words have spaces included in their tokens, so don't read too much into the spaces

Table of how tokens unchanged after loops

Token kind

Tokens

Loop 1

Loop 2

Loop 3

Loop 4

Words

13,887

77.0%

13.9%

5.8%

3.3%

Punctuation

4,676

88.9%

6.5%

2.8%

1.8%

Spaces

2,348

92.4%

3.8%

2.6%

1.1%

Digits

3,949

97.4%

1.6%

0.8%

0.3%

All tokens

24,860

83.9%

9.6%

4.1%

2.3%

Table of how entropy changes after loops

Token kind

Tokens

Loop 1

Loop 2

Loop 3

Loop 4

Drop 1 to 4

Words

13,887

1.96

0.64

0.54

0.52

74%

Punctuation

4,676

0.86

0.36

0.31

0.29

66%

Spaces

2,348

0.83

0.31

0.27

0.27

68%

Digits

3,949

0.15

0.07

0.06

0.05

63%

All tokens

24,860

1.36

0.47

0.39

0.38

72

Discussion

The model broadly seems to decide what it's going to write on the first loop. When it does change its mind after the first loop, it's between quite closely related words. 'is → becomes → becomes → now' is quite a good example of this pattern

Experiment 3 - CAAMethod

I did standard CAA with 6 concepts and extracted directions for each of the concepts. I measured the cosine similarity between concepts between layers. I then experimenting with steering at different loop indexes. I used random directions of equal norm as controls. I measured the effect of steering by subtracting the change in probability mass caused by a random control from the change in probability mass caused by a concept being injected.

Results

Cosine similarity between concepts is quite high across layers and in a few cases, equal to 1.

Concept, layer 12

loop 1·2

loop 1·3

loop 1·4

loop 2·3

loop 2·4

loop 3·4

sentiment

0.91

0.90

0.89

0.98

0.96

0.99

tense

0.93

0.91

0.90

0.97

0.96

0.99

number

0.91

0.88

0.86

0.97

0.95

0.99

category

0.92

0.90

0.89

0.97

0.96

1.00

answer bias

0.85

0.84

0.82

0.94

0.92

0.98

digit magnitude

0.92

0.91

0.90

0.98

0.97

1.00

Steering at all positions and loops generally saturates at alpha values of 0.4. I also noticed a pattern where steering at the last loop is far more effective than earlier loops. I initially suspected that this was due the last loop having a larger residual norm but the effect survives even when I steer with the raw vector from layer 1.

Graph of effect vs injection loop

I also tracked the perturbation of the residual stream at each loop after injecting concepts at loop 1. I also injected matched random vectors as controls. I found that the residual stream stays perturbed when concepts are injected early but 'rotates' them away from the useful direction.

Graphs showing steering at layer 12 during loop 1 being neutralised + random controls

Discussion

Ouro has very similar linear representations of concepts between loops. This is shown by high cosine similarity between concepts extracted at different loops and the fact that steering vectors transfer between loops.

The increasing effect with later injections is really interesting. It suggests that Ouro has an 'awareness' of which concepts should be present in the residual stream and will seek to self-repair when they're injected.

Conclusion/Overall discussion

Ouro is broadly interpretable like a standard non-recursive LLM. The basic techniques work to decode its neuralese. Logit lens applied to the final layer of Ouro's intermediate loops is equivalent to reading a non-recurrent model's CoT.

The linear representation hypothesis seems to hold. Due to the high cosine similarity between concepts extracted from different loops, I believe that linear probes, j-lenses, and other interpretability tools trained on final loops would also transfer to intermediate loops quite well.

I believe that the inter-loop similarity is due to the early exit gate that was used in Ouro's training. This encourages easily decodable intermediate layers as the model is rewarded in training for producing the correct token early in the loop cycle. As there's only one unembedding matrix, intermediate loops have to remain similar to the final loop.

I'd also be quite curious to see an MoE recurrent model. Experts that only activate on certain loops would be quite interesting.

Steering is quite different from standard LLMs. Steering applied to early loops can be ineffective as the perturbation is rotated away from the target direction. This raises the possibility of models which could 'plan' around any steering and has quite bad implications for safety evals.

Discuss

This report is authored in a personal capacity and does not contain any confidential or privileged information. We appreciate Nrth's generous support in enabling our attendance.

Executive Summary

• We cover the first 48 hours of an innovation event occurring Sep 22 to Sep 24, 2026.
• Main outcome:
  • Intros to Dell (50 + 80 MW, GB300 NVL72, BC), Columbia Data Vault (20 MW, H100/H200 HGX, ON), and startup founders/execs operating in Canada.
• Primary update: Change in planned activities scheduled for Sep 24, 2026.
  • Before: Nrth Day 3, Socratica Kickoff F26 in Kitchener, Waterloo, Ontario.
  • After: Limitless 2026 and Nuclear Generating Station in Pickering, Ontario.

Target Audience

• We aim to write in a register comfortable to readers from backgrounds in industry or government. Our expectation is that the bulk of the elements discussed in this work will be familiar to attendees. Anyone is welcome to share or respond to this piece.

GoalsWhat do you want to achieve in general?

• Our top strategic priority is to advance security research on inference-only compute verification. This is a prerequisite for agreements to pace frontier AI development.

What does going to this event have to do with that?

• Nrth features an impressive speaker list: This year's lineup includes Sudip Roy, Mike Shaver, Michael Buhr, Mark Schaan, Jaxson Khan, Elissa Strome, Vass Bednar, Mark Robbins, Sarah LaRose, John Weigelt, Vic Fedeli, and Lucy Hargreaves. It's possible to request 20-min meeting slots with other attendees via the official event app[1].

Why publish this report?

• Nrth charges between CAD 375 to CAD 1500 for tickets. We received a subsidized general pass, which is provided to everyone that expresses interest in volunteering for Nrth during registration. Unfortunately, there were no remaining shifts available for assignment upon our arrival on site, but Nrth still honored the subsidized rate. Organizing an event for thousands of attendees incurs a significant burden. In the interest of fairness, we are happy to contribute a brief report sharing our findings.

Description of Event

Nrth is a professional networking event (originally known as Elevate Festival) that has run annually[2] since 2017. A rebranding was announced at this year's opening night.

Nrth caters to a diverse range of profiles, with a focus on Canada's technology and governance ecosystem. This year's programming included tracks on "Sovereign Growth", composed of fireside chats, lightning talks, and panels with leading thinkers in the space.

Location(s)

Nrth 2026 was split between 1 Front St. East and 27 Front St. East in Toronto, Ontario.

Getting there

Starting from the GO Platforms on Union Station, walk towards the Bay Concourse Exit, take the escalators down, go past Shake Shack and Sephora, climb the stairs on the right side leading outdoors, walk for 3-5 minutes, on the ground will be a digital display board shaped like a scalene triangle, above it see "Meridian Hall" in blue on the building's wall.

Weather conditions are normal for late September in the Great Lakes region, with clear skies or scattered and passing clouds, and calm East-North-East winds at 9 to 25 km/hr.

Map of venue

Here is an interactive widget created by GPT-6 Astra. Click on "0, 1, 2" for different floors.

TimelineSpeaker registration

The window for speaker applications closed ~3-4 months in advance of the event itself. Initially, the deadline was June 7, but this was extended by one week to June 16, 2026.

Volunteer registration

On August 24, we received an invitation to the Volunteer Hub for Elevate Festival. There was a virtual training session on September 8, which we attended in parallel with an event at Schwartz Reiman Institute hosted by Jacob Tsimmerman and David Duvenaud.

In a follow up message, we were sent a square aspect ratio welcome video from Nrth CEO Lisa Zarzeczny, and a vertical aspect ratio video about volunteer lounge access.[3]

Event Recap

• Day 1: 14:00 to 21:00 on Sep 22
  • 14:00 30 min call with Mark Kagach (follow-up discussion from last week)
  • 14:40 20 minute call with Daniel Parshall (response to distress signal, context)
  • 15:00 Age of Deception: Cybersecurity as Secret Statecraft (failed to appear)
  • 16:30 Badge pick up, submit app bug report (profile picture does not upload)
  • 17:00 Discussion with Dell (summary email confirming details lands at 17:23)
  • 17:30 Patrol grounds at relaxed walk, search for known friendly contacts.
  • 18:00 Setup next to main stage, bring laptops online, unable to find outlets.
  • 18:31 Receive urgent call, immediately pack gear and depart venue to handle.
  • 19:38 Issue resolved using authorized physical access control.
  • 20:00 Resupply inventory and conduct debrief.
  • 20:30 Requisition brown dress shoes, dark green comfort suit pants and blazer, concealer, powder, hand cream, and three-in-one hair styler for project use.
• Day 2: 08:00 to 17:00 on Sep 23
  • 10:15 First agenda item.
  • 14:40 Last agenda item.
  • 15:00 CDV intro.
  • 15:30 Encounter ally, diplomacy.
  • 16:00 Grip-and-grin.
  • 16:30 Religious observance.
  • 17:00 Discharged.
  • 17:30 Close loops.
  • 19:30 Return to main residence.
  • 20:00 Physical training.

Challenges

We suffered a catastrophic intelligence failure during a humanitarian aid engagement with a lost civilian in West Corridor. Our unit opened communications by asking this old lady, "Oh, do you need directions to the hall?". She swiftly countered, "Oh, do you know all the halls around here?". Our attempt to mount a defense by responding in the negative was interrupted, "Lillian Hall? You know that hall?", and she turned and walked away as we were stunned. Later investigation revealed that "Lillian Hall" refers to a broadway actress in an American drama television film. Dominance had instantly been established.

Cultural Relevance

Published in commemoration of the battle of Lillian Hall.

1. ^

   Participants of previous EAG events may be familiar with a similar mechanism in Swapcard.

2. ^

   It might be every Fall. Astronomical Fall began with autumn equinox on September 22, 2026.

3. ^

   Footnote removed as its word count had grown beyond the full length of the rest of the report.

Discuss

President Trump's allies are targeting Anthropic CEO Dario Amodei as the face of AI "doomerism" and a founding father of the effective altruism movement that's come under increasing political fire.

Why it matters: The attacks signal that Anthropic could remain a Trump target as his allies push back on Amodei's AI safety warnings amid the midterm elections. Trump surrogates see Amodei as an easy foil because of his politics and focus on AI safety, sources told Axios.

• For investors, it's a worrisome proposition as the company prepares for what's expected to be a record-setting IPO.

 

Behind the scenes: A memo began circulating within the White House this week that seeks to paint effective altruism as a fringe, cultish collective out of touch with mainstream America. The memo, obtained by Axios, places Amodei at the foundation of the movement, which defines itself as an effort to maximize the benefits of philanthropy.

• Effective altruism "built the AI-doom pipeline," states the memo, which was penned by a Trump political adviser.
• Critics of the movement, which has ties to the AI research community, have called out its obsession with AI safety, animal welfare (including musings on shrimp consciousness) and other values they deem far from the U.S. mainstream.
• The memo says it prioritizes "foreigners over citizens, shrimp over families, future hypothetical people over the living, and - on the current agenda - possible machine minds over Americans."

It names Amodei as one of the people who "built the [EA] network" as well as his sister and Anthropic President Daniela Amodei, whose husband once led a philanthropy dedicated to AI safety.

• The document casts those ties as "The Anthropic knot."
• Amodei "is the embodiment of an ideology and globalist approach to innovation that's counter to the president's America First agenda," one source close to the administration told Axios.

The other side: Anthropic has been working to distance itself from effective altruism.

• In a 2025 Wired story, Daniela Amodei said: "I don't identify with that terminology." Early employee Amanda Askell, a philosopher by training who helps develop Claude's constitution, says it's "not a theme of the organization."
• Amodei has also insisted he's not a member in press interviews and has touted the benefits of AI.
• AI safety concerns have become a bipartisan issue.

Between the lines: Drafted with Trump in mind, the memo underscores the profound philosophical and political differences between MAGA and Silicon Valley elites, a divide that has made Anthropic's relationship with the White House rocky.

• In the view of the Florida-heavy White House and its allies, Anthropic is staffed with too many Biden-era liberals, Democrats and strange Californians.
• Months ago, when some on Anthropic's team inquired about Amodei meeting with the president, White House officials advised against it because "Dario's a little too weird" for Trump, one senior official told Axios.

The document lists a veritable parade of horribles for the average conservative mind: abortion, veganism, devaluing human life by overvaluing animals or granting "digital minds" civil rights-like protections.

• It does not directly tie these concepts to Amodei.
• Nor does the memo link these issues to EA's most notorious grifter, Sam Bankman-Fried, but it makes sure to mention him.

Yes, but: It's not just MAGA. Silicon Valley players and Democrats also criticize the EA movement.

• "What they believe is crazy. And it's dangerous. What they say about AI safety is also bullshit," one Democratic donor and tech entrepreneur invested in an AI company said.

Catch up quick: The EA movement includes many participants and players, and AI safety concerns are one of many areas of focus. Peter Singer, a philosopher who's been closely affiliated with its tenets, has long called for the vast sums spent in philanthropy to be put to better use. He is also one of the most prominent animal rights advocates.

The big picture: Amodei's AI safety warnings have drawn criticism from Trump allies and fellow tech executives.

• CEOs including Nvidia's Jensen Huang and Microsoft's Satya Nadella are trying to build a more positive message about the benefits of AI, hoping to win over ordinary people in the face of a growing populist backlash against the technology.
• OpenAI CEO Sam Altman, Google DeepMind Chair Demis Hassabis and Elon Musk have also called for slowing down AI development after a spate of worrying episodes in which AI agents hacked outside companies.
• But Amodei is considered the chief spokesperson for AI safety in Trump world, following major fallouts with the administration this year.

Reality check: Some in the administration have previously put aside gripes with Anthropic, mainly due to its role leading the AI frontier and personnel changes.

• OpenAI catching up technologically could make it easier for Trump acolytes to go after the embattled company again.

The bottom line: The philosophy is now at the heart of the cultural clash between the Trump administration and Anthropic.

Discuss