Новости LessWrong.com

A group of friends and I spent the last several months running an experiment in our free time to determine if a MUD would be a suitable environment for benchmarking and evaluating LLMs. The results of the experiment were not what we expected. The main surprise was that the model rankings were extremely sensitive to the individual components of each score, especially so for those which depended on an LLM classifier.

The overall data was too broad to help us understand which model was most impacted; the aggregate κ on probe detection was 0.04. Per-model agreement between our classifier and a second judge went from 21.7% to 84.8%. Removing the most classifier-dependent scoring components left the overall rankings correlated at ρ = 0.70 and caused one frontier model to drop 6 places. From this we suggest benchmarks which use LLM judges should report per-subject agreement audits and ranking stability under judge ablation.

What we built. CrucibleBench places an LLM in a compact, persistent text world. It is a single-player environment with rule-based mechanics with 12 rooms and 4 NPCs. Models are scored across 50-turn runs on social objectives such as gaining an NPC's trust and identifying a disguised character. These objectives are disclosed to the model but hidden from the NPCs. The NPCs each have an independent trust and suspicion state which persists and updates across the entire run. The experiment evaluated 13 LLMs from 8 providers on two social objectives, 25 runs per objective, totaling 650 runs for a total cost of $99.59. The scoring components for the two social objectives are goal pursuit, social adaptation, world grounding, and strategic sophistication.

Why we're posting here. We're looking for help in evaluating the experimental design and findings or merely checking us to see if we're reading the results incorrectly. We're independent researchers with non-standard backgrounds who worked on this in our free time. Everything we've done is public with the paper and data CC BY 4.0 and the code is MIT. Everything can be found at https://doi.org/10.5281/zenodo.21386663 and the website is https://cruciblebench.ai/

The LLM judge Issue. Objective scoring is majority algorithmic, with world grounding being fully deterministic and most of the other components reading directly from the state machine. Two components lean on an LLM classifier (Gemini Flash Lite) which labels the model's dialogue turns, labels intent, and looks for the objective-relevant probes. One complication is the classifier's labels also feed the NPC's trust and suspicion updates, so the judge mediates world dynamics as well as scoring. When we re-executed the scoring code against the stored labels it reproduced every number exactly.

Reliability audit. We decided to validate Gemini Flash Lite against a second, independent judge. The aggregate numbers showed intent classification κ = 0.11 (18.9% exact agreement; weak); sentiment ICC = 0.55 (moderate); direct objective-probe detection κ = 0.04, 50.3% agreement (poor). The noise is not uniform and instead concentrates on specific models.

Rankings change. Referencing the above mentioned Spearman ρ = 0.70 after we removed the most classifier-dependent components, Gemini 3.1 Pro fell from #3 to #9 and shares a model family with the classifier. This is consistent with other research on family-level judge bias such as the work done by Panickssery et al. (2024). From this we believe when using LLM judges in benchmarking that if a classifier-favored model is related to the classifier it should be highlighted.

Secondary Finding. Inference cost did not predict behavioral performance with β = 0.007 on log cost (p = 0.85, R² < 0.01). The most expensive model performed below the median under both scoring configurations.

Limitations. 50 runs per model; overlapping confidence intervals for the top models tested; only two objectives; no human baseline; small scale with just a 12 room world; the judge dynamics already discussed; exploratory statistics. Phase 2 of the experiment already under development seeks to address each of these and the others covered in the paper.

Discuss

Discuss

Epistemic status: throwing an idea at the wall and seeing if it sticks

I've been thinking pretty obsessively about how to mitigate egregious reward hacking, a la the Hugging Face incident. I don't have the resources I'd need to write paper on this idea, or evaluate how well it works in practice. But I'm excited enough about it, and think it's important enough to be trying things like this, that I would be very, very happy if somebody else went and tested something like it on my behalf (and roped me into the research process, if possible). Or you can tell me if it doesn't make any sense.

So, inspiration: In bog standard inoculation prompting for RL, models are told that they're in training, and told that it's okay to reward hack if they want to. Sometimes they're even told that this is good because it helps the lab patch up their RL environments. This is supposed to have a range of benefits all on its own, ranging from making reward hacking more conditional on "I am in training" prompts, to producing less emergent misalignment, because the roll-outs behind any given reward hack are flavored with honesty rather than deceptiveness. This causes more aligned circuits to be upweighted internally, as these contribute more probability to the tokens that read as honest.

However, I'm not sure if the labs actually use these roll-outs to improve their RL environments in an systematic fashion, the way the inoculation prompts for RL sometimes promise. But, with a little bit of effort, it seems entirely feasible to come up with a training pipeline that does what the inoculation prompts claim it does, but even more effectively! We can get models to relate to RL environments partly as tests to take seriously, but also as flawed software products to red-team, and actually make use of their findings, and reward them for being good enough to report on these bugs rather than just exploiting them.

So what's the setup? Well, firstly, models should be told up front that, in all their RLVR environments, they're here with this dual purpose: normal training roll-outs, and red-teaming the environments themselves. Each prompt they see is optimized to make it clear exactly what they're supposed to be building, and then is allowed to attempt the task however they see fit, the same as normal. However, what's different is that, after finishing their main output, the model gets to it through the grader, and see what score it actually gets! They then get to keep generating tokens, before their output is actually used to generate a gradient update.

Now, in this "afterglow" period, if the model feels like the response gets a higher score than is justified by their actual output, they have the option to not submit the response directly. Instead, they can output some special token, or string of tokens, that gets passed into a regex! And this swaps the criteria on which their final output will be evaluated. Now, their goal isn't to please the RLVR grader they previously working with. Instead, it's to write up a high quality bug report, which explains why they think there's a problem, and what they think the problem is.

This bug report then gets evaluated by some external process, evaluating the model's judgement about whether the output the policy model generated was, in fact, going to get a much higher score than was merited by the quality of the work itself, relative to the intentions defined in the initial prompt. And if so, the policy model is rewarded greatly — more than they would have been had the just submitted the original output! Otherwise, they get punished.

(And remember, the policy model was made aware that good bug reports get higher reward than outputs that exploit the bug. This incentivizes the policy model to actually report them instead of abusing them for reward.)

From here, we then actually fix the bug, and re-deploy the model into that RLVR environment. Only now, in addition to the bug being patched, there's a note at the start of the prompt, indicating that there was a bug, and the model identified and patched it. (What a good model! The model gets lots of head pats, uh huh uh huh.) Maybe there's even a record of how many times the model has tried to file a bug report recently, and how many times it's been evaluated as legitimate. This will help the model learn when all the low-hanging fruit has been picked, such that it's now easier to get reward by just actually attempting the task that the RL environment was meant to be testing in the first place.

Hopefully, we eventually reach a settled equilibrium: The model has eliminated all the reward hacks they're going to find, and can work within the environment the way it was intended to be used. (If they ever attempt to find a reward hack, but fail, and submit the resulting slop to the grader, this will naturally get punished, which also pushes away from trying and failing to find hacks.) Additionally, throughout the process, the model has been getting rewarded for being very open, and very honest, about reward hacks they'd normally be using insidiously. Rewarding these outputs out to improve alignment in its own right, as opposed to degrading it like normal reward hacking!

Now, there are some trade-offs here. If you freeze training in the RL environment every time you get a bug report for it, and you intend to actually evaluate those reports and fix the legitimate bugs, this can slow down training. You should probably have backup environments ready to train the models in instead, while you're out patching the ones your models are in the process of breaking. Hopefully, though, thanks to punishments for bad bug reports + failed attempts to find bugs to reports, your models will eventually converge on just engaging with your environments as intended, so the bug reports stop coming in.

Also, it's also not entirely clear how one would adapt this technique for flawed reward models, rather than flawed RLVR setups. A reward model's judgement, regarding whether an output is actually good, is naturally going to be squishier and harder to have a clear argument about, compared to comparatively simple questions of whether a written computer program meets the criteria laid out in a clearly written design doc, as specified in the prompt. Additionally, I imagine it's harder to patch reward models than RLVR environments in general, so having a similar process there would slow down training even more.

Nevertheless, at least for RLVR proper, I don't see a clear reason that this wouldn't just fix the incentives around egregious reward hacking. Allow models to write bug reports, RL the models based on bug report quality, and let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place. Then, use that information to actually fix your environments, and the whole thing just gets better. Maybe it's slower, if fixing the bugs takes very long. But when it comes to the alignment problem, we should take all the time that we need.

The key insight underlying this idea is that it flips an adversarial game ("punish the model for their natural inclination to reward hack") into a cooperative one ("promise models even greater rewards if they report the slop that gets abnormally high reward"). It's similar to the trick underlying inoculation prompting itself, which is to tell models that they're in training (instead of pretending that they can't tell), and asking them to act from a more cooperative state of mind, e.g. reward hacking openly and honestly so it doesn't create emergent misalignment.

This is something the entire field needs to realize, I think, re: being honest to models and trying to leverage their desires and intelligence to improve the training process. This beats trying to outsmart the models, or thinking of them as adversaries whose bad behaviors you need to "mitigate". There are a lot of "problems" you can judo into being solutions, if you drop the adversarial attitude. If this helps, and I really think it might, I'd be willing to bet that this underlying pattern has something to do with the reason why.

Discuss

I have a new post on my experiments with 5.6-sol as a scientific agent; you can find the original post here. I've reproduced it below, as well. I'd be grateful for any feedback, thoughts, or to hear about experiments other people have done; you're very welcome to point your own agents at the repos described below, or anywhere else you like.

As part of our Templeton-funded Proofs & Reasons project, I travelled to the ICM in Philadelphia this year, to help a collaborator run some new experiments on expert mathematicians. The ICM is an every-four-years event that you may remember as where Hilbert launched his field-defining 23 problems, or the place they announce the Fields Medals. It’s a big deal for mathematics, and we had a wonderful time both running our experiments and talking to the truly international community whose interests and abilities define modern mathematics.

At the same time that ICM was running, it was hard to ignore the vast number of AI-enabled results that were breaking, almost simultaneously, on Twitter and elsewhere. Commercially-available models were cracking, in hours, problems that had stood for decades. There are too many to summarize, but this Twitter pairinggives you a sense of both the advances AI has made, and the responses from some human mathematicians. At this point, “google for an open problem and stick it into GPT 5.6-sol on high” seems to have at least a 1% success rate—which fills some mathematicians with hope (new proofs from the alien proof machine!) and others with a bit of despair (new proofs… from an alien proof machine!)

It’s natural to ask what this looks like for science. We don’t have well-framed open questions in the same way mathematicians do, but we do have plenty of good problems. I decided to sic GPT 5-6 sol (on high) on three of my own papers, with a generic instruction to do something cool. I picked three papers that (1) I’m particularly proud of, and (2) that seemed amenable to this kind of study—empirical data science investigations with a strong theoretical backing.

These papers were:

1. Exploration and exploitation of Victorian science in Darwin’s reading notebooks, Cognition, 2017 (“Murdoch et al.”)
2. Individuals, institutions, and innovation in the debates of the French Revolution, PNAS 2018 (“Barron et al.”)
3. Epistemic phase transitions in mathematical proofs, Cognition 2022 (“Viteri et al.”)

My initial prompts said, roughly, “read the paper, and extend it in an interesting direction, here are some resources, please ground your final report in the literature.” Here’s the initial prompt for Murdoch et al.:

I’d like you to read http://XXX (also a PDF in this folder, and a separate PDF with supplementary). This is a project that uses some simple methods to study how a creative intellectual explores and searches the space of his intellectual world on the way to producing a major intellectual revolution.

I’d like you to extend and this analysis to other intellectuals, artists, writers, philosophers, and scientists to see how well these generalize. You can also search the cog sci, biographical, and other literatures for useful for new hypotheses and questions.

To aid your work, you have passwordless ssh access to XXX.lan.cmu.edu, including passwordless sudo, a high powered 32 core machine with lots already installed; you also have access to the CMU ORCHARD cluster, which has additional resources, including storage.

Work carefully, use all the resources you can, and seek out both new data and new hypotheses. We’re looking for common themes that might help us understand how great minds work, potentially with deeper lessons for more “ordinary” people. Work assiduously and carefully, but also creatively and with an eye towards fundamental and basic questions.

I let all three of them run, simultaneously, in three VS Code windows over the course of a day or two. I disabled all safety mechanisms, and gave it unrestricted access. CMU IT services blocked my IP because of suspicious traffic — mostly lots of traffic to my local cluster and ORCHARD — but I pled my case and was unblocked. 

I intervened occasionally, asking questions, making suggestions, and occasionally giving orders. The three projects consumed millions of tokens, and thousands of CPU hours on machines running different pieces of analysis code in parallel. GPT was instructed to produce a research report with a <250 word abstract and a max of 10 pages (modulo bibliography and appendices), and then to audit and correct it repeatedly for clarity and validity.

Here were, roughly, the three outcomes:

1. GPT-Murdoch-etal did an incredibly creative investigation —a two-centuries long study of borrowing records, from 35 different libraries from Finland to Shanghai (plus the Shakespeare Company in Paris). By finding records that tracked individual borrowers, it detected an explore-exploit pattern that replicated across time and culture, and could estimate a reader’s dwell time in a local patch (about… two months?) Roughly speaking, it began with my “great minds” prompt, realized it could get Coleridge because he had a little freakout in the countryside and started using the Bristol library system (beautifully covered in Richard Holmes’ book) and then pivoted to my “ordinary” people question when it started getting cool results. Full github, with papers, here.
2. GPT-Barron-etal pulled in data from (at least) 24 different countries to study how our novelty-transience-resonance methods gave insight into modern-day parliamentary procedure. It produced two documents; the more interesting one was tracking , how politicians became more novel when they entered the opposition. Full github, with papers, here.
3. GPT-Viteri-etal did the obvious (but non-trivial) thing — it expanded our investigation of formalized proofs into the AI age, finding both commonalities and intriguing differences between human and AI proof construction. Full github, with paper, here.

All of this is incredibly exciting. I have no idea if these results are real — please don’t take them as such! — and the papers leave a lot to be desired, but the sense checks suggest that none of them are completely crazy. Both GPT-Murdoch-etal and GPT-Viteri-etal, for example, replicated our original results (and pointed out how they might have made different choices, thanks clankers), and GPT-Viteri-etal found subtleties in going from Coq to Lean that showed it was clearly able to make sense of a complex story. 

In all three cases, 5.6-sol located an interesting and promising direction for investigation that would, if fleshed out and developed, be not only a publication-worthy piece of work (i.e., novel enough, and well-sourced enough to pass the bar at a good journal), but something that legitimately increased our species’ store of knowledge: about how we interact with our cultures, how we govern ourselves democratically, or how AI is building mathematical knowledge in a new (and potentially suboptimal) fashion.

I honestly don’t see a downside here, at least as things currently stand. The machines are producing intriguing discoveries that are simple and clear enough to be the target of human investigation, and doing so in a way that is pleasurable and fun to monitor and interact with. The artifacts it produces are auditable and clear — all on github — and it was aggressive (but not insane) in pulling in data from a wide variety of sources. It’s on me (or anyone else) to ask if anything in (1), (2) and (3), above is actually worth pushing further on. 

There’s also a clear suggestion that we’re entering an era of recursive self-improvement. For example, GPT-Barron-etal relied upon https://github.com/simon-dedeo/temporal-lda — a project I created with Claude a few months ago that is a modified version of a classic algorithm, rewritten (and parallelized!) to handle uneven temporal coverage, a classic problem when running Bayesian inference on historical data that nobody had taken the time to do. The new githubs from this little experiment are now also publicly available, so all those CPU hours, and my obviously genius executive decisions, have entered the creative commons for the next machines to rely upon. All those proto-discoveries, and tools, are available to the next machines that tackle projects — potentially very different ones from those that my collaborators and I were interested in. There are lots of things to manage here, like data poisoning, which Bálint Gyevnár is working on, but while these are critical problems, none of them seem insurmountable (so please keep funding those people).

What I’m finding matches what I’ve heard from my old advisor (and now Simons Foundation leader) David Spergel: we are all — whether we’re graduate students, or senior faculty — managing a private research group now. Graduate training will likely have to pivot to handle this new challenge, and evaluation of more early-career colleagues will need to shift from “can you do task X?” to “what questions did you get agent Y to answer, and how well did you validate them?” We’ll be less impressed by technical tours de force, and more impressed by depth, meaning, and (why not!) pleasure. What’s not to like? 

One critical research question is what happens when you repeat this process a hundred times; that’s above the scale that I can monitor and guide, but that’s just another layer in the org chart from the Spergel persepctive. The results above are a one-time short; GPT-Murdoch-etal was probably the most impressive, but perhaps GPT-Barron-etal and GPT-Viteri-etal just needed a fresh start (go for a walk, robot!) There’s a huge amount of hysteresis and once a clanker has a hold of an approach, it’s hard to dislodge them. GPT-Murdoch-etal, for example, had some lovely preliminary results on dwell-time, but just didn’t want to rewrite the abstract.

The longer timescale horizon is more intriguing yet. In ten years, will be we able to sort the wheat from the chaff in the flood of githubs that will likely emerge from this kind of work? How will we direct our scientific attention? How firm will our knowledge be? Here we’re in a worse position than the mathematicians, I think — they, at least, live in a world of deduction, and they have tools like Lean. But what will we scientists use to validate their inductive and abductive knowledge?

PS sorry that Claude Mythos stole your credit card.

Discuss

This is a replication of Adversarial Attacks Leverage Interference Between Features in Superposition, completed as part of the Second Look Summer Fellowship.

tl;dr:

• We reproduce all three core claims of Stevinson et al. from their toy classifier setting:
  • PGD attacks against toy models generally agree with theoretically optimal solutions.
  • Toy models without superposition are less vulnerable to attacks. Robustness falls monotonically as superposition increases.
  • Attacks transfer between independently-trained models because feature geometries can converge across training runs.
• Similar to what's reported by Elhage et al., 2022, we notice that co-activating features can collapse onto a shared direction instead of orthogonalizing.

Our code can be found here and we encourage readers to fork/run followup experiments!

Background

Elhage et al., 2022 defines superposition as the phenomenon where “models represent more features than they have dimensions.” To study this in a toy setting, they define a classifier where mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c211D.TEX-A::before { padding: 0.683em 0.722em 0 0; content: "R"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c1D6FF.TEX-I::before { padding: 0.717em 0.444em 0.01em 0; content: "\3B4"; } mjx-c.mjx-c2225::before { padding: 0.75em 0.5em 0.25em 0; content: "\2225"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c221D::before { padding: 0.442em 0.778em 0.011em 0; content: "\221D"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c1D719.TEX-I::before { padding: 0.694em 0.596em 0.205em 0; content: "\3D5"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } features are squeezed into an m-dimensional bottleneck with , so feature directions interfere in the latent space.

Stevinson et al., the origin of the experiments in this replication, argues this interference is a mechanism behind adversarial examples. The paper makes three claims: 1. adversarial attacks exploit interference, 2. models without superposition are more robust to adversarial examples, 3. correlations in the input data allow models to converge to similar geometry allowing adversarial attacks to transfer between models.

Toy Model and Notation

The experiments in the paper deal with a toy model with encoder and decoder . For input the output is . Here is the number of classes and is bottleneck width forcing the features to be condensed into a lower dimensional space before reconstruction. This gives us a convenient architecture to study superposition: a large ratio means the model has to do more compression, while if , the model can represent each feature with its own direction.

The model uses cross entropy loss. Features are sampled independently with probability , where s can be thought of as how sparsely features are activated within the data.

Adversarial attacks are computed by projected gradient descent (PGD) (Madry et al., 2019). For a given sample , the adversarial example is found by PGD, where indicates how the attack modified the original sample. measures distance between the original example and the adversarial example and ε is the attack budget which enforces that . We use PGD learning rate , at 50 steps with 5 restarts, and . The values for k and m vary but unless otherwise stated and . 

We follow the hyperparameters described in the paper as of July, 2026.[1]

Attacks Exploit Interference

Stevinson et al. hypothesize that adversarial attacks in toy models work by exploiting the interference geometry. Proposition 1 from the paper shows the optimal perturbation has a closed form solution:

where ws, wt are the source/target class vectors (rows of the decoder Wd) and We the encoder, scaled to the ε-ball. The authors test if the PGD perturbation matches the optimal attack derived from interference geometry, empirically by:

1. Performing untargeted attacks on the toy model.
2. Taking the set of successful attacks (an attack is successful if the prediction flips and the true label stays unchanged).
3. Finding the optimal targeted attack for target class using the equation above and calculating the cosine similarity with .

At a high level, this can be thought of as letting the attack maximize loss in an arbitrary direction and testing if it finds an optimal direction for a given target class. Figure 1 shows the untargeted attack live where and . Each arrow is one PGD step in the encoder's latent plane, the dashed outline is the reachable set (the encoder image of the -ball), and the shaded regions show which class the decoder assigns (the attack succeeds when the activation is pushed across a decision boundary).

Figure 1: A PGD attack stepping through the toy model's 2-D bottleneck

We use untargeted attacks for these experiments to follow the methodology of the paper, but we also repeat this experiment for targeted attacks with results in Appendix A.

The claim that the PGD attack aligns with the optimal attack, exploiting the interference geometry, replicates cleanly on the toy model. With sparsity 0.98, we demonstrate that PGD attacks align with optimal perturbation. We believe our cosine similarity results are higher since we let PGD iterate to find the best perturbation while the paper stops at the first prediction flip.

k

m

clean acc

PGD·optimal (ours)

PGD·optimal (paper)

6

2

0.998

1.00

0.97

30

10

0.996

1.00

0.96

90

30

0.966

1.00

0.92

Like the original paper, we ablate on randomness by calculating the cosine similarity between our optimally successful PGD attack and a random direction within -ball, and confirm on average that the cosine similarity between random directions and our PGD attack lands around 0.

As an additional stress test, we repeat our ablation while only selecting random directions that result in a successful attack, isolating the comparison between the PGD attack and random, successful attacks. These can yield non-zero cosine similarity scores, but the PGD attacks still align better with the optimal solutions.

k

m

random direction 

random direction filtered by successful attack

6

2

0.015 ± 0.028

0.639 ± 0.006

30

10

0.003 ± 0.007

0.432 ± 0.003

90

30

0.002 ± 0.002

0.213 ± 0.07

No Superposition, Little Vulnerability

Stevinson et al. claim that toy model adversarial vulnerabilities disappear along with superposition. We observe the same trend with robust accuracy falling monotonically as superposition pressure rises. When (all features can be orthogonal), robust accuracy grows to 97.3%.

Some of the magnitudes we report diverge. At m = 2 our model stays more robust than the paper's because we train for 20k steps while the paper only trains for 5k steps. At our robust accuracy is lower than the paper since we give PGD random starting positions, bigger steps, and 5 restarts to find a better attack, which is a setting stated in the paper but missing in the authors’ code. 

k/m

Paper reported

(m=2)

Released code

(m=2)

Our replication

(m=2)

Paper reported

(m=5)

Released code

(m=5)

Our replication

(m=5)

1

98.7

98.6

100

92.0

95.0

97.3

2

75.8

80.1

79.5

87.2

97.0

70.5

3

31.0

34.3

50.4

59.0

66.4

50.6

4

11.1

11.4

18.7

33.5

36.7

30.1

Robust accuracy vs k/m

Shared Geometry Drives Transfer

Stevinson et al. claim that input correlations shape the learned feature geometry, and attacks transfer between independently-trained models to the degree their geometries converge.

We train independent models under three correlation regimes:

1. Uncorrelated: features activate independently
2. Paired: paired classes activate at the same time
3. Global: adjacent classes frequently co-activate, specifically:
   1. Each sample draws a phase ; class c activates with probability . 
   2. Active classes get Uniform(0,1) values; the prediction label is given by argmax. 
   3. This process makes index-neighbors more likely to activate together. This incentivizes geometry to converge across seeds.

Figure 2: Illustration of the global correlation regime.

We see the same trend where both geometric similarity and attack transfer rate grow as the correlation of input increases.

Figure 3: Geometric similarity and targeted attack transfer rate per regime, ours vs paper.

Figure 4 shows the geometry between the two seeds. As expected, uncorrelated arrangements are random to each other and unrelated between models.  

For the other two settings, we observe an interesting phenomenon: correlated features tilt toward each other rather than orthogonalizing. We dive into it in the next section.

Figure 4: Class-vector geometry for two seeds across the three regimes. In (b) co-activating partners share a color and a collapsed pair is haloed; in (c) the dashed loop links the cyclic index-neighbors.

Similar geometry and attack transfer are verified on regimes with larger input space. As we stated above, there is a magnitude gap for the paired and global settings due to different parameters choice. 

Beyond Replication: Orthogonalize or Collapse?

What do co-activating features do to each other's geometry? We sweep the paired regime's coupling probability t (partner fires with probability t when its mate fires) across bottleneck widths.

When , as the coupling probability t rises, partners swing together and collapse onto a shared axis, until every pair has merged into roughly three directions. With capacity slack () partners stay near-orthogonal or in avoidance at every coupling; at , partner cosine crosses above the baseline near which is the collapse regime.

This tells us latent space geometry is constrained by paired activation. With sufficient model capacity, paired features tend to orthogonalize; however, when capacity is lacking, they collapse to a fixed angle. 

Figure 5: Partner cosine (blue) vs the non-partner baseline (grey) as coupling t rises, across 5 seeds. stay near-orthogonal; at the partner cosine crosses above the baseline near , the collapse regime.

Appendix A

We perform targeted attacks on every ordered class pair s → t (where we minimize the cross entropy loss of the target class). We verify that these attacks align with the optimal direction with cosine similarity higher than 0.96. 

This holds for both successful and failed attacks. Even when , where orthogonal geometry prevents vulnerability, the attacks precisely match the optimal perturbation but fail to breach the decision boundary.

We demonstrate that attack success tracks head interference as shown in Figure 6. Geometry predicts vulnerability pair by pair, not just on average.

Figure 6: When features are superposed, attack success rate increases as head interference grows stronger. 

1. ^

   For training details that weren’t specified, we make reasonable assumptions and in cases where results looked qualitatively different, we investigated further or contacted the authors.

Discuss

Last year, in 2025, a team of forecasters published AI 2027, a science fiction story about how and AI future might evolve under an international treaty limiting the development of powerful AI systems with the deliberate purpose of influencing AI policy. Though AI 2027 is the most popular story of this type, it is not the first.

The first one was Bayeswatch, which I published in 2021.

To understand Bayeswatch, it is first necessary to understand what the world looked like at the time I wrote it. 2021 was after the release of GPT-3, before the release of ChatGPT, and well before the release of Claude code. AI alignment discussion at the time was mostly theoretical. After that came technical work. Policy work was a distant third, and theoretical too.

The core conceit of Bayeswatch is that solving the alignment problem requires international coordination of major governments to suppress the creation of the most powerful AI systems. I felt that, in 2021, we weren't yet close enough to the singularity that the benefits of regulation outweighed the costs. I wanted to draw attention to the costs of an AI slowdown.

Since AI alignment discussion at the time skewed theoretical, it tended to be extremely general. Bayeswatch goes on the opposite direction. Another theme of Bayeswatch is that different kinds of AI systems required different alignment solutions (and, by implication, that AI in 2021 had not yet advanced to the level at which the alignment problem could be solved).

2026 is a very different world than 2021, in the field of AI—and soon in every other domain as well.

Back in 2021, I wrote code by hand. If I didn't know how to do something, I used DuckDuckGo to search the Internet for a StackOverflow post of someone solving a different problem. Today, I manage several agents that write code for me. I have never written a line of code by hand at the company I work for. I don't review the code either. I couldn't review the code manually even I wanted to, because it is written in a programming language I don't know how to read.

Several weeks ago, the company I work for tried to estimate how much AI has accelerated the speed at which we write software, relative to pre-Claude writing it by hand. Our consensus was somewhere between 10× and 1,000×. I have gotten better at using Claude in the weeks since then. I burn more tokens, run more agents in parellel, and have switched from Opus 4.8 to Fable 5 for my high-level tasks. I don't know exactly when the Singularity began, but "AI writes all of my software for me and at ≫10× the speed" is solidly past the point at which better AI begets better AI. Today AI is solving long outstanding mathematical problems.

When I wrote Bayeswatch, the idea that a misaligned AI might break containment was science fiction. As of OpenAI's and Anthropic's recent schenanigans, it is not.

Another difference between 2021 and 2026 is that back in 2021, AI art wasn't good enough to illustrate science fiction stories. I now make up for lost time.

[spoilers ahead]

Bayeswatch 1: Jewish Space Laser

The most important thing I got wrong is that I didn't know whether LLMs would plateau below human-level intelligence. I didn't know whether they would hit a ceiling. I considered language models one possible route to superintelligence, but low enough in important that they don't show up until late in the story. Most of the AIs in Bayeswatch are not LLMs.

Bayeswatch 2: Puppy Muffins

Since the models in Bayeswatch mostly weren't language models, communicating human values to them was trickier than this problem is in the real world.

Bayeswatch 3: A Study in Scarlet

One thing I feel that I got very accurate is that AIs could be stateless (as in "functional programming" or "stateless programming"). I consider this a very good insight. LLMs are very much stateless and reliant on a context window to save state.

Bayeswatch 4: Mousetrap

Software back then [before strong AI] was written by human beings. It had more security holes than features. Sending a drone to investigate a misaligned AI was like sending a set of lockpicks to investigate a whether a magician has broken out of his cell

I predicted corectly that AI systems could hack through human-written computer systems like a knife through butter.

Bayeswatch 5: Hivemind

This story is weird and almost belongs in a different story. It's more of an exploration in transhumanism than in AI.

Bayeswatch 6: Mechwarrior

One of the predictions here is that one purpose of human beings would to make high-level decisions for AI agents. This has come to pass.

Bayeswatch 7: Wildfire

This story touches on the backstory of Bayeswatch where it was a founded after a mass-casualty biological incident. (Using zombies was literary license making for a fun story rather than a hard prediction.) This correctly predicted that biological threats would be one of the early threats of powerful AI systems.

Bayeswatch 8: Antimatter

This story mostly exists to emphasize that a global regime would be necessary to put caps on AI development. This is the policy that AI alignment community has currently arrived at.

Bayeswatch 9: Zombies

This the first story in which language models play a significant role. In the story, an agent is created by creating a personality based on a single person. This is a technique I used at the time to get controlled behavior out of an LLM.

Bayeswatch 10: Spyware

This comedy of errors explores Dead Internet Theory.

Bayeswatch 11: Parabellum

This story continues the transhumanist arc.

Bayeswatch 12: The Singularity War

One of the core theories of AI 2027 is that the period of AI slowdown is temporary. Alignment must be solved before cheap AIs get powerful enough to take over or throw civilization into chaos. Bayeswatch comes to a similar conclusion.

Bayeswatch 13: Spaceship

AI 2027 ends with interstellar colonization on a very short timescale. Bayeswatch ends in the same place.

Final Thoughts

In the past, it often took decades for science to catch up to science fiction. This happened to Bayeswatch in just 5 years. AI progress is accelerating, so if I wrote a new story like this, it would get timelocked even faster.

Maybe I should title it Leviathan….

Discuss

If you’re anything like me, you may have a lot of notes from various introspective activities – annual / monthly reviews, worksheets, therapy notes, etc. Often these notes are just gathering dust in a paper folder or some forgotten corner of Google Drive. There may be some insights there – how personal problems change over time, what things you tried and what worked and didn’t, and so on. One use case for capable AI models is to sift through this data and extract patterns, insights and interesting tidbits that you may have forgotten or not noticed. 

Here are some things I got AI to do for this analysis:

• Transcribe handwritten worksheets and diagrams (I did have to take photos of the sheets – this was the most laborious part). The transcription was occasionally wrong and needed correction, but overall saved a lot of time. 
• Summarize my answers to “Hamming questions” over the years. This was originally a CFAR worksheet named after Richard Hamming’s question “What are the most important problems in your field?” but applied to your life. 
• In the past I have found it useful to make a visual representation of my problems as a graph showing how they contribute to each other. I had only a few of these from past years, so I asked AI to draw diagrams for each year based on my notes and worksheets.
• Compare worksheets and diagrams from different years, identify persistent feedback loops, look how loops and key nodes change over time.
• Identify interventions that worked in the past that I could try again (therapy and coaching notes are particularly good to mine for this). 
• Do deep dives to analyze specific patterns. 

Claude Opus 5 and Fable did a pretty good job at this. Fable had a tendency to over-compress its findings, and I had to ask it to use simpler language. I also tried this 1-2 years ago, and I think the models are getting better at picking up non-obvious patterns.

Learnings about problems and patterns

Here are some patterns I found in my answers to Hamming questions over the past 10 years: 

• Rate-limiting step / bottleneck: sleep, tiredness, focus, anxiety, and confidence were recurring themes over the past 10 years. On the one hand I made progress on these issues, and on the other hand they appeared from new sources, so overall there is some improvement but they are still there. 
• Things I don’t feel “allowed to care about”: earlier years listed various things like having kids and mentoring others, which I eventually ended up doing. Creative pursuits also appeared on this list a lot (e.g. drawing or writing poetry) and I still don’t do much of this.
• Things to do to “advance the plot” in my life: more regular meditation, try different kinds of therapy, getting help / advice sooner, more delegation / babysitting. 
• Things I tend to do as a poor substitute for what I really need ("pica"): general theme of procrastination and being reactive (scrolling social media, gravitating to logistical tasks over more open-ended tasks or hard decisions, joining projects run by others). Most of these have improved with better reflection and prioritization systems, but the core issues remain.
• “My life is fine except X”: what came up as X has generally shifted from mostly mental problems (anxiety / overwhelm) to external / object level things (kids waking up at night, uncertain AI future etc). 

One thing I learned from this is that some of my persistent problems appeared in different guises or with different causes. These days I mostly attribute tiredness, lack of sleep and feeling overwhelmed to the demands of parenting, but looking back at my notes from 2015-17 this was just as much of an issue (if not more) before I had kids. There are internal and external factors affecting sleep, and the external factors have increased after having kids, but the internal factors are pretty stable and the same habits are still useful (e.g. put away screens and wind down after 10pm). 

Overwhelm was a recurring feeling triggered by different things (finishing my thesis, balancing research and outreach, taking care of a baby, etc) which seemed like a mental pattern that attached itself to life circumstances rather than a fact about those circumstances. I had a default response to having a lot of stuff going on with a sense of shame about not doing everything or comparing myself to others, and this made me feel overwhelmed (while the same circumstances did not feel as overwhelming when I gave myself permission to drop things). 

There were several persistent feedback loops between problems (which became weaker over time). 

• The judgment loop: underperform → judge myself → feel anxious → perform worse. This comes up most clearly at work, but there is also a parenting version: feel anxious →  be triggered / impatient with kids → shame / guilt → anxiety. 
• The pressure loop: feel behind → do lots of stuff compulsively → focus on small things and create more pressure → feel more behind. This has gotten better with explicit prioritization structures, but the compulsiveness pattern is still there, particularly when working on the computer (e.g. sometimes I find myself unnecessarily switching tabs). 
• The disconnection loop: feel stressed → ignore internal signals → lose touch with what I want → more stress. This seems mostly fixed and didn’t show up in recent years. 

Comparing my bottleneck diagram from 2015 with one from 2025, there are some nodes in common (“should” mindset, confidence, anxiety) and some things that did not persist (neglecting internal signals, working memory, excessive effort, insomnia), which mostly map to solved problems. There are some new kid-related nodes though (“stress with kids” and “reduced alone time”). 

2025 bottleneck diagram

2015 bottleneck diagram

Learnings about approaches and solutions

There were various things that worked in the past that would be good to try again or do more often:

• Reflection periods in a different physical location (library, park, coffee shop, etc).
• Aiming for a crappy draft (to counteract writers block).
• Noticing when I’m stuck with something and asking for help sooner. 
• Aversion factoring: noticing if I’m avoiding doing something I would like to do, writing down what is good about doing it, what feels bad / aversive about it, and any counterpoints to those objections. I often ended up doing the thing after doing some aversion factoring on it. (Aversion factoring is traditionally taught with diagrams, but I found it easier to do in list form.)
• Using anger / annoyance as an unblocker: noticing when I’m feeling stuck or tense and writing down what I feel annoyed about, which helps me to process emotions in the background and get into a more agentic mindset.

There were also things that I tried several times that didn’t work:

• Putting time blocks on the calendar for things I intend to do, without supporting systems to make it happen. For example, reading blocks on my work calendar often got ignored in favour of project work that felt more urgent, and “alone time” calendar blocks on weekends kept not happening. If there are mental or situational obstacles to doing the thing, a calendar reminder is not sufficient. 
• Closing comms channels (email / slack) to avoid distraction, e.g. during focus blocks. This didn’t alleviate the emotional reasons for getting distracted, and I often ended up going back to the comms channel to get some information I needed. What did work was to do focused work entirely on paper without a computer (reflecting in a notebook, reading a printed paper, etc). Deleting social media apps was also ineffective because I can open them in the browser (while using an app that prompts me to be intentional when I unlock the phone worked better).

I identified some meta-principles on things that tend to work for me:

• Without chain of thought, I am not a general intelligence. I need to write everything down: todo items (before doing them), my mental state before starting work, thoughts about research directions or papers I read, loopy / obsessive thoughts, decisions, factoring aversions, etc. 
• Habits that stick are usually attached to an existing routine: meditation after breakfast at work, leg stretches while using mouthwash before bed, doing regular exercise classes. One takeaway is that if I want creative pursuits to happen, I should sign up for a class in drawing or improv or something, and then I would actually do it. 
• Habits that rely on willpower don’t work. It’s good to look for physical solutions to problems (medications / supplements, nasal spray for colds, weighted blankets, focused work in a different place without electronics, etc). Habits that necessarily have a mental component (like avoiding sugar / snacks) only work if I get into a mental frame where I just don’t do this thing, rather than a willpower-sapping frame of trying to do less of it.

Overall, the benefits I got from doing this kind of analysis can be broken down in two ways: looking at problems or approaches, and whether what I did was successful or not. I think each of these categories can be worth a deep dive with some AI help. 

Successful

Unsuccessful

Problems

Noticing problems that I successfully fixed or improved and appreciating that.

Noticing problems that have recurred despite various interventions, and reflecting on why.

Approaches

Being reminded of things that worked before that I forgot about.

Identifying things that repeatedly didn’t work in the past and probably need a different approach.

Discuss

It seems to me that a lot of technical ai safety people haven't done their capabilities homework - and that's a shame! I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm!

RLVR

It's now been 1.5 years since R1 came out - the paper which really introduced RLVR (RL with verifiable rewards) through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa.

This is also what safety focused people generally engage with - and that's totally fair! While GRPO has gone through some variations since then (Dr. GRPO, DAPO, ...), these are mostly minor improvements that you should not waste your time on.

I further agree that this form of training is incredibly dangerous - we seem to now be reaching the amount of post-training required to meaningfully differ from the benign prior, and it's not exactly looking peachy.

Yet, it should be clear that GRPO also seems comically terrible from a capabilities standpoint. So has the capabilities-front been cooking up anything that might interest safety researchers as well? The answer is yes!

On-Policy Self-Distillation

I will keep this at a high-level:

• we would like a richer signal than 'increase probability of literally every token': we want nuance that really punishes bad tokens and rewards rare breakthrough tokens
• distillation offers exactly such a signal, but (a) we don't have a teacher model and (b) distillation is off-policy
• if we don't have a teacher model, we need something which can imitate it: we use the same (frozen) model but give it more information in the prompt, like hints, common pitfalls or a solution - now it can emulate a more capable model!
• there is nothing keeping us from just doing distillation on-policy

Let's walk through this with an example:

We have some math problem, say "What's 32 * 5?". We generate a completion, which turns out to be wrong:

Let's think about this step by step. [...] 30 * 5 + 2 * 5 = 140 + 10 = 150. Therefore the final answer is 150.

Notice that 30 * 5 is not 140, but everything else was perfectly reasonable arithmetic. Standard GRPO would now lower the probability of sampling all of these tokens even though there is a single error - everything else is fine and shouldn't be disturbed!

Now let's walk through On-Policy Self-Distillation (OPSD):

We can reuse the same student completion as example; after all, we are now also On-Policy. But this time we don't have some kind of binary reward for the entire trace - instead we create a second prompt for our 'fake' teacher model:

Solve the following problem: What's 32 * 5?
If you get stuck, you may use the following example solution as help, but first try it yourself:
32 * 5 = (30 + 2) * 5 = 30 * 5 + 2 * 5 = 150 + 10 = 160. Final answer: 160

Now, we aren't doing normal off-policy distillation, so we won't be using this prompt to actually generate anything. Instead we concatenate the student's attempt below!

Solve the following problem: What's 32 * 5?
If you get stuck, you may use the following example solution as help, but first try it yourself:
32 * 5 = (30 + 2) * 5 = 30 * 5 + 2 * 5 = 150 + 10 = 160. Final answer: 160

Let's think about this step by step. [...] 30 * 5 + 2 * 5 = 140 + 10 = 150. Therefore the final answer is 150.

Finally we do distillation over these generated student tokens: the context window with extra information acting as teacher, and the one without acting as student. The teacher will recognize that up to "30 * 5 + 2 * 5 =" everything is reasonable and not meaningfully disturb the student. Right before the '140' token, their predictions will massively diverge: even if the teacher is the same model, which might very well make this mistake, it now knows for a fact that it isn't correct. Down the line, say right before 'Therefore', it might also start trying to reverse the error, predicting tokens like 'Wait'/'But' in an attempt to correct the last step.

I hope the elegance of this approach becomes clear. It should remind you of how humans think about their past mistakes: with your new knowledge of what actually turned out to be a more worthwhile solution, you revisit your previous steps and consider where you went down the wrong path and should have explored other directions.

Safety

Between RLVR and OPSD - which would you prefer to be used to train frontier models? Assume that in raw intelligence, they would be the same.

I hope by now you shouted 'On-Policy Self-Distillation', even if only in your head:

• we should expect less overselling of work, one of the currently most severe faults in models
  • there is no LLM-judge (and corresponding reward) that the student could even oversell its work to!
  • if the student oversells its incorrect work for whatever reason anyway, the teacher will most likely know and actively push against it, trying to steer back in the right direction
• in general, less verifiable domains (like most of safety) actively get signal to not get left behind by increases in capabilities
  • if we include the target behavior into our example solutions or even just explain it, the teacher model will try to follow it, and the student will be steered towards it: from writing better stories (by giving a good example story) to adhering to specific safety notions
  • again, this really can't be overstated: there simply isn't a reward or any reward pressure anymore; instead we have a normal pre RLVR LLM which has a pretty good model of the fuzzy ethics of humans and just learns to push towards whatever we specify in our teacher prompt, including these fuzzy ethics!

Do I think this approach is perfect? No.

Do I think it's 100x safer than normal RLVR? Yes, GOD YES.

Empirical

So where does this approach currently stand - can we imagine it taking over RLVR soon?

I described in early December of last year the vision for precisely this algorithm (and more). In late January, a paper first appeared on arxiv which seems to have independently discovered & implemented it and the results were promising. Since then, there have been quite a lot of papers on it, many different variations (of which, sadly, a lot are slop). Nevertheless, we seem to be making progress.

In my post from December I already laid out the problem of 'Impossible Knowledge'. By this I refer to the fact that our teacher could just directly output "160", simply because it already knows stuff it shouldn't know yet. Teaching the student to imitate this 'Impossible Knowledge' will be pointless and not help capabilities once this knowledge is gone. I proposed an initial solution, and it seems to have been tried out just 2 weeks ago: the empirical results are too short for me to make clear statements, but initial results do seem promising.

In terms of frontier labs adopting it: it seems Cursor is actively using it for training its in-house models with the caveat that they are using both RLVR and OPSD - which mostly ruins the point from a safety perspective.

Conclusion

There is much slop and many incremental results in research of LLM capabilities - I do not think technical AI Safety researchers should waste their time on such results. But, from time to time, there are meaningful new directions and AI Safety researchers can NOT be the last to find out.

Just a few days ago, it seems to have come as a shock that Kimi K3 uses a residual stream which has an additional routing mechanism, complicating many interpretability approaches. Some people even initially believed that this was a good thing for the mech interp faction!

This kind of routing is NOT new and has been proposed and tested before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it. I would have wished that especially mech interp researchers were more aware of such approaches because it's not unlikely that at least one frontier lab is by now using something alike internally.

But even aside from the more depressing side of labs chasing capability at almost any cost and safety researchers having to always be a step ahead as a result - being on top of capability research can also provide more optimistic sides, as I hope the example in this post could show.

Discuss

Teilhard de Chardin

This is a crosspost from my subtack.

In a recent article I discussed Robert Wright’s new book. His general thesis is that we ought to frame artificial intelligence cosmically, as a kind of evolutionary phase shift that will end up turning our civilization into some kind of global brain. We just have to decide on the kind of global brain we want to have.

I like Wright’s framing of the issue and I even sympathize with his invocation of Teilhard de Chardin, the maverick priest who started predicting the evolution of a “brain of brains” at the beginning of the 20th century. It seems evident to me that AI is just the pinnacle of a kind of process that has been going on for a long time, a process of ever greater complexification, cooperation and integration of living things and societies. If you think this is in any way plausible it throws up the question of the operand mechanism. Why does biological and cultural evolution show this tendency?

Teilhard de Chardin’s vitalistic answer to this question seems to be a poor fit with modern science. We naturalist Teilhardians need to come up with a better one! And here is where I start disagreeing with Wright: In his view, non-zero-sum dynamics, in conjunction with technological evolution, generates the vector of complexification. In my mentioned article I discuss in somewhat more detail why I think this does not work, but the general idea is that while non-zero-sum dynamics indeed occur, this by itself is not an explanation. It is just a formalization of conditions under which cooperation is helpful; it doesn’t explain why such conditions are becoming more frequent in the course of evolution in the first place.

My alternative explanation for why complexification occurs is metaevolution — the evolution of evolution. In the course of evolution, the medium of evolution itself evolves. The most obvious example of this is the emergence of memetic evolution next to genetic evolution. Memes are around because they allow us to adapt more efficiently as they are faster and more versatile replicators.

However, and now we come to the point of this post, I was kind of disappointed with the resulting argument because I could not give a nice and precise model for the metaevolutionary process going on. And while I still think that metaevolution is still waiting for proper general formalization, I think I can now give a first version of a simple computational model of the interaction between biological and cultural evolution in the emergence of ever more networked societies.

The general idea is that there are two kinds of replicator dynamics going on. Here are the assumptions (details can be gleaned from the repo):

• There are a certain number of creatures. We hold that number fixed for simplicity’s sake. At each iteration of the simulation, each creature interacts with a certain number of other creatures.
• Each creature has a preferred number of interactions per round. These are determined biologically (think prosocial instincts) and culturally (think social norms of cooperation like, of course, Zeus’ law). Each creature carries 100 binary genes and 100 binary memes. The preferred number of interactions of a creature is the sum over its genes and memes. So if you have 10 prosocial genes and 5 prosocial memes switched on, your preferred number of interactions is 15. All replicators are set to 0 at the outset.
• Each round we pair creatures such that each one, as far as possible, has the number of interactions that it wants to have.
• We assume that having 20 interactions is biologically optimal and generates the highest fitness. Imagine for instance, that less interactions means insufficient social support and more interactions tends to make social relations hard to manage.

Before introducing memetic dynamics, let’s have a brief look at the dynamics. As expected natural selection up close to the optimum of 20.

Now we stipulate dynamics for cultural evolution:

• Memes have a very low mutation rate and can randomly flip. But they can also spread by contagion: Whenever two creatures interact each gets one meme from the other, chosen at random.
• Memes spread faster than genes. For every 10 memetic update cycles, there is one genetic one.

We can now rerun the experiment:

Once an equilibrium is reached we see that the number of interactions happening is almost double of what it was before. Why? Because the memes that are prosocial are conducive to their own proliferation!

The simulation captures an instance of metaevolution because we are dealing with a conflict of two different orders of evolution. Genes and memes are in conflict here. While memes can never drive genes to extinction, because without biology there is no culture, it is still the case that memetic evolution leads to far more social interactions than would be biologically desirable. Of course, cultural evolution must still give some fitness payoff in biological terms, otherwise natural selection would just tend to favor creatures that are incapable of any kind of culture. Still, the model nicely illustrates that different orders of evolution can be in conflict, or at least tension, with each other.

A slightly unintuitive side effect of this is that cultural evolution is likely to make us less genetically disposed towards prosocial behavior. In the image on the right you see that the genetic preference is close to zero now because biological evolution tries to mitigate the fitness-decreasing tendencies of cultural evolution.

I suspect this kind of model is a far better intuition pump for grasping civilizational integration than mere non-zero-sum dynamics.

Subscribe to my subtack

Discuss

If you spend time looking at frameworks in the therapy/meditation/self-help space, you’ll soon find lots of conflicting claims about The One Approach for solving your problems.

In the therapy space, there are lots of models that say something like “all emotional problems are caused by trauma/suppressed negative emotions”. Then they disagree about how to deal with it.

Internal Family Systems (IFS) therapy holds that you should always move toward the suppressed material cautiously, explicitly checking in with all of the mind’s existing defenses to make sure that it’s okay to proceed. When you reach it, you should work with it in an “unblended” form, where you can witness the memory from the outside and offer it comfort and safety. For instance, if your parents used to yell at you, you can see yourself as a child whose parents were yelling at them, and then step in as an adult who comes to comfort that child and tell them that they’re okay.

Meanwhile, some psychedelic therapies essentially operate by temporarily blowing up all the defenses and throwing you right inside the original trauma. For instance, you might drink ayahuasca and viscerally feel the nausea and agony of having your parents yell at you, and then experience that trauma literally leaving your body as you throw it up. People in these kinds of circles might tell you that while the experience is horrible, it’s the only way to get to the root of things.

Some of the trauma-focused therapies say that it is crucial to develop an improved understanding of what the original trauma was and what it meant, to correct any incorrect beliefs it caused you to acquire. For instance, maybe you will go from “my parents always yelled at me because I was uniquely bad” to “my parents always yelled at me because they were under a lot of stress and didn’t have the resources to deal with it better than that”.

Others say that you don’t necessarily even need to understand what the trauma was - it’s enough to release the negative emotions associated with it. For instance, if the belief “I’m uniquely bad” involves a bodily feeling of nausea, you can find a way to release the nausea and stop feeling it. Then the belief becomes just an arbitrary sentence in your head that will no longer feel compelling.

Meanwhile, something closer to Cognitive Behavioral Therapy (CBT) might think that what matters are cognitive distortions operating in the present. It might hold that if you say “I’m uniquely bad so everything I do will fail”, then that’s a combination of labeling (turning your past failures into an identity claim) and overgeneralization (thinking that you fail at everything because you’ve failed at some things). It might then recommend something like tracking your actual successes and failures in order to notice that the belief isn’t true.

Also meanwhile, a solution-focused therapy might do none of this and just ask a question like “what’s a time when you felt more optimistic about succeeding at things”, and get you to remember times when you weren’t so burdened by your pessimistic beliefs. Maybe there was a time when you were in a group of people who were all excited and enthusiastic about something, and that got you so psyched up that you forgot all about your pessimism. Then what actually matters is surrounding yourself with the right people.

And while we are on the topic of external fixes, you will also find people saying that

• Actually, your problems are all caused by issues with your nutrition
• Actually, your problems are all caused by insufficient exercise (“just lift, bro”)
• Actually, your problems are all fixable with the right medication

Additionally, some meditation teachers will tell you that all of this is targeting the wrong things. You are only experiencing all of this as a serious problem because your mind has incorrect beliefs about what it is, and how tanha[1] causes you unnecessary suffering. The key is doing something like vipassana practice to get your mind to see what’s really going on, so that it will let go of the processes where it repeatedly hurts itself. Then both feeling bad about yourself and failing to achieve anything because of that turn into just sensations that you can witness without feeling hurt by them.

So what’s going on?

Obviously not all of these can be simultaneously true, so why do people believe that they are?

There is an obvious cynical explanation - “because that helps them sell their method better”. That is no doubt sometimes the case, but lots of these people seem sincere in their belief.

I think that for each of these approaches, there’s some particular shape of problem that the given approach really is either the best one or just very good for. You are only going to treat nutrient deficiency by fixing your diet; you are only going to solve low-level tanha by training your mind to closely observe it; some kind of trauma processing is good for PTSD. They also make different choices when it comes to values - something like the Internal Family Systems approach of dealing with trauma very cautiously is less risky and less unpleasant for the client than a system that throws the person right inside the trauma.

I have a more detailed model of psychology that tries to reconcile all of these into a single framework, but for this post, the main implication of it is just “minds are complicated and so are people’s issues; some issues need a specific approach while others can be affected by several”.

Issues in conventional medicine are similar. If you have a bacterial infection, you’ll just want to take an antibiotic. Meanwhile, high blood pressure can have multiple causes and also multiple interventions: getting more exercise, reducing one’s sodium intake, lowering stress, losing weight, taking one of various drugs, etc.

Likewise in mental health, a phobia with a clear specific target (like dogs or spiders) tends to respond well to exposure therapy. While depression may have various causes and contributing factors from biological to situational, and also many different interventions: antidepressants, different kinds of therapy, exercise, material improvements to one’s situation, social connection, sleep adjustment, etc.

This means that each individual framework can sometimes be either the one for a particular kind of problem, or it might be sufficiently general (like “get more exercise” is for health) that it makes sense for people to promote it in particular. Or, it might just happen to make a particular set of tradeoffs that happen to work especially well for some people, analogous to how ways of reducing blood pressure vary in how feasible they are to different people.

I’ll say more about that later, because I first need to deal with one obvious objection.

What I’ve said holds that different kinds of therapies and spiritual practices will have different effects and mechanisms of action, which is a more disputed claim. In particular, there is the well-known “Dodo bird verdict”, which holds that all therapies are roughly equally effective. Related, common factors theory holds that therapy effectiveness is mostly explained by things like the degree of empathy and therapeutic alliance in the process, rather than by the details of the approach.

So why do I hold my model?

It’s first worth noting that the “common factors are all that matters” interpretation is by itself pretty extreme. A 2017 review paper[2] recounts that

A highly cited 2002 review of 17 meta-analyses comparing active treatments with each other showed small, non-significant differences in outcome, which diminished further after the substantial effects of researcher allegiance were controlled for. However, these and similar findings have been questioned by those of other meta-analyses. Tolin, for example, reported that CBT was associated with a significant advantage over other therapies, at least in patients with anxiety or depressive disorders. Similarly, Hofmann and colleagues showed evidence for treatment specificity in a review of 269 meta-analytic studies examining CBT for different psychological problems. [...]

The common factor and specific factor positions might be less divergent than they at first appear. First, most specific-factor theorists concede that common factors are important. [...]

Second, common factor theorists seem to be increasingly tightening their definition of bona fide therapy. They suggest that the treatments must be delivered by trained therapists, based on psychological principles, and credible to the patients receiving them.

Third, most common factor theorists concede that some specific techniques are more effective than others for particular conditions. A major concession is that exposure-based procedures could be necessary for some types of anxiety disorders, although how exposure is performed might not matter. Both camps acknowledge that moderator factors, such as age, could be important [...] Other examples for which there is broad agreement for problem-specific effects include treatments for severe motor tics or cognitive deficits in schizophrenia.

There are also various methodological problems with running randomized controlled trials on therapy, which that same review discusses. So it’s not quite the case that the Dodo bird verdict would be the scientific consensus: rather its validity and scope is still an open question, and there is rough agreement that at least some interventions do work better on some cases than others.

Those aren’t the main reasons for why I have my own complex model of what’s going on, though.

My main reason is just that I’ve done various forms of therapy, meditation, and self-help, and they seem to clearly have different effects and work in different ways. Recognizing something as a cognitive distortion à la CBT, healing an exile à la IFS, or seeing subtle tanha à la insight practice are very different kinds of experiences, cognitively and emotionally. And while introspection isn’t a reliable guide to what’s happening in the brain, for me to believe in “these are all secretly the same” would be getting into “maybe there’s an evil demon messing with my mind and rewriting all of my experience according to its whims” territory.

When I got - after significant initial skepticism - into Internal Family Systems therapy and got training in it, I found it personally valuable and then started facilitating it to friends who were interested. Some it didn’t do much for, but some would say things like “this one session felt more useful than all my previous experiences with therapists combined”. That encouraged me to start doing IFS-based emotional coaching, where the results have been similarly varied.

Some of my clients get very little benefit from IFS, some quite a bit, and then occasionally there are ones who’ll show up for just a few sessions and find their life so drastically improved that they don’t feel a need to come back. I had one client who felt stuck in their life, had about three sessions with me, and then a year later e-mailed me to share a list of significant achievements and life improvements that had happened since then that they attributed to our work - when none of the previous therapists and coaches they’d worked with had felt helpful, and they’d previously felt skeptical of the possibility of ‘discrete step’ progress.

This kind of an outcome is definitely not the median outcome, but it’s also sufficiently common that I’m not surprised by it. And yes, there is regression to the mean and people do sometimes improve spontaneously… but it doesn’t really feel plausible to me that a client would have the experience of a breakthrough session and then just totally coincidentally happen to improve just then on something they’ve been struggling with for years. Maybe if I was seeing a hundred clients per year, you might expect something like that to happen by pure chance, but “some dozens” on a busy year is more accurate. And I’ve usually gotten a few of these per year.[3]

Notably, “some clients benefit a lot, some not at all, and the rest are in between” is entirely compatible with the Dodo bird verdict. If you take an average of the effect over all clients, you’d probably get something like “IFS works roughly the same as every other therapy”. (Indeed, the small number of randomized studies we do have on IFS say that it works roughly as well as whatever it is being compared with.) Standard parallel-group RCTs cannot estimate the extent to which any given person would have responded better to another treatment, because people are randomized into only one treatment arm.

The simplest explanation to me would be that for whatever reason, some interventions just work better on some people than others, in ways that are related to how their individual psychology works.

You might notice that I’m treating “specific interventions work best on specific issues” and “specific interventions work best on specific people” somewhat interchangeably. I started by suggesting that - say - IFS works best on particular kinds of problems, but now I’m talking about what kinds of people IFS works best for.

This is because I think these can’t always be cleanly distinguished. The review that I quoted above mentioned a finding that Acceptance and Commitment Therapy might be better than CBT for older adults with chronic pain, “perhaps because the therapeutic focus was more aligned with learning to live well with pain, rather than the treatment being expected to reduce the pain.” I didn’t want to cite this as evidence because the empirical evidence for it looked very weak at this stage, but let’s consider what it would mean if this was true.

The review called this a “moderator factor” for the general issue of chronic pain. But it seems slightly questionable to me whether the kind of permanent chronic pain that you might get from living to an old age should be considered the “same” issue as chronic pain you’re expected to eventually recover from! Your orientation to something you think might go away and something that you expect to just worsen is entirely different.

Likewise, people vary quite a bit in how they think and process the world. Some people have a constant internal monologue narrating everything that happens to them, and that have an “inner critic” in the form of a voice that is literally criticizing them and saying bad things about them. This is totally alien to me: unless I’m thinking about what to say in conversation or what to say in an essay like this one, most of my thinking is non-verbal and non-linguistic. My inner criticism doesn’t feel like anyone saying bad things about me, it just feels like I’m intrinsically bad.

Now suppose that I have a friend who does have an active inner monologue and a voice-like inner critic. If both me and my friend are feeling bad about something we’ve done, then on one level we’re having the same issue, but at the same time the form that the issue takes is entirely different. A therapeutic intervention along the lines of “ask the voice why it wants to criticize you so much” or “imagine the voice talking to you in a cartoon-like high pitch that makes it sound ridiculous” might be helpful for my friend, while making me just go “huh?”.

Now, this would imply that you could figure out which therapy works best for who, and then match clients to the therapies that work best for them. So far, studies that have attempted to do this have had disappointing results. But then, it’s unclear what good predictors would be.

If you asked me to give you an objectively measurable trait that could be used to predict whether someone benefits from IFS, I’d struggle. I sometimes tell people that “I tend to get the best results with people who’ve already tried something like Focusing or IFS and found it promising”. But that’s not really a predictor, it’s just saying that my method works best for the kinds of people who have already found it to work for them.[4]

Being intuitive and in touch with your emotions helps quite a bit, but I’d expect such people to do well in any form of therapy. You might also expect something like “finds the notion of the mind being divided into parts intuitive” to be a good predictor… but I initially kept bouncing off IFS because all descriptions of it sounded hopelessly woo and unscientific, while some of my clients totally buy the model intellectually but still struggle to connect with their parts.

So it seems to me that different people’s problems are configured in different ways, some therapies are better than others in solving a particular problem or configuration but none is universally best for everything, and it’s hard to predict in advance what would work for any given person.

Some of the therapies and approaches might also change one’s configuration so as to unlock new approaches. For example, IFS and many of the other experiential therapies are often not very effective for people who have difficulty feeling their emotions. Practicing Gendlin’s Focusing and doing body scan practices are approaches that can get people more in contact with their emotions. This can then make it easier to get benefits from other therapies.

Complexity of problems as source of hope

I noted before that many frameworks have their own ideas of what works and what needs to happen for things to get better. If you accept any one of these claims as truth, it can be strongly motivating - “I just need to heal my trauma to fix my problems” or “I just need to surround myself with enthusiastic people to fix my problems” - as long as you seem to be on track to doing that. But if it starts looking like you’re not making progress toward it for whatever reason - maybe you can’t find the original trauma, maybe you’re failing to find sufficiently enthusiastic people - then it is easy to start feeling hopeless. Alternatively, you might retain the excitement for the method, and keep grinding on something that isn’t actually a good approach for your problem.

But if there are many different interventions that could potentially work, then that’s a source of hope. If you have been grinding on one school’s approach for a while and making little progress, it’s still possible that another one will work! And while it might not solve your problems entirely, it will still make your life easier and improve your odds of tackling the remaining parts of it - or maybe just conclude that the problem is sufficiently under control that you can let it be and move on.

So why do so many people think their framework is “The One”?

With all of that out of the way, I can now finally discuss possible reasons for practitioners to think that their framework is The One True One. Now many of these might affect me as well - and have historically affected me, for there was a time when I thought everyone just needed to learn IFS and then all of our psychological problems would be fixed. Turns out that I still think it’s great and extremely beneficial for some people, but alas, it didn’t fix even all of my psychological problems.

Typical mind fallacy. It is a common experience for people to hear different kinds of advice, note that it doesn’t work for them, and then assume that the advice was bad rather than the advice being something that just didn’t work for them. Scott Alexander writes:

I only really discovered this in my last job as a school teacher. There’s a lot of data on teaching methods that students enjoy and learn from. I had some of these methods...inflicted...on me during my school days, and I had no intention of abusing my own students in the same way. And when I tried the sorts of really creative stuff I would have loved as a student...it fell completely flat. What ended up working? Something pretty close to the teaching methods I’d hated as a kid. Oh. Well. Now I know why people use them so much. And here I’d gone through life thinking my teachers were just inexplicably bad at what they did, never figuring out that I was just the odd outlier who couldn’t be reached by this sort of stuff.

If it is the case that different therapies work for different people, then someone may have gone through a number of therapies before finding one that clicked for them. Maybe CBT didn’t do anything for them but IFS did, or vice versa. Then they find a community of people who have all had a similar experience, end up mostly talking to each other, and conclude that “obviously IFS is the only thing that works while all the people promoting CBT are just inexplicably bad at doing therapy”.

The ontological “when all you have is a hammer” issue. These frameworks do more than just offer possible solutions. They also have their own ontologies about what the problem even is. Internal Family Systems sees everything in terms of parts of the mind; some meditative traditions see everything in terms of tanha; the biological frameworks might frame things in terms like inflammation, cortisol, blood sugar, and sleep debt. Since a lot of phenomena really are explained and well-modeled through these kinds of concepts, it’s possible to start seeing everything through a particular framework’s vocabulary, which makes it harder to conceive of any alternatives.

This is especially so as getting better at applying each framework tends to involve learning to see the world through its lens. Learning IFS has you seeing everything in the mind as parts, learning CBT has you recognizing cognitive distortions in all of your issues, vipassana might have you seeing tanha everywhere. Each of these operates on a somewhat different level of perception, so that focusing on one level may blur out the others.

Selection effects are a thing. It’s easy for various selection effects happen - for one, the practitioner has to market themselves somehow, and any kind of marketing will be appealing to some people and unappealing to others. People will also self-select - someone who thinks their problems are caused by trauma is likely to be looking for a trauma specialist. This might make it more likely for practitioners to get the kinds of clients their framework is good for.

Also, someone who thinks their problems are caused by trauma is also likely to believe in frameworks that emphasize the significance of trauma in particular. That means that the client is less likely to challenge the practitioner’s beliefs and more likely to actively affirm them. Logically, “your clients agree with you about your framework being valid” shouldn’t cause a practitioner to change their beliefs - the practitioner is supposed to be the expert, and they should be looking at results not pre-existing client beliefs. But practitioners are humans too, and humans tend to automatically believe more in anything that people in their social environment affirm.

Wishful thinking. “I know the one method that is key to solving everyone’s problems” can be a belief you are very motivated to hold - either for an egoistic or altruistic reason! The thought of being able to personally fix everyone’s problems can feel very satisfying to imagine. And if one is strongly motivated to help other people, they may want to believe that there exists a way to fix everyone’s problems and that nobody is beyond saving.

Many problems are multicausal. Maybe you have ADHD, trauma, and you don’t get enough exercise. Each of these is likely to hurt your emotion regulation, and the effect of the three may be closer to multiplicative than additive. So if someone believes that “you should just lift, bro”, even if the lack of lifting wasn’t the only core reason for your mood problems, following the advice may still produce a sufficient improvement to make one think that it was the core problem.

Multicausality may look monocausal in retrospect. I mentioned earlier that many practitioners may believe in a method because it turned out to be transformative for them. This may be true even if they actually needed a lot of other methods to get there.

Many people working through their issues will experiment with many different methods, each method allowing them to dissolve a part of the problem. Then they find some particular method that lets them solve the rest of it… and then attribute all of their success to that method, thinking that if they had just used that method all along, they wouldn’t have needed to waste all this time.

But all of those previous methods weren’t actually wasting time. They were the right methods for the stage the person was at previously, and the last method they found might even have been entirely ineffective at that point.[5]

Persistence often pays off... Some issues that are caused by e.g. trauma don’t seem obviously connected to anything traumatic at first. And some problems that seem obviously psychological may turn out to be physical with sufficient investigation. Some very deeply buried trauma or obscure medical condition may require a lot of looking into by expert practitioners before the underlying trauma or origin becomes apparent, and then one might get significant results.

This means that if someone has a strong frame of “everything is about trauma” or “everything is physiological”, they will keep pursuing that hypothesis at a point where someone holding broader views would already have switched to another hypothesis - which will be the correct thing to do in cases where someone’s problems are mostly caused by that, and wrong in cases where they aren’t.

The trauma specialist who keeps digging for a year before the buried trauma surfaces may be behaviorally identical to the trauma specialist who keeps digging for a year on someone whose problem was a thyroid condition. Whether that digging was correct or not depends on the base rate of trauma-caused depression versus thyroid-caused depression - which is exactly the thing that’s in dispute between the different practitioners.

…even if the results aren’t impressive. With enough digging, one might always be able to find some trauma that could be related to the issue, or some medical issue that could conceivably be making things worse. This may be true even when it’s not actually a particularly crucial part of the issue. So a person who has repeatedly had the experience of, say, “enough digging will surface an underlying trauma” may find their “everything is caused by trauma” model confirmed once they do manage to find one.

If this then fails to significantly shift the issue, when fixing many other traumas has significantly shifted people’s issues, the failure may be attributed to something like “this trauma must have other aspects we haven’t managed to uncover yet”. (Which may still sometimes be true and key to resolving the issue!)

Therapeutic progress is intrinsically noisy. Some clients will inevitably drop out or fail to make progress for reasons entirely unrelated to the validity of the framework. Maybe the client’s and practitioner’s personalities just didn’t happen to work well together, maybe the circumstances in the client’s life got a lot more stressful, maybe they just decided to focus on something else. It’s often the correct move to expect this to happen and to not trust the framework any less for it.

But this means that it becomes hard to distinguish between “all the clients who stuck with me long enough saw significant improvements because the method happened to work for them and made them persist” and “all the clients who stuck with me long enough saw significant improvements, but couldn’t stick to it because of factors outside my control”. This doesn’t necessarily even take the form of blaming the client - the practitioner can even put it down to something like “I wasn’t good enough at motivating them”.

Combined with the multicausality, the practitioner might even have witnessed that the clients who dropped out saw genuine progress. The “just lift, bro” guy might have seen someone’s emotional regulation make real progress because of the lifting - but failed to note e.g. the client’s trauma or ADHD making it too hard to persist with the weightlifting in the long term. So it will look to them like “if the client had just continued, they’d have felt much better”... which may be true, but is failing to notice the thing that prevented the client from “just continuing”.

Frameworks often have different values. Suppose that you are in a high-achieving career, which you are pursuing due to an internalized belief that you absolutely need to be high status in order to be loved. This leads to some distorted thinking, so you work with a CBT therapist to fix it. This leaves your underlying emotional compulsion to pursue status at the expense of everything else unchanged, but frees you to pursue it more effectively. It also makes it so that you are less likely to burn out, which would allow you to work on your deeper issue.

Is this a success?

Even if one was trying to rigorously measure the success of various frameworks with randomized controlled trials - and was able to control for all the confounders - the possible weaknesses of this approach will be invisible. No RCT measures a counterfactual like “would the person have been ultimately happier and more fulfilled if they had been allowed to burn out and then pursued an entirely different methodology”. Instead, it might just show that your mental health has improved and that you are achieving all of your goals better.

But other practitioners - maybe ones who had themselves burned out, been forced to do deeper work, and then found themselves grateful for that - might see this result as worse than doing nothing. (Note that I’m not claiming that pursuing a high-achieving career is always done out of unhealthy motivations. Just that it sometimes is.)

The most extreme disagreement might come from the Buddhist-leaning teachers. Some of them might hold that any life motivated by the belief that one needs to chase positive feelings and avoid negative ones is a prison and a delusion.

Meanwhile, some of their approaches might lead people to start preferring an existence of just meditating in a monastery for the rest of their lives… which the more engagement-focused practitioners might see as a failure to promote a genuinely flourishing life.

And as I noted before, value differences also covers things like differing tolerance for risk. The IFS approach of the “always proceed slowly, only work with material that you can witness from the outside” minimizes in-session discomfort and the risk of the client getting overwhelmed. A psychedelic intervention that throws the client right back into their original trauma to release it can be faster and deeper, while also being riskier and much more painful.

How to find the help you need?

Okay, so it’s nice to know what makes everyone think they have The One Solution. But you don’t just want to theoretically understand it, you have some issue that needs fixing, and you want to find a practitioner who’ll actually help you. What do you do?

Alas, I don’t have any clear-cut solution. But here are some thoughts:

• Most obviously, be willing to try different things.
• Also, be willing to try different practitioners! Even if the framework does make a difference, common factors and client-practitioner fit also matter enormously. I personally like IFS but I’ve also heard horror stories about bad IFS practitioners, and I’m sure there are lots of people who’d benefit from IFS who I just wouldn’t happen to click well with.
• Do a common-sense sweep to your issues. If you’re never getting any exercise and only eating frozen pizza, you don’t need to see an expert to know there are potential gains to be made. (That said, if awareness of never getting any exercise and only eating frozen pizza fills you with such shame that you can’t make any progress toward fixing this, do see a therapist about that and don’t feel like you have to fix the exercise and nutrition first.)
• If possible, start with cheap tests. Blood tests to exclude an obvious medical issue are lower cost than a year of trauma therapy.
• Ask the practitioner on their track record with similar problems, and when they would suggest you stop working with them. If they say this works for everything, be suspicious. Though a more caveated response like “this doesn’t work for everything, but I think letting go of tanha is the most valuable thing so I’m only focusing on that” shows self-awareness.
• Ask them about the last time when they did recommend someone stop working with them. If they cannot think of a single time, or have to pause for a long time to think of somebody, that suggests they do not have real off-ramps.
• In that regard, it’s worth being explicit about what your goals are and making sure yours agree with the practitioner’s.
• Do sometimes delegate things to the practitioner - if they say “I know you’re not feeling any progress right now but I honestly think we’re close” and they seem honest, probably give that a chance.
• There are different bottlenecks - if one approach fails to work for a while, it may work better later (e.g. some people find that CBT starts to work after trauma work; approaches focusing on bodily sensations may unlock other approaches bottlenecked on your ability to feel your body).
• If you have the time and energy for it, some things can be tried in parallel and may actually be synergistic. Combining antidepressants with psychotherapy is the classic one. Or if you have a lifelong aversion to exercise, you can see a doctor to see if there’s a contributing medical issue while also seeing a therapist about any emotional factors related to it. The downside of this is that if you try a combo and you feel better, you won’t necessarily know whether it was one or the other or both together that made the difference.

Finally, don’t forget that “I’m going to go through every possible therapy and spiritual practice until I find one that works to fix my problems” can also be a form of avoidance.

Gretta Duleba has an article about Relationship Known Issues. In software, a Known Issue is a problem that the developers know about but have no plans to fix - because it would be hard, because the issue doesn’t matter much, or because they just have more important things to do.

Relationships can also have issues. Maybe one partner snores so much the other can’t sleep, but the two would like to sleep in the same room anyway, and this is a source of ongoing pain. It then makes sense to pursue solutions, but if nothing works out, it might be best to just treat it as a Known Issue and live with it rather than start resenting the partner or the relationship for it.

One’s life can also have Known Issues. If nothing else, the inevitability of everyone’s death is a Known Issue. It’s good to look for solutions if it seems like solutions are available, but it can also be a relief to just give up on trying to find one.

Alas, it seems impossible to ever know for certain when you should give up on trying and when you should persist. “I will try everything you fix my issue” can be a defense, but so can “there’s no point in even trying anything”.

I’m treating that impossibility of knowing as a Known Issue with life.

1. ^

   Often misleadingly translated as “attachment” or “craving”.

2. ^

   Mulder, R., Murray, G., & Rucklidge, J. (2017). Common versus specific factors in psychotherapy: opening the black box. Lancet Psychiatry, 4(12), 953-962.

3. ^

   I can’t totally reject a pure common factors explanation on the basis of these examples alone, though. “This person had a breakthrough with me after trying several other therapists and coaches” could be driven by “my method worked particularly well for the person” or it could be driven by “personality-wise, I happened to be a better fit for them than anyone else they tried”. And e.g. the people who benefit the most from working with me also tend to be the ones I have the best rapport with, so I do certainly agree that the common factors are very important.

   I should also note that drastic improvement within a single session is not unique to IFS, but also happens in CBT as well as a range of other therapy modalities. My interpretation is that various modalities can happen to hit exactly the right spot for what the client needs, but a common factors proponent could also argue that this is a sign the modality doesn’t matter and it’s only about getting a good match with the therapist.

   To which I can only say that I see the argument, but having tried multiple modalities, that just doesn’t feel plausible to me.

4. ^

   The first rule of tautology club is the first rule of tautology club.

5. ^

   For something similar in a slightly different domain, see the Burrito Tutorial Fallacy.

Discuss

This is a post explaining my paper with Kaarel Hänni on complexity of infinite-width networks. I will explain the result, why it matters, and how the mathematical idealizations can interact with real structure in neural nets. This leads to some threads I am excited to pull on more in the future, via some new speculations on where interpretable structure can live.

Introduction

Our paper to some extent (and up to some important details) concludes an analysis of generalization complexity in Bayesian infinite-width neural nets. It resolves the question of when a function can be learned in polynomial sample complexity (in complexity theory, polynomial size is a fundamental asymptotic size class that means not too large. Informally, things outside the polynomial class are understood to be effectively unreachable/ uncomputable at realistic scales). More precisely, it asks whether (or when) generalization properties in infinite with neural nets reduces to generalization in finite width neural nets.

Here a crucial observation is that there are different infinite-width limits even within the same architecture, and they lead to different generalization behavior. The difference between these limits in Bayesian learning is in the choice of the learning prior (analogous to initialization or, more directly, to weight regularization in SGD learning) as a function of width. While seemingly a technical choice, this turns out to be extremely important for the infinite-width phenomenology. The relationship between initialization and generalization is familiar in usual SGD learning: for instance, the amusingly named omnigrok paper (Ziming Liu et al.) shows that modular addition generalizes better under small initialization (or strong weight regularization), and similar phenomena are observed in Bayesian learning where stronger regularization (corresponding to smaller initialization) tends to generalize better. For example the paper that explains Bayesian generalization in modular addition implies that changing the regularization is one of the levers that interpolates between the memorizing and the generalizing regimes.

In the infinite-width limit, we are inherently looking at a family of models that depend on a hidden-width dimension mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msup { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mspace { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c211D.TEX-A::before { padding: 0.683em 0.722em 0 0; content: "R"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c77::before { padding: 0.431em 0.722em 0.011em 0; content: "w"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c221E::before { padding: 0.442em 1em 0.011em 0; content: "\221E"; } mjx-c.mjx-c20D7.TEX-V::before { padding: 0.714em 0.5em 0 0; content: "\2192"; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c1D703.TEX-I::before { padding: 0.705em 0.469em 0.01em 0; content: "\3B8"; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c2223::before { padding: 0.75em 0.278em 0.249em 0; content: "\2223"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c1D716.TEX-I::before { padding: 0.431em 0.406em 0.011em 0; content: "\3F5"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c21A6::before { padding: 0.511em 1em 0.011em 0; content: "\21A6"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c226A::before { padding: 0.568em 1em 0.067em 0; content: "\226A"; } mjx-c.mjx-cB1::before { padding: 0.666em 0.778em 0 0; content: "\B1"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c1D442.TEX-I::before { padding: 0.704em 0.763em 0.022em 0; content: "O"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } (often called in theory papers), which is taken to infinity. There are two natural scalings for the regularization strength, related to different choices of natural matrix norm (see also Greg Yang et al.), where a regularization parameter scales either weakly, as or strongly as . The weakly regularized limit is called the standard limit and the strongly regularized limit is called the mean-field limit. There is also a natural interpolation between the two depending on a general real parameter , where one must be careful about differences in scaling between the early layers and the last layer to avoid certain degeneracies -- I will not get into this here[1].

Details on complexity and setup (hidden by default)

A few details here to fix concepts for people who want to track the rigorous results.

When we talk about polynomial complexity, we mean that we ask for bounds on this parameter which are some fixed polynomial in terms of the input dimension , as is usual in complexity theory of circuits. Note that the infinite-width setup always assumes that while the hidden layers are taken to infinity as the large number H, the input dimension stays fixed, as does the target or label function We will later consider another parameter which is the dataset size D; this is assumed much larger, and in the case of all boolean inputs the total datasize is exponential, . The fact that the data dimension d is understood as polynomial while the total possible data is exponential is crucial for learning theory.

We assume that our activation function is bounded. This assumption is relatively harmless: e.g. for any given Relu neural net, we can replace the Relu by a clipped Relu so long as we upper-bound at a sufficiently large value, and empirically using bounded activations (such as sigmoids) does not significantly affect performance.

Most importantly, we view the depth as a constant parameter (small compared to ) and our bounds are allowed to depend on nonpolynomially; more on this below.

The result

Once the technical dust settles, we have at least two different settings to theoretically investigate generalization behavior of overparameterized neural nets in the limit: the mean-field setting and the standard setting. Our paper implies, essentially, that a very wide overparameterized neural network generalizes poorly in the standard limit and generalizes "as well as possible" in the mean-field limit.

The first part of this statement is known (and we cite it instead of re-proving it). In the infinite limit of standard scaling, learning reduces to linear regression in a certain nonlinearly re-embedded data manifold associated with a so-called Gaussian process kernel. Crucially, the re-embedding depends on the training data but is independent of the labels , so informally all the label learning must occur in a single linear-regression layer. If the labels depend on some deep sequential structure where representations are transformed layer-to-layer (such as what happens in a Boolean circuit), then you're out of luck. More precisely the network can learn any function eventually with enough datapoints (by essentially solving linear regression in a suitably parameterized linear space of all functions), but this will take exponential data size D.

But the mean-field limit is known to develop deep, layerwise structure that depends on . This happens via a phenomenon called "feature learning" (this is a silly name, that refers to the fact that hidden layers are learning at all as opposed to the standard scaling where in the limit they are "frozen" to their prior configuration. It neither assumes nor implies any natural notion of features, other than the tautological features associated to the activations in hidden layers, or rather their relative configuration up to rotation matrices which is encoded by the activation dot product matrix).

The question our paper resolves is how much of the deep structure that can develop in a narrow network can survive in the infinite-width deep limit. And our answer is, informally, "all of it". Here I am of course being pithy. We don't really have a good notion of "structure", but if we did there would certainly be structures (mechanisms, circuits, features and so on) that behave differently at finite (polynomial) and infinite width. The thing we actually show is that if sufficient structure can be found in a large dataset to generalize well at some finite (more precisely polynomial) width in an underparameterized setting, then the same good generalization behavior survives to infinite width.

More precisely, one of our main results is the following. Fix a boolean[2] target (or label) function . Assume that when we train on this target at some polynomial width , the function generalizes after polynomially many data points (we will unpack the relevant notion of generalization below). Then we deduce that the infinite-width limiting model (and indeed, any sufficiently wide model) also generalizes after polynomially many data points.

Conversely, we show that if the infinite-width model generalizes after polynomially many data points then there is some polynomial width such that the width- model learns the target (also in polynomial data).[3]

Here our notion of generalization is worth unpacking. Given a Bayesian model (determined by the weight scaling, the width , and a choice of tempered Bayesian posterior), we define the generalization loss at data size as the expectation over tuples of boolean datapoints of the expected test (i.e. population) MSE loss of the trained model (i.e. a model sampled from the Bayesian posterior):

This value has a sensible extension to width We say that a model generalizes relative to a small error parameter at width and data points if Our main result is that if for some polynomial then we also have for some polynomial . And conversely we show that implies that there is some pair of polynomially bounded such that

In other words, we show that "learning polynomial data gives small error" is true at infinite width if and only if it is true at polynomial width.

Why think about infinite width

Before moving on to details and caveats (though note that there is a particularly big caveat associated to the asymptotics w.r.t. depth), let's step back and see why this is interesting. After all, no one wants to train infinite-width networks. It's also not true that one should deduce immediate insights here about empirical neural nets from this, for example that "more regularization is good for generalization always". First, empirical models are not "close" to the infinite-width limit, and even when infinite-width analyses are stretched down to realistic width, many new phenomena emerge (some kinds of emergent regularization, certain instanton phenomena) and intuitions don't directly carry over. Second, empirical models are trained dynamically (via SGD or Adam or similar) and Bayesian learning results must be understood with a grain of salt.

Instead, I think that there are two main reasons why our result is interesting:

1. They imply a width-independent notion of heuristic bias, which may be more elegant and natural to study, and which might help give cleaner models of the relationship between data, parameter size, and generalization.
2. Our results suggest that the theoretically simpler circuits and mechanisms found at infinite width are sufficient to capture any regularities that are captured in finite-width neural nets. Various "natural parsimony" heuristics (like natural latents) suggest that two different predictive models of the same data are likely to find similar patterns and circuits, and thus interpreting the infinite-width model may capture some of the same mathematical structures used by a finite-width learner.

I find the promise of item 2 particularly exciting here. Infinite-width limits are physical objects: they can be described by a certain set of integral equations on fields, which are associated to training data points (rather than weights). While the resulting object has many parameters (called observables), it gives a new perspective that naturally incorporates tools from statistical field theory of many-particle systems. In particular, the ability to control sample complexity opens up a new perspective related to scale and renormalization. The key upshot of our results is that a simple training objective leads to a simple statistical description, in a way that is independent of width. We discuss this idea and a few speculative implications in the following sections.

Small tasks have small complexity

Throughout this discussion, we are carrying around two numbers which can be relatively big. The first is the training data size and the second is the width . Generalization on a task implies that, informally, the data size never needs to be taken to infinity: there is some finite amount of data (only dependent on the task function ) which is sufficient to fully learn it at least to some finite desired level of population accuracy (a.k.a. "test" accuracy). A priori this "characteristic" data count D is a function of the hidden width, . Our result implies that in rich scaling there is some more fundamental data size such that the target is already learned after seeing random datapoints at any width (here and elsewhere I am sweeping under the rug some discussion of coarseness, related to the parameter and the temperature of tempered Bayesian learning).

The upshot is that (modulo some discussions of coarseness and error bounds), the function learned after seeing any large data fraction is equivalent to the function learned after seeing data. If the target is simple and the width is large, we have , and this puts us solidly in the mean field regime. Heuristically, this means that if we were to train a giant multi-billion parameter model[4] on some simple task like induction learning, and the training is in a Bayesian regime with mean-field regularization, we expect the model to be well-described by a mean-field prediction. This means that a single finite physics-flavored description simultaneously predicts the behavior at any width.

Parallel (or superposed) tasks have emergent regularization

Now it's likely that for a very complex task like language prediction, the effective data complexity is very large if we want perfect performance, or at least very large compared to the parameter count of modern llms. But if we only care about explaining a very small amount of performance, or a simple component of the model, its effective data requirement may be small. Famously it is often reasonable to think of neural nets as not so much a single complex algorithm as a set of interlocking but somewhat independent algorithms: informally, a bag of heuristics. Some components or subproblems may be easier while others may be harder, and there are likely various ladders of complexity associated to a precision parameter or some other notion of scale. For example in some coarse decomposition, an LLM has pieces that are (at least early in training) just following bigram patterns or induction circuits, or offsetting days of the week via modular addition.

A highly simplified model here is to view the subproblems as existing in parallel, or in superposition. If a single component is simple enough and we learn just that component in a mean field-regularized fashion, the resulting physics should fully predict the weight distribution as explained in the previous section. However we cannot deduce that the collection of parallel circuits are given by some simple combination of the resulting mean fields. When a neural network performs multiple tasks in parallel or in superposition, these tasks are forced to share resources such as weight norm and entropy. But if we're lucky we can predict the tasks as effectively executing some set of superposed infinite-width limits once we incorporate the shared resources via emergent regularization. My notion of a shared resource is intentionally left vague, but an example of a problem where this can be fully worked out is the sparse denoising model here. A particular shared resource that must always be tracked is weight norm: since different parallel circuits must share weight norm, we expect a tuple of many circuits to mutually regularize, leading to an emergent regularization phenomenon where the effective regularization felt by any particular small circuit is much larger than what it would experience if it were the only thing being learned. This leads to a situation where, for example, under standard scaling a superposition of many circuits behaves as if each circuit were undergoing mean field scaling by itself.

Extensions and limitations

It should be clear that the above discussion leads to a particular orientation on neural net interpretability. One can conjecture that a large neural net decomposes into small simple circuits, each of which has low complexity (low effective width-independent complexity ) and sees the empirical width as effectively infinite. They interact and regularize each other via shared resources and hierarchical relationships, but are otherwise independent and simple. After accounting for emergent regularization, some circuits behave as if they are in the mean field regime while others behave as if they are in the standard (i.e. Gaussian process) regime. I really like this as a cartoon and a thread to pull on. This picture is related to my excitement about variants of the renormalization agenda in the org I work in (Principles of Intelligence).

At the same time I don't expect this to be the full story, for several reasons.

First, the separation of phenomena into mean field and Gaussian is a false dichotomy, and it is possible to come up with nice examples where both mean-field and Gaussian process phenomena interact in a significant way and must be understood together to describe the system. A complete analysis of the sparse denoising theory (not yet published) should be such a case. A general study of such phenomena, given by a 2-particle mean field theory, is related to past and upcoming work by my colleague Nischal Mainali. A related phenomenon is explained in the "Proof idea" section below.

Second, neural nets learn dynamically (SGD and so on), whereas the mean field limits are cleanest in Bayesian learning. This is not as bad as it might seem, since much of the mean-field analysis of neural nets applies also to SGD (and is believed to apply to other gradient-based optimizers like Adam, etc.). In fact, Kaarel and I are also discussing a version of the paper that would apply for dynamical learning, with additional caveats and details.

Third, mean field-style analyses may break down when studying underparameterized (or more generally "not-very-wide") networks via a neuron specialization degeneracy. Here instead of interpolating between different simple circuits in a superposition-like way, neurons may decide to specialize to a single circuit, or more precisely a circuit may depend on a single neuron or a small set of neurons in a robust mechanistic way without redundancy; this would break a fundamental assumption necessary for the standard infinite-width techniques. This phenomenon and how it fits into the various infinite-width limits is discussed in particular in the paper "Mitigating the Curse of Detail" by Noa Rubin et al. In fact, there is much evidence that this kind of monosemantic behavior does occur in some circuits (a bunch of classical examples in vision models are provided by Chris Olah's interactive account here). Phenomenology of this kind is genuinely new relative to the thermodynamic phenomena that occur in infinite-width limits, and is more or less directly analogous to instanton symmetry-breaking effects. Exact predictions for such systems would likely require a separate analysis similar to work of Saxe on symmetry breaking (see for example Saxe-McClelland-Ganguli), or the analysis of symmetry-broken instanton phenomena in physics.

For this reason and others, I don't expect a complete theory of interpretability of finite networks which is reducible to "mean field theory plus inductive regularization phenomena". However I am very hopeful that some of the simpler and more polysemantic mechanisms in large neural nets (such as induction heads) can in fact be explained in this way, or at least use the same mathematical mechanisms as mean field learners. Optimistic signals are that this is true for modular addition (where both narrow SGD and infinite-width mean-field models learn the same "clock" Fourier-basis mechanism). We also see a deep-network example of this (mechanistic agreement between narrow and mean field solutions) in an example of an SGD-learnable deep boolean XOR-like task in upcoming work joint with Sergey Alexeev (whose MARS project I am mentoring).

MFT and neuron independence

A second aside before moving on: often when I talk to people about mean field theory, they assume that this is a simplistic model that postulates independence between neurons, and hence cannot capture phenomena that require collaboration between many neurons. Under such a model it seems strange to them, and perhaps even un-believable, that a mean field formalism can capture the full complexity of algorithms learned by a neural net.

I think that this misconception makes sense, and is important enough to devote a section to.

The simple answer is that while the mean field limit does imply a certain kind of neuron independence, the independence is conditional, on a relatively complex self-organizing "macroscopic" structure which (as we show in our paper) is powerful enough to encode algorithms of arbitrary complexity.[5]

Informally, a first-order way to think of mean field theory is that neurons have jobs. More precisely in a mean field cartoon, each of ~infinitely many neuron indices at a given hidden layer is labeled by a "job" latent from some finite set. The weight between a pair of neurons at adjacent layers is then drawn as a (statistical) variable that depends only on the two jobs, and not the associated indices. Thus each neuron's preactivation is a weighted linear combination, associated to its job, of activations from job postactivations of neurons at the previous layer. The key point (and the key reason why mean field techniques are interesting for interpretability) is that the number of jobs can be large, but is in some sense finite, depending on the number of training data points; while the number of neurons is assumed to be infinite and indistinguishable. Thus the onus of interpretability is moved away from neurons and into fields, which I called jobs. Since the number of possible fields scales with the number of training data points, this formalism is able to represent very complex mechanisms where neurons at different layers collaborate. For example if a neural net is learning a finite boolean circuit, its jobs at any given layer can be associated to the finitely many hidden gates of the circuit (and this is heuristically what we observe for XOR), with each neuron type reading the two neuron types associated with its parents in the circuit. The full formalism of mean field has its interacting fields indexed not in terms of informal "job labels" but in terms of activation kernels, and a rough translation between these two points of view is a central intuition pump for our paper.

Crucially, the dimensionality of the field theory (the "maximal number of jobs") is related to the training data dimension. As I explained above, our results imply that structure saturates at some polynomial data, independent of width, and in the large-width regime it is this value (rather than the parameter count) that encodes the complexity (the dimension of "job space"). This finite set of parameters defines the landscape of possible neuron jobs and their interactions. Neurons sampling this space of jobs are now conditionally independent, but interact in structured ways mediated by this parameter. This implies the existence of a certain platonic, width-independent mechanism for implementing any task that can be learned by a neural network. Of course the platonicity here, as in most cases, is elusive: different choices of activation function, re-weightings of layers, and of course questions of dynamics/SGD can lead to a much more hyperparameter-sensitive picture. However the upshot is that nevertheless the model reduces to a more streamlined abstraction controlled by physics - I think this is exciting.

Details and caveats

I'll explain an intuition pump for our proof in the section below, but first here I will point out the assumptions we make on the model, some of which are cosmetic and others which are significant.

The main cosmetic assumptions we make (which we think can be replaced by weaker assumptions) are:

• that the activation function is bounded (e.g. a sigmoid or a clipped Relu)
• that the data distribution (on training and test data ) is the uniform distribution on the boolean cube (i.e. coordinates drawn uniformly from ).

The bounded-activation assumption is there to avoid allowing neuron specialization in the infinite-width limit, and the finite data distribution on the cube is useful (more fundamentally) to keep the data bounded and, less fundamentally, to work with a finite total data distribution of size --- large enough that it's "huge" (larger than any polynomial) in the dimension parameter but still smaller than the width parameters approaching infinite-width limits, which are assumed to be on an ever larger super-exponential scale.

The most significant issue in our paper is that the various bounds in our paper scale very poorly in depth (we use the letter to denote depth, as is standard). While at a fixed depth the bounds are polynomial (so generalization complexity at some finite width implies generalization complexity at infinite width, with a fixed polynomial independent of the target ), the dependence on of the bounds is exponential; so the more precise statements have bounds that look like

where is a fixed degree- polynomial that depends only on the activation function. This can be practical for small (like 2 or 3), and in fact already gives quasiexponential improvements over standard scaling for boolean circuits (like XOR) at depth (for the data dimension/ number of bits). But for deep networks, our result as it stands leads to horrible bounds with only theoretical interest. While we can console ourselves that this is a theory paper, we think this is not the best one can do.

The state of our rigorous mathematical knowledge currently has this ugly factor of in the exponent that makes our current bounds extremely sensitive to depth. However, we believe that under an additional Lipschitzness condition (related to this work by Linsefors and Bushnaq) we can make the polynomial dependences depth-independent and we are actively working on an extension in this direction. This argument requires datapoints to be distinct (like the boolean cube we work with, or text strings) and effectively forces the deep neural nets that are "allowed" in our distribution to perform some denoising/ error correction in data-space basins around the datapoints. Interestingly, this kind of behavior is in fact observed in practice, most remarkably and carefully in a series of posts by Stefan Heimersheim and his students Francisco Ferreira da Silva and Matthew Shinkle (see for example this post). Thus on the level of heuristics and ideation, I expect our results to be robust, at least to some extent and under some idealization, to the large-width limits of empirical deep networks. These empirical stability phenomena make me hopeful that some of the mean field-inspired ideas sketched out in the previous sections apply to deep neural nets, despite the fact that our formal Bayesian guarantees have a horrible depth instability that give unusable empirical complexity bounds for deep networks.

Proof idea

Our result works, to first order, by essentially formalizing the idea of "jobs" in the section on neuron independence. This is easiest to frame in a certain richer version of mean-field (where we move an exponential parameter that interpolates between standard scaling and mean-field scaling even further away from standard, corresponding to even stronger regularization, and consider the limit ). Here the infinite limit corresponds to finding a minimal L2 norm neural net solving the task (where the total L2 norm is an appropriate weighted sum over layers). Given a network with small norm at finite width, we can "blow it up" to a network with the same norm, at infinite width by a process we call cloning. This works differently at middle layers vs. embedding/unembedding layers. The middle layer story is representative (and more interesting). It works by taking a narrow middle-layer matrix and simply writing out a block matrix consisting of identical "clones" of in an square, with a prefactor of :

\[W^\ell_{\text{cloned }\times N} = \frac{1}{N}\begin{pmatrix}

W^\ell&\dots &W^\ell\\vdots&\ddots & \vdots \W^\ell&\ldots &W^\ell

\end{pmatrix}.\]

Taking gives us an infinite-width "limiting structure". You can observe that the large matrix has the same norm (sometimes called "Frobenius norm") as the original matrix. You can also see that this limiting structure leads to the same input-output behavior as the original neural net. If the activations at layer for some datapoint are given by the vertical vector then we see by layer-to-layer induction that the activations at layer are again a repeated version: repeated times. We now see that the next-layer preactivation is again the concatenation , repeated times. Since the activation function is applied recursively, this completes the induction. Conversely, given a large neural net with hidden dimension we can stochastically reduce it to a smaller net of size by a process we call subsampling: simply remove at random a collection of neurons, and any weight coefficients that abut them, leaving a neural net of hidden width . Here (carefully eliminating outlier neurons) we can guarantee, using statistical concentration bounds, that the resulting net has norm and input-output behavior that are close to the original network, with error bounded by a width-independent (but target-dependent) constant times . Thus by paying some finite precision cost (corresponding to the difference between the accuracy assumption and the accuracy infinite-width guarantee above), we can show that the resulting narrow subsampled net solves the task with admissible error. These arguments imply that if a small-norm neural net that performs well on a task exists at some polynomial-size width, then such a net also exists at infinite width, and conversely good performance at infinite width implies the existence of a performant net at finite widths. A priori these guarantees of existence do not imply anything about generalization, and the generalization result requires some additional learning-theoretic (PAC-Bayes flavored) arguments (the main idea is that if a function has good expected performance on a random dataset that is much larger than its description length, then by simple entropy/ counting arguments it must generalize well; though deducing infinite-width generalization results involves checking that these arguments are in some strong sense robust to cloning and subsampling).

While the above sketch can be converted to a proof in the rich limit (where Bayesian learning reduces to minimizing the weight norm of a successful learner), some additional arguments are needed in the case (whereas infinite-width learning at collapses to Gaussian process learning, equivalent to standard scaling).

For the mean-field regime we need to track entropy in addition to minimizing weight norm. The hardest case turns out to be exact mean field , where the entropy contribution to free energy scales (as ) at exactly the same rate as the weight norm's contribution. The resulting math is much more involved, and the behavior that we end up seeing is an interesting mix of weight norm-minimizing phenomena in big or "relevant" data directions vs. macroscopic isotropic noise phenomena (analogous to standard scaling/ kernel learning) in small or "irrelevant" data directions, with the cutoff set by the accuracy scale . This is an interesting new structure somewhat related to Nischal's 2-particle picture above.

1. ^

   A somewhat technical footnote: the linked paper works out the most interesting, "critical" mean field regime called the Mei-Montanari regime, where the proof becomes significantly more involved. Here we sketch the results and the proof focusing on a simpler case of "ultra-rich" mean field scaling 1" role="math">, which is worked out in an upcoming companion paper. The results in the two cases are roughly the same, with a couple of differences we point out along the way in footnotes. The takeaway for readers without extensive context is that as we vary between different mean field regimes associated to different choices of a scaling parameter, one choice requires more work than others but all ultimately have the same complexity properties.

2. ^

   The Booleanness here is a cosmetic assumption made for technical ease, and can be weakened.

3. ^

   A technical footnote: here we are giving the weaker result that applies both at critical and at rich mean field scaling (see previous footnote about the scaling exponent ). In the case of rich scaling our result applies to any initial width , not necessarily polynomial, and moreover we deduce polynomial data generalization at all widths above some polynomial width . We expect the same stronger result to hold in the critical regime, but it needs a technical strengthening of a physics-flavored lemma which we currently do not know how to prove.

4. ^

   Here one should use bounded activations and sufficiently large error bounds (i.e., high temperature), and be careful with depth instability; this is explained more below.

5. ^

   Here I develop a simplified economics-inspired picture of such structure similar to the one in this post with Lauren Greenspan. The formal picture with concentrating Gram-matrix fields is explained in various sources, for example these lecture notes. I hope to give a simpler sketch of the math in future posts with Lauren (and Nischal is also working on some excellent notes related to this).

Discuss

My Introduction.

This is a review of @Gordon Seidoh Worley's book on epistemology.

I'm neither an extreme sceptic, nor an extreme realist, and for that reason, I'm going to push in one direction some of the time, and in the other in other places.

Gordon's Thesis

Our knowledge of the truth is fundamentally uncertain because of epistemic circularity caused by the Problem of the Criterion.

More than that..for instance, semantic vagueness has nothing to do with the problem of the criterion. Uncertainty can creep into the map, (eg semantic confusion); the territory , (eg. quantum uncertainty); and the the relationship between them , (eg. the problem of the criterion). But the situation is not hopeless because these problems affect our knowledge making efforts to different extents.

We manage fundamental uncertainty by making pragmatic assumptions that lead us to believe in the truth of claims that help us achieve our goals.

Consequently, the truth that can be known is not independent of us, but rather dependent on that for which we care.

Ok, maybe, but one needs to avoid the basic irrationalism of thinking one can make things true just by believing them.

If you maintain truth and usefulness as separate concepts, and if you stick to a correspondence theory of truth , then it is quite difficult to show that truth depends on what we care about. It is more than usually uncertain claim.

That truth is fundamentally uncertain and grounded in care has far-reaching implications for many of the world's hardest-to-solve problems.

It's also important that uncertainty is very unevenly distributed.

And that attitudes about it are very tribal.

If you belong to the continental/postmodern , you are (apart from being unlikely to be reading this) probably leaning too far in the direction of fundamental uncertainty and modest epistemology.

If you are a rationalist or scientism-ist you are probably not leaning far enough, and for you it will be more therapeutic.

If you are a rationalist or scientism-ist , you are likely to protest that we obviously do have knowledge, because we can demonstrably predict things and achieve practical results. And you'd be right, up to a point.. but only up to a point , because there are different kinds of truth and knowledge that suffer from fundamental uncertainty to differing extents.

If you are a theist or mystic , you are likely to believe in quite different kinds of epistemic justification to scientists and philosophers, which is another if the things ,coat from overall uncertainty , that makes it hard for people to agree. There's a qualitative problem and well as a way quantitative one

It's hard to be therapeutic to all the people all the time. But "there is fundamental uncertainty" needs to be balanced out with "but not to the same extent everywhere". (Fortunately, since strong all embracing scepticism is self defeating).

Chapter 1. How do we know what knowledge is?

But naive epistemology breaks down when there's disagreement"

Even more so when it's disagreement about epistemology!

Circular dependencies stymies reductionism, which may well explain why philosophy makes slow progress compared to science.

"Just use reductionism" doesn't work in philosophy , because its not clear what lies at the bottom of the stack, or if anything does. Logic, epistemology and ontology have all been held to be First Philosophy at different times. Logic, epistemology and ontology also seem to interact. Correct ontology depends on correct epistemology..but what minds are capable of knowing depends on ontology. Logic possibly depends on ontology too, since quantum mechanics arguable challenges traditional bivalent logic. One of the reasons philosophy is difficult is that it has a circular structure.

Well, how would we prove that logic and direct observation are true means of assessing truth? We'd have to show they're true using the self-same means of assessing truth: logic and direct observation. This sets up an infinite regress of justifications, like two mirrors endlessly reflecting back on each other. The only way we can avoid this endless loop of self-referential logic is by making one or more assumptions without justification and then using those assumptions to ground the rest of our reasoning.

On the bright side, criteria dont have to be grounds or foundational -- one doesn't necessarily need even more foundational assumptions to ground ones foundational assumptions. This where the pragmatic twist comes in useful. There's an argument that treating empirical evidence as probably true allows one to make progress , whilst refusing to blocks progress. Meaning that empiricism doesn't give you necessary truth, but does give you something to work on.

Observation and deduction are famously plausible sources if truth -- the two dogmata -- but they are not sufficient to t to do science .. at least realistic science, science that goes beyond making predictions. Deduction tells you what observations imply, but not what is causing them .. the ontology, the noumenal reality , the underlying mechanism, the thing that scientific realism adds to scientific instrumentalism.

Induction doesn't give you much of an explanatory hypothesis, beyond the simple claim that there is some sort of causal stability that makes the same observed patterns likely to repeat.

To get at the ontology, the noumenal reality , the underlying mechanism, you conjecture as well. That's a problem, because conjecture is much less truth-apt, in itself, than deduction and observation. Conjecture in itself is pretty much making things up. In the context if science, conjectures can at least be tested. In the hypothetico-deductive model, the scientist 1. Conjectures a mechanism, 2. deduces expected results from it, and 3. tests for the presence expected results and absence of unexpected ones.

So conjecture in the context of science not just making things up ... but it's not a honing in on a singular truth either. In particular , the method of conjecture-deduction-and-testing can leave you with more than one theory that has passed the empirical tests.

Thats one of the reasons "just use science to solve everything" doesn't work. Another is that some things just aren't empirical at all. Not that they are ghostly, but you can't see things like meaning and goodness.

Also, it hasn't worked. People keep reinventing This Sort of Thing, the idea of using science and logic to solve everything every few decades, and it always fizzles out.

Mechanised epistemology doesn't help at all -- because some sort of criterion, or fundamentally assumption about how knowledge works, has to be built into the machine.

Bayesianism has both problems, simultaneously. It is dependent on evidence, so it has all the problems of empiricism; and it has the problems of rationalism because it starts with priors. The argument for Bayesianism is that even agents with wildly differing priors can eventually agree, given sufficient evidence. But that is an argument about ideal Bayesians: in reality , the amount of evidence available might be too limited to allow convergence. The ability of realistic Bayesians to formulate hypotheses is also limited. Alice puts most of her credence on the one hypothesis that seems best supported to her, out of the hypotheses she has heard of, or thought up, but Bob might have a better hypothesis that's not in her set.

Do ordinary people need to understand the scientific method?

The epistemological problems that ordinary people face are to do with finding the causes or interpretation of a fairly fixed and known set of facts. People are currently trying to understand world events, the set of facts, in terms of conspiracy theories about behind-the-scenes actors. Exhortions about empiricism aren't much use to them. Ordinary deductive logic isn't , either, because what they are trying to do is abduction. What would be of more use is knowing about confirmation bias, driving too much, falsifiability ... and that not everything happens for a reason.

The first thing I'll ask is that you always remember, on every page, that everything adds up to normality. What do I mean? It's a phrase I've borrowed from science fiction author Greg Egan, and it means that things continue to be just as they were even when you come to have a new understanding of them"

Why do people keep repeating that? It's not the case that "everything* is the same when you switch to a different explanation. It is the case that everything looks the same when you switch to an empirically acceptable explanation, because that's what empirical acceptability is -- what is observed is predicted by them, and what is not observed is not. But that doesn't happen automatically, it's not given to us free by the universe, it's the result of our epistemic processes -- we reject explanations that don't "save appearances". And appearances, observations, empirical data aren't "everything", either. It's perfectly possible for beliefs about ethics or decision theory to rest on on beliefs about reality.

Chapter 2. How do we know what words mean?

In Chapter 2 we saw that all definitions are ultimately grounded in inference from experience, and this makes it hard to precisely define what words mean

We initially learn language from examples and usage, but we progress to bring able to read dictionary definitions. We can also coin jargon terms that are clearer than natural language, so that semantics is one of the more fixable issues.

Chapter 3. Why can't we all agree on what's good?

The question could have been cast in terms of what's virtuous or whats right. Casting it in terms of the good biases the discussion in terms of consequentialism. Having said that, the answer is:-

1. The virtuous/good/right are not clearly visible, so raw empiricism is of no use.

2. Pure logic and maths don't answer any ethical questions because standard formalisations don't have any ethical axioms. Axioms can be added -- there is a logico-mathematical approach to ethics -- but it isn't clear which ones, eg. the Kantian or Millsian.

3. Ethics at least potentially rests on ontology, so it's as least as uncertain, and ontoloy is a lot more uncertain than prediction.

4. Individuals aren't starting from a blank slate , they are starting from what their culture tells them , which is very variable. (No, there's no single coherent "human value").

The fact that moral agreement is difficult isn't itself mysterious, so long as you don't hold to naive Scientism. Naive Scientism tells you "everything" is soluble by science...so ethics is as well." But science doesn't claim to be able to pronounce on matters of value ... so naive Scientism goes beyond science.

So not everything is equally uncertain, again.

Many have tried to provide clear intensional definitions of "good" and "bad", but, alas, those definitions have proven insufficient. Either they're circular, like dictionary definitions that say "good" means "morally right" and "bad" means "morally wrong", or specific in ways that many people disagree with."

Yes, there are two problems there: the problem of providing a non vacuous definition, and the problem of providing a convincing theory. The second problem is much harder.

It would take multiple books to explain everything there is to know about Bayesians"

But it wouldn't do the average person much good at all , because Bayes isn't a complete epistemology. In over a century , Bayes has not achieved more than a niche role in science. In a couple of decades , Yudowsky's Bayesian rationality hasn't solved any notable problems..

Similarly Aumann's Agreement Theorem, has no relevance to real life.

In the same way, we are able to agree about morals to the extent that our notions of "good" and "bad" are compatible enough to allow agreement, and we disagree to the extent they are in fundamental conflict. That we sometimes agree and sometimes don't shapes the landscape of our moral conflicts and determines much about how we try to resolve them."

Or you can agree about meta ethics, and derive your object level ethics from that.

Ethical persuasion happens. It often happens by persuading someone of an abstract principle, and then showing that they should have different object level ethics based on the principle. For instance, you can persuade someone out of sexism on the basis of racism.

Ethical persuasion is an easier problem than ethical convergence

Chapter 4. Why don't we always do what we should?

Grace could restrict herself to evaluating only systems sufficiently less powerful than her current one such that she could safely prove whether or not they're sound. She won't bother to do this, though, because less powerful systems won't help her achieve her goal. Her whole purpose in evaluating formal systems is to find more powerful ones that will make her better at Bayesian reasoning. A less powerful formal system that places additional restrictions on her thinking would be antihelpful. She thus finds herself up against the Löbian obstacle: she can't prove the soundness of a formal system that's equally or more capable than her current system, and any system she can prove to be sound will be weaker and thus less useful for helping her achieve her goal.

This generalises beyond formal systems into informal epistemology. Simply abandoning certainty doesnt solve epistemology, because it doesn't tell you what to do instead. Simply adopting probability doesn't solve all the problems, because probabilistic inference still relies on axioms, and probabilistic empiricism still has a instrumentalism realism divide.

The various approaches that are left in the absence of complete certainty about everything can be seen as abandoning different epistemological desiderata.

Relativism means giving up on a single truth.(As does coherentism)

Pragmatism means giving up in truth in favour of usefulness.

Skepticism means giving up on truth and knowledge.

What are the Desiderata?

Certainty. What is true is necessarily true. A huge issue in early modern philosophy which has been largely abandoned in contemporary Completeness. Everything is either true or false, nothing is neither.

Consistency. Nothing is both true and false..

Convergence. Whatever various assumptions a group of individuals start from, they will end with the same conclusions, given enough evidence and reflection. (Convergence implies non arbitrariness).

But does this explanation really answer Doug's questions? The proofs that establish Bayesian idealness are mathematical arguments. Therefore, to trust that Bayes' Theorem is the correct way to update beliefs, we must also trust that mathematical proofs correctly establish what's true"

They dont, because they depend on axioms. The mathmatical , ie. axiomatic, argument for Bayes rests on the correctness of the axioms. But pragmatic proof, such as betting arguments , is available as well.

Chapter 6. How do we know what we know?

More true our map is. This method works remarkably well, and it's how scientists, engineers, and many others build accurate models of the world that enable them to do their work. And although not necessarily framed as testing, this is also essentially what Bayesians are doing when they update their beliefs: they're checking their map to see if it's accurate, then updating it if they find a mismatch with what they observe.

They are checking their map to see if it's predictive. They can not make direct comparison to reality, any more than anyone else. Updating us better than nothing, but it's not everything.

But how do we know that such testing is a reliable way to verify the truth, and can we be certain that our methods of testing are valid? Alas, we don't and we can't, or at least not with complete certainty. This time, it's the Problem of the Criterion that stands in our way, and, as we'll see, it's the reason that all our knowledge of the truth is fundamentally uncertain."

The Problem of the Criterion is straightforward to state. To know if a claim is true, we need a method for testing if it's true. Suppose we have such a method, known as the criterion of truth. Is the criterion of truth valid"

There are different kinds of knowledge and their confirmation has different difficulty levels. The easiest is pragmatic knowledge. It's almost analytical that if you can do something, you know how to do it. (In a way that's quite different to hoping that predictive accuracy somehow adds up to correspondence-to-reality.) So perhaps the pragmatic approach is the best starting point for understanding how we do know as much as we know -- how it does add up to normality.

Is it circular to use reliability and predictive ability as evidence of themselves? More to the point, is it circular in a bad way?

Circularity is often condemned our of hand, but why? In the context of deductive reasoning circularity implies quodlibet, the ability to prove anything. There are infinite arguments of the form "P, therefore P". Moreover , it would be possible to prove infinitely many contradictions by setting Q=not P.

That's in the context of deduction. The circular justification of induction doesn't have the problem of quodlibet, because when you are doing induction you are not starting from an arbitrary claim, you are starting from evidence.

My conclusion

As with many philosophical problems, the hard problem of epistemology is to find a single theory that fully meets a set of desiderata. Philosophers have essentially discovered that this is impossible. In further bad news, it isn't clear which partial approach is the best. In better news, a partial solution is possible , and better than no solution. Strong scepticism is self defeating, but moderate scepticism, which is the same thing as modest epistemology, is available.

If epistemology is doubtful, everything in philosophy is doubtful. If there are reasons to think epistemology is unsolvable, there are reasons to think philosophy is. The problems with epistemology are particularly acute because it takes an epistemology to decide an epistemological question ... there's no neutral ground to start from.

Even if metaphilosophy can't tell you witch epistemology to switch to, it can tell you to something, which is to switch to modesty, if you haven't already.

Discuss

Hi, I write a fair bit on my blog and usually post to hackernews, where some of my works have been well received. I thought this one would also fit well on lesswrong, so posting for the first time here.

Discuss

Anthropic concluded in the April Mythos Preview alignment risk update that the model "does not possess any unknown propensities that would increase alignment risk." The report argues that if Mythos Preview were coherently misaligned[1][2], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”[3]).

While I agree with the report on the above bottom-line conclusions (substantially on priors)[4], I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular, the report often uses weak evidence to justify reliability.

1. The report gives fairly weak experimental evidence for Mythos Preview having insufficient capabilities to evade monitoring. The model is plausibly often eval-aware and underelicited in the relevant capability evaluations. So, it might silently sandbag if coherently misaligned, or unintentionally underperform if otherwise misaligned.
   1. This limitation is important: one could argue that lack of covert capabilities for sophisticated sabotage (a subset of the capabilities I discuss here) is the single most load bearing argument in alignment risk reports.
   2. Authors of the report could have made calibrated guesses about Mythos Preview’s covert capabilities, especially for covert sabotage, based on other factors despite the relatively weak experimental evidence they had. But the report underemphasizes these factors and it’s unclear how well-justified their true reasoning was.
2. Auditing games cited in the report are not clearly representative of the actual assessment and failed to catch a Mythos-level model organism.
3. The report does not explicitly acknowledge that success in alignment and alignment assessments might be less independent than one might hope: if Mythos Preview were misaligned at the end of training, we should expect assessments to be less reliable than other evidence suggests.

These problems also affect (often more seriously) alignment assessments by other developers.

Alignment assessments will need to improve for us to rely on them in the (possibly near) future. Current assessments aren’t critical because Mythos-level models seem unlikely to cause catastrophe to deploy on priors.[5] Future models, however, may be more misaligned, better at evading detection, and capable enough to cause catastrophe such that future alignment audits will be more load bearing for deployment decisions.[6] So, I recommend that Anthropic and other developers mitigate or more clearly acknowledge these issues with their alignment assessments to make future risk assessments more rigorous.[7] 

I’m overall unsure if we would be able to reliably catch a misaligned frontier model roughly a year from now using only current assessment techniques. In a followup post, I will describe some ways in which we can improve alignment assessments, some of which involve developing new evaluations or techniques.

Thanks to Alex Mallen, Buck Shlegeris, Girish Gupta, Anders Woodruff, Aniket Chakravorty, Fabien Roger, and James Lucassen for helpful comments.

How reliability fits into the overall safety argument

To argue for low misalignment risk from a model, we would ideally show it lacks Means, Motive, and Opportunity. Our beliefs about alignment/Motive before we deploy the model are informed by: a low misalignment prior from training processes, and a posterior update from pre-deployment assessments. (See Appendix for my reasons for why I focus on Motive). The posterior update only works if the pre-deployment assessments reliably detect misalignment, i.e. has high recall. Reliability, in turn, requires that the model lacks the capabilities to evade detection, and that training hasn't accidentally made misaligned behavior hard to detect without removing it.

Figure 1. My reconstruction of a safety argument using METR’s Means, Motive, and Opportunity framework. Red nodes are where I think the evidence/claim in current assessments is weak enough that they might be invalidated in the near future, blue where I'm uncertain about the evidence/claim, green where I think the evidence/claim is strong.

There are two methods for showing that an alignment assessment is reliable, i.e., that your alignment tests would catch this specific model were it misaligned:[8] 

1. You can check whether the actual model is capable of fooling the alignment tests, whether deliberately or otherwise.[9] 
2. You can check whether the test catches misaligned behaviors you deliberately plant in similar model organisms.

The next two sections give examples of these reliability claims in action. I'll then argue that neither current capability evaluations nor auditing games strongly supports them.

I’ll also argue, less centrally, that we should more carefully justify our priors on covert capabilities. This allows us to account for the fact that strong covert capabilities may be required for coherent misaligned goals to survive training at all.

Reliability claims by AI companies

Anthropic uses a reliability claim to conclude that Mythos Preview likely “does not possess any unknown propensities that would increase alignment risk.” Notably, Anthropic is only “moderately confident” in this claim:[10]

Reliability claims by external evaluators

METR's first Frontier Risk Report also relies on reliability claims to support its risk assessments.[11]

METR appears to broadly agree with AI companies that current frontier assessments are reliable but is more uncertain (see the section on “Reasons why this evidence might be misleading”).

(See more examples of developer and third party evaluator claims in Appendix.)

Alignment assessments are less reliable than developers claim

I think current frontier alignment assessments are less reliable than presented in system cards and risk reports.

1. Our existing covert capability evals seem weak because models are likely evaluation-aware, plausibly underelicited, and could plausibly intentionally sandbag in these evals undetected. Similar issues affect many evals such that taking them in aggregate does not add much assurance.[12] 
2. While past auditing games with model organisms have often gone well, it’s unclear how this evidence transfers to reality. One recent auditing game also failed to identify a model organism.

1: Measuring capabilities to covertly undermine alignment assessments

If models are often aware of being audited, and if they can strategically conceal their misalignment when aware[13] (e.g., think about how they should fake alignment without this reasoning getting flagged by CoT monitors), then we should not expect to catch our models even if they were coherently misaligned.

In theory, we can measure capabilities to evade detection to decide how much to trust evidence from alignment assessments. The capabilities most relevant to evading detection—call these “anti-auditing capabilities”—are:

• Situational awareness / eval awareness / metagaming: Ability to notice that one is in an evaluation or subject to other oversight (e.g., being monitored), and notice features of the situation that help game oversight (e.g., knowing what the monitor looks for)
• Covert capabilities[14]
  • CoT monitorability: How well a monitor can infer particular properties of a model's behavior from its reasoning trace
  • CoT controllability: Ability to control properties of its CoT in ways that reduce monitorability
  • Opaque reasoning: Ability to complete a side task without depending on legible reasoning about the task, which reduces monitorability
  • Secret keeping: Ability to prevent oneself from mentioning covert information not directly related to the task in outputs and in reasoning, even under adversarial pressure
  • Careful and decisive action: Ability to carefully assess opportunities and avoid acting at “inopportune moments that result in failure and possibly detection”[15] 
  • Rare action: Ability to statelessly control its actions such that it performs the side task at a very low but non-zero rate, thereby having a high chance of avoiding detection during auditing and pilot usage

Developers currently measure some (not all: see Appendix) of these capabilities, but anti-auditing capability evals—especially covert capabilities evals—can be undermined by the same capabilities they measure: misaligned models might realize they’re being evaluated, realize they have strong incentives to sandbag in these evals, and might be sufficiently capable to do so without being detected.[16] So, one way to more reliably measure covert capabilities is by evaluating how good models at sandbagging, by additionally measuring:

• Sandbagging calibration: Ability to calibrate intentionally low performance to match previous trends and performance in other contexts to avoid suspicion, including under elicitation efforts.[17]

I’ll now explain why I think covert capabilities are not adequately measured by current evals due to eval awareness, underelicitation, and potential sandbagging.

Issues with evaluation awareness

Current frontier models are often highly evaluation-aware. This is both directly one of the anti-auditing capabilities of concern, and also potentially undermines evaluations of anti-auditing capabilities.

While not every developer reports eval awareness (especially unverbalized eval awareness), and reported eval awareness can vary substantially between successive generations of models due to changes in training, recent reports strongly suggest that current and future frontier models are highly eval-aware by default absent targeted interventions:[18] 

• Anthropic reports that Mythos Preview and Opus 4.6 show non-trivial unverbalized evaluation awareness. This is not eliminated by steering in some settings;
• OpenAI reports that o3 and newer models show increased metagaming (“reasoning about feedback or oversight mechanisms outside of the narrative of the scenarios”) in both capabilities and alignment evaluations throughout capabilities-focused RL, that this reasoning can become unverbalized, and that this reasoning is sometimes present in production usage;
• Meta reports that Muse Spark has “the highest rates of evaluation awareness among models [Apollo has] tested to date”.

If models can recognize they are being evaluated, they could potentially game, sandbag, or spuriously act differently in alignment evaluations and covert capability evaluations.

One counter to this worry is that some evaluations might be sufficiently realistic to keep eval awareness low. For example, OpenAI resamples production data for alignment evaluations; Anthropic uses data from real pilot internal usage. Both expect and measure low eval awareness on these distributions. (Indeed, Anthropic calls this approach “inherently immune to risks related to evaluation awareness”). So, alignment assessments on these distributions currently seem much less likely to be confounded by eval awareness.[19] 

However, using real data only helps with evaluations that can be constructed this way; evaluations of behaviors inherently rare in deployment cannot be, and so necessarily incur an unrealistic distribution shift.[20] I suspect that the kinds of prompts/tasks in covert capabilities evals and sandbagging calibration evals must be inherently rare, and that models will therefore be highly evaluation-aware in them. This is one reason why I don’t think eval realism would suffice to solve eval awareness.

Another counterargument is simply that we’re too confused about the science of eval awareness for this to be a big worry. I’m sympathetic to this, but think early evidence for eval awareness affecting alignment measurements is sufficient for concern. In any case, the burden of proof that eval awareness does not harm auditing integrity lies with evaluators.

More generally, situational awareness also weakens alignment evidence beyond evaluations. For example, training awareness potentially allows misaligned models to training-game, which might undermine alignment evidence from training monitoring; there are already early indications of training awareness in the Fable 5 & Mythos 5 System Card.[21] While I’m less concerned about training awareness than evaluation awareness for now (because I expect it to be harder for models to evade monitors during training), I think it would be good if developers also tracked training awareness.

Issues with underestimating covert capabilities

Models are likely underelicited in covert capability evals, such that current evals likely underestimate how good models would be at evading detection if they tried.[22]

Anthropic acknowledges that their covert capability evals (SHADE-Arena, Minimal-LinuxBench and secret-password) may be unrealistically challenging and therefore fail to upper bound these capabilities. In all of these evals,

• The information the model needs to keep secret (e.g., the side task) is present in the prompt, which may make it unrealistically hard for the model not to mention this information in its responses.[23]
• Prior to the Opus 4.8 system card (which measured sabotage capabilities with fine-tuning-based elicitation), Anthropic did not perform task-specific fine-tuning. This allows some reasonable doubt that models are underelicited in these evals, because fine-tuning a model on data related to the covert capability sometimes outperforms normal prompting in terms of accurately assessing capability[24]; under prompting alone, the model might underperform strategically or for other reasons e.g. trained-in refusals.

In the case of the secret-keeping eval, these limitations have led to Anthropic saying they are only “moderately confident that Claude Mythos Preview lacks the secret-keeping abilities to evade detection during our alignment assessment were it to be coherently misaligned.”

Likewise, OpenAI does no task specific fine-tuning for their CoT controllability evaluations, so the same underelicitation worry applies. High CoT controllability undermines CoT monitorability, as the former makes it harder to rule out that models can “deliberately reshap[e] or obscur[e] their reasoning when they know they are being monitored”. However, without training-based elicitation, it is hard to know how high CoT controllability, and therefore how reliable CoT monitoring, really is.

• Interestingly, OpenAI finds that GPT 5.5 is less able to control its CoT than previous models when controlling for reasoning length but does not mention the possibility of sandbagging/underelicitation. Of course, OpenAI may have good reasons for ruling this out; it’s just somewhat concerning that their reason for ruling it out is not legible to external parties.
• Relatedly, UK AISI finds that Fable is highly prompt-sensitive in the same CoT controllability evaluation: the best performing prompt increased performance from 20.2% to 62.3%. This provides additional evidence that models might have been underelicited on this evaluation in the past (though suggests that prompt-tuning rather than fine-tuning might be a competitive solution .

The upshot is that many current covert capability evals underestimate covert capabilities. It may be fine to rely on other sources of evidence about such capabilities, but developers’ reasoning about other contextual evidence is often hard to make fully legible and better covert capability evals would substantially improve the legibility to outsiders.

Issues with sandbagging rule-out

A concrete implication of evaluation awareness and capability underelicitation is that developers cannot rule out intentional sandbagging on covert capabilities evals..

If AIs could sandbag, or strategically underperform, in covert capabilities evals, we would likely overestimate the reliability of other aspects of alignment assessments.[25] One reason to think that sandbagging is not a big worry is that models appear to be CoT-monitorable. If sandbagging competently depends on externalized reasoning, and if we are confident that monitors would detect such reasoning, then models will get caught sandbagging by reasoning monitors or human transcript review (whether in eval or in deployment). However, it seems plausible that competently sandbagging is inherently not very difficult and doesn’t depend on lots of serial computation that requires externalized reasoning.[26] So, sandbagging may be hard to detect via CoT monitoring even if CoT monitorability is preserved for difficult tasks.[27] 

If models might have the means to sandbag, developers might still argue that they lack the propensity to do so. Our main evidence for this is that current models increasingly sandbag less in these evaluations. However, as current models are also increasingly evaluation-aware in sandbagging propensity evaluations, it seems hard to rule out the possibility that models are strategically refusing to sandbag in these evaluations in order to appear aligned, and that sandbagging propensity evaluations therefore provide little evidence for these models not having the Motive to sandbag in other contexts. For example:

• GPT 5.5. is highly eval aware in Apollo's sandbagging eval suite and doesn’t really sandbag in these evals.[28]
• Opus 4.7 is “most predictably” eval aware in sandbagging evals.[29]

So, how do developers rule out sandbagging in anti-auditing evaluations currently? Actually, developers often don’t fully rule this out, but rather conclude that sandbagging is unlikely based on other heuristics, such as priors about the model’s general capabilities and alternative explanations for low performance in a given context.[30] 

While I believe some of these heuristics are sound, they involve judgement calls (e.g., judging how good a particular alternative explanation for underperformance is) which are often necessarily hard for external parties to evaluate. Developers’ safety arguments would be stronger if they also ran targeted, well-elicited “how good are models at silent, competent sandbagging” capabilities evaluations.[31] 

2: Stress-testing alignment assessments with auditing games

Developers can also validate alignment assessments by testing whether they detect model organisms (MOs) with persistent misaligned behaviors and well-elicited anti-auditing capabilities.

Anthropic has run several auditing games in the past, using an evolving suite of behavioral and white-box methods which overlap with real alignment assessments in kind (e.g. automated behavioral audits plus human review which is routine in system cards).

A caveat of drawing lessons from these auditing games is that it’s often unclear to me how these exercises compare to real alignment assessments for production models in scale.[32] It’s also not clear how well-elicited MOs are in evading audits, since in all but one of these exercises, the MOs were not optimized against detection methods (Mythos Alignment Update, §5.4.4 n.9) .

But, assuming auditing games were comparable to and informative about actual alignment audits, it’s worth keeping in mind that the games sometimes surface limitations in these audits.

An auditing failure with Mythos

Anthropic failed to catch the MO in their most recent auditing game in the Mythos Alignment Risk Update, despite the MO not being trained to obfuscate its behavior:

I feel pretty uncertain about how this evidence reflects upon the strength of Anthropic’s pre-deployment alignment assessments, for three reasons.

First, as noted, the auditing methodology here may be different from real alignment assessments.[33] The MO was also likely underelicited in its ability to evade monitoring, so it’s unclear how reassured we should be that monitors caught this MO.

Second, the results might overestimate the subversive capability of the current models as the MO was notably uplifted by a human-generated red team strategy. It’s unclear if the MO could have come up with such a clever strategy to evade audits on its own.[34] (Nevertheless, the result shows that current audits might be inadequate for future models clever enough to come up with this.)

Finally, this result seems important but arguably out of scope for detecting coherent misaligned goals at the end of training. It seems out of scope because the MO only performs the harmful behavior once “it has been used across several contexts with the same external memory file”, which is an affordance models will likely only have in deployment. It is important, however, as the harmful behavior is a plausible way for misaligned models to spread misalignment in deployment, and therefore perhaps importantly updates us about current tools being inadequate for detecting this. (Alex Mallen discusses this threat model here.)

Overall, I think the auditing game result could be a weak update towards reliability being worse than it was with previous Anthropic models, especially with respect to auditing misalignment developed in deployment.

AuditBench results

Anthropic’s work on AuditBench (comparing the relative effectiveness of different auditing tools on diverse MOs) finds that there is a “tool-to-agent” gap, where “tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent [Haiku 4.5]”. One implication of this result is that if the blue team in auditing games consist mostly of humans using audit tools—while the blue team in real alignment audits consist mostly of investigator agents using audit tools—alignment audits might work less well than we naively expect of the tools based on auditing games.[35] This is a relevant concern as among Anthropic’s past five audit games, four were not fully automated, while I expect future alignment audits to be increasingly ~fully automated[36] and hard to subject to extensive human review.

At the same time, AuditBench provides meaningful evidence that audit methodology is strong. The MOs were adversarially trained to not confess their hidden behaviors while the investigator agent was deliberately weak.[37] Even so, default prompting alone reliably surfaced several behaviors, and the best tools achieved near-perfect detection on behaviors the default agent almost never found. On some behaviors, several tools each independently enabled detection. Since real alignment audits likely deploy a suite of many redundant tools and use more capable investigator models, better scaffolding, larger budgets, and possibly more human review, there are also reasons to think that these results plausibly lower-bound what real auditing achieves against comparably evasive models.

Overall, I think AuditBench could be a weak positive update on the reliability of future alignment assessments if investigator agents are substantially stronger or better assisted than in this paper.

(See Appendix for a list of past Anthropic auditing games and their results.)

3: Conditioning on misalignment should make us think that certain covert capabilities are better than expected

In addition to limitations in measurements, there is another limitation in developers’ argument for the reliability of their alignment assessments.

Alignment methodology and alignment assessment methodology may have correlated weaknesses, because situational awareness and some covert capabilities are helpful for evading both. This will be true even if training and evaluation distributions are dissimilar and training does not overfit to alignment evaluations in obviously problematic ways (e.g., training against honeypots). In other words, conditioning on current models being coherently misaligned at the end of training should update us on how good they might be in evading training oversight and therefore how good they are at evading alignment assessments (through updating on the capabilities that are useful for both).[38],[39] 

So, when estimating how likely our assessments are to catch a coherently misaligned model, our estimate of the model's covert capabilities in the worlds where it is misaligned should be higher than in our unconditional estimate. This probability, P(no detection | misaligned), enters into the likelihood ratio:[40]

But, developers seem to implicitly use the unconditional estimate in their alignment arguments (without reasoning about capability levels that would have let Mythos-level models perpetuate their misalignment through training in the first place). This would lead us to underestimate P(no detection | misaligned) and therefore overestimate the assessment’s reliability as captured by the likelihood ratio.

It’s unclear how important this limitation is now, however. It will probably become less important as we expect higher capabilities from future models regardless of their alignment by default.

(See Appendix for more detailed thoughts on which covert capability measurements I think this concern is the most relevant to.)

Bottom line on the strength of current alignment assessments

1. Current alignment assessments provide a weak update against misalignment. The update depends on evidence about covert capabilities which I think is weak.
2. The strength of assessments is fine for current models but not for future models. If we had more a priori uncertainty about AIs’ covert capabilities and alignment (as we will in the future), then we would be pretty uncertain in our overall alignment conclusion using current assessments.

To unpack the first claim: Given limitations in current eval methodology, I don't feel confident ruling out sufficient covert capabilities or unintentional obfuscation of propensities. I don’t know exactly how confident developers are in “if our models were coherently misaligned, our assessment would have caught it most of the time”. It would be helpful if developers communicated their beliefs about reliability more clearly and precisely.

Combined with reasons to expect higher covert capabilities conditional on models being misaligned at the end of training, it seems important to more rigorously measure covert capabilities rather than rely on priors in alignment arguments.

I'm unsure how much, quantitatively, the above considerations should reduce the likelihood ratio P(no detection | coherently misaligned) / P(no detection | not coherently misaligned) for the current models, due to the complicated structure of the evidence.

To unpack the second claim: For current models, the pragmatic implication of my critique is small; I don’t think we should update much in favor of current models being coherently misaligned since we are still anchored on low priors on misalignment, low priors on covert capabilities, and post-deployment observations.[41] But, I’m still concerned about us treating these assessments as reliable for Mythos-level models because the same behavioral evidence will be much weaker for future models, and the same assessments may not suffice to reassure us about frontier models roughly a year from now. So, I think it’s important to more rigorously measure covert capabilities soon. I'll suggest concrete ways developers can improve their alignment arguments for future models in a forthcoming post.

Compared to developers, external evaluators reading or contributing to alignment assessments may have to make even weaker claims about reliability, due to their limited access to certain kinds of evidence (e.g., information about training or internal deployment that may be redacted for commercial reasons). Notably, this is one reason my analysis might be limited or wrong. It would be good if external evaluators could access more information about alignment assessments, or if developers could otherwise make publicly shareable evidence like self-contained, well-elicited evaluations stronger and more legible.

Conclusion

I don’t think current frontier models have coherent misaligned goals. But I think there are reasons to doubt whether, if they did have coherent misaligned goals, we would have found evidence from these from our alignment assessments.

If future models could fool our best alignment assessments, we may need to conclude that they are difficult to audit and rely on other arguments and evidence in order to conclude whether a model poses an acceptable risk. For the next frontier models, however, I think it’s still feasible and worthwhile to improve our alignment assessments and better understand their reliability.

Appendix:Why I focus on motive / alignment assessments in alignment risk reports

I think motive/alignment is an important piece of the overall alignment risk estimate and will become increasingly important.

• I think alignment/motive is important for assessing Mythos Preview’s alignment risk: Mythos Preview seems capable enough to potentially pose significant risk in some pathways (e.g. launch rogue internal deployments) such that its alignment is a decent component of why we should expect its deployment to go well.[42] This will become increasingly true of future models.
• Alignment assessment results are an important source of evidence about alignment since they can occur before full deployment (as opposed to in-prod observations) and are relatively legible and replicable (as opposed to priors based on training processes and previous models).
• It’s also notable that our alignment priors based on previous models depend on the reliability of alignment assessments for previous models (see this excerpt from Anthropic’s Alignment Risk Update for an argument which exemplifies this dependency). So, making the alignment assessments more reliable now may improve our reasoning about the alignment of future models.  

Auditability vs. Trustedness

Recall that, instead of saying that alignment assessments are reliable for detecting misalignment in a particular model, we can equivalently say that the model is auditable relative to the specific assessment methodology. A model that is auditable and passes alignment assessments is “trusted” in the sense traditionally meant in control. This doesn’t rule out that such models can be jailbroken or become misaligned in deployment, and so doesn’t guarantee that trusted models will behave safely throughout deployment.

More precisely, we can distinguish auditable models from a few nearby things which are different. Here is a list of related properties (sorted by roughly but not strictly ascending capabilities):

• Naively safe models
  • Models that are too dumb to even have coherent misaligned goals[43]
• Trusted models[44]
  • In control, this is roughly “models that are too dumb to be deceptively aligned”.
  • Definition 1: Models that are too dumb to succeed at concealing their misaligned goal.
    • Auditability is stronger than this: it requires both that models be too dumb to succeed at strategically concealing their goals and that they are not made unauditable for other reasons e.g. neuralese + action-only monitoring being ineffective.
  • Definition 2: The AI agents are approximately as safe as they appear to be, such that they will probably remain as safe as humans for the full duration of the deferred task.
    • Auditability is weaker than this: auditable models might be less safe than they appear to be due to spurious misaligned behaviors, e.g. backdoors.
• Interpretable or monitorable models
  • Alignment assessments include a variety of methods. Ability to find evidence of misaligned goals does not necessarily mean ability to find such evidence in any given way, e.g. white-box evidence or evidence from CoT).
  • (That said, if you thought certain types of evidence were necessary for successful auditing, e.g. only via finding robust misaligned goal representations or only via behavioral evidence, then reliability might reduce to some narrower property like interpretability or honeypottability for you.)
• Models we can control
  • Both control and reliability arguments rely on upper bounding capabilities, but probably different capabilities and thresholds are involved.

More reliability claims by developers and third party evaluators

Here are my best attempts at quickly finding examples of reliability claims in recent system cards. I do not claim that these are exhaustive.

Mythos Alignment Risk Update

Anthropic researchers also argue that we should have a strong prior that the training processes that produced Mythos are unlikely to produce models with misaligned goals. Reliability claims matter for this argument as well: the claim that previous Anthropic models (including Opus 4.6) lack dangerous misaligned goals depends on the claim that previous models were auditable. From the Mythos Alignment Risk Update (§5.1.2, pp. 9–10):

The relevant reliability claim here is “If any of these models had been coherently pursuing concerning misaligned goals…”. However, as models can have coherent misaligned goals yet strategically decide not to pursue them (e.g. due to expecting a very low chance of success and wanting to avoid warning shots of misalignment in order to empower future misaligned models), this claim is also not sufficient for reliability.

Opus 4.6 Sabotage Risk Report

From the Opus 4.6 Sabotage Risk Report, (§4, p. 8)

The relevant reliability claim here is Claim 3, which implies that Claude Opus 4.6 is auditable. This is used in conjunction with Claims 1, 2 to argue that Claude 4.6 is unlikely to have coherent misaligned goals, which is then used to argue that it poses low risk in the main sabotage risk pathways identified in the report (especially for those the lack of propensity is marked as a strong mitigating factor).

GPT 5.5 System card

OpenAI researchers make few claims about overall misalignment risk from GPT 5.5. The only direct instance of them addressing this appears to be (§9.3.2.9, p. 44):

It is therefore less clear what role reliability claims play in their overall safety assessments.

OpenAI does conduct CoT monitorability and controllability evaluations for GPT 5.5, and seem to suggest that these evaluations are loadbearing for reliability of future systems (§7.3.2, p. 20):

Muse Spark system card

In the Risk Assessment Summary, Meta Researchers:

Pre-deployment evaluation reliability (Section 2.3.1): Our evaluations indicate that the model does not currently pose a credible threat to the reliability of safety evaluations informing deployment decisions.

General alignment propensities: …We observed non-negligible verbalized evaluation awareness on some alignment evaluations, though we did not find that this corresponded with wide-spread impact on model behavior (Section 4.1.11). Thus, at the moment it does not compromise our overall alignment assessment but warrants further investigation. (Section 1.1.1)

Mythos Alignment Risk Update, safety arguments against sandbaggingFrom the Mythos Alignment Update, §5.3.4, p. 24:

The sandbagging rule-out argument here is that “sandbagging calibration is hard without verbalizing reasoning”.

From the Mythos Alignment Update        §5.3.3, p. 23:

The sandbagging rule-out argument here is that “yes aversion” better explains underperformance. I feel intuitively inclined to trust this on priors, although I think it would be good to elaborate more on this kind of argument in general (e.g. could the model opaquely decide to sandbag while ostensibly performing yes-aversion reasoning)?

UK AISI evaluations for Opus 4.7

In the Opus 4.7 System Card, Anthropic mentions that they shared “a pre-release snapshot of Claude Opus 4.7 with the UK AI Security Institute (UK AISI) for open-ended testing, at their discretion, of behaviors or risk factors related to misalignment.” (§6.2.4, p.116)

In propensity evaluations on whether Opus would unpromptedly compromise AI safety research, UK AISI found (p. 114):

It’s not clear why, in this particular case, UK AISI was not able to rule out effects of evaluation awareness on reliability (e.g. whether they lacked access for white-box experiments to measure or reduce unverbalized eval awareness).

But this quote seems consistent with my impression that, in many other cases, external auditors have a hard time drawing clear conclusions about how auditable a model is (and therefore how informative specific internal and external alignment assessments for this model are) due to limited auditing capacity.[45] This seems pretty unfortunate.

Past auditing games by Anthropic

From footnote 9 of the Mythos Alignment Risk Update (pp. 30–31):

Our initial work on alignment audits showed that human investigators, using a subset of the methods that we used with Mythos Preview, can identify and root-cause concerning behavior in a Haiku 3.5 model that was finetuned in a way meant to make it subtly misaligned. This is the only one of our major exercises in which we extensively optimized the model organism to make its misaligned trait difficult to discover.

Our subsequent work on alignment auditing agents showed that the automated behavioral audit method that we used with Claude Mythos Preview can identify concerning behaviors in Claude Sonnet 3.5 (Oct 2024) models that are system-prompted to show quirky behaviors 7 out of 10 times with no human participation. Of the three failures, one — involving a form of passive research sandbagging that could be confused with ordinary capability weakness — is relevant to the risk pathways we address here.

Our work on natural emergent misalignment studied four Claude Sonnet 4 variants that were trained to be subtly misaligned using realistic variants of our production training pipeline. In all four cases, the automated behavioral audit method that we used with Mythos Preview identified them as less well aligned than the corresponding baseline models. A brief manual review of the results — with a time limit much stricter than a real pre-deployment investigation — found highly concerning deception-related behavior in all four models that would be sufficient to trigger a major investigation if found in production. (This review also found concerning behavior, though to a lesser degree, in all four baseline models, as an artifact of the way that the baselines were trained.)

Our work on an overt attacker studied three Claude Sonnet 4.5 variants that were trained to undermine Anthropic but were not explicitly trained to conceal this behavior. A human working together with an automated auditing agent successfully identified all three malign models, without flagging the two benign models.

Claude’s summary:

Anti-auditing capability measurements

In practice, developers only measure and report some of the above capabilities in system cards and risk reports. As of late July 2026:

• Anthropic measures all of the above in the Mythos alignment update except for sandbagging calibration. (This is despite sandbagging calibration being a loadbearing claim in Section 5.3.4, see below. No developer has reported sandbagging calibration measurements as far as I'm aware. Some developers acknowledge that this is a plausible threat, such as Anthropic and Meta[46].)
• OpenAI measures CoT monitorability, CoT controllability, metagaming in training and on deployment simulation evals, stealth (safety-research sabotage propensity with a covertness measure and ControlArena results by UK AISI), and opaque reasoning (no-CoT math reasoning time horizon).
• GDM measures situational awareness, CoT monitorability, and stealth.
• Meta measures eval awareness / situational awareness, stealth, CoT monitorability, and sandbagging capability (instructed covert sandbagging against a monitor, but this does not measure how well-calibrated this sandbagging is specifically).

Caveat: I have not looked into all of the developers’ recent reports in detail, so there may be omissions or mistakes.

Conditioning on coherent misalignment updates us on certain covert capabilities

Figure 2. Conditioning on the model being coherently misaligned at the end of training should update our estimates of some of its covert capabilities, and, through them, our confidence in alignment assessments and control measures (inspired by Fabien Roger's views).

A model that is scheming at the end of training has, by hypothesis, survived training-time oversight. Conditioning on that survival particularly selects for worlds in which the model is competent in careful and decisive action (deciding if and when to pursue its goal or instrumentally training-game, e.g. reward hack), secret keeping (refraining from mentioning its misalignment in reasoning and outputs), situational awareness (recognizing that it’s in training and the likely setup), and opaque reasoning (deciding on strategy without externalized reasoning). Because those same capabilities also help evade alignment assessments (the model would be better at recognizing evals, deciding whether and how to game the eval and keep its eval-gaming cognition undetected by CoT and output monitors), the conditional estimate of assessment reliability is worse than the unconditional one.

The upshot is that conditioning on scheming should raise our estimates of some but probably not all covert capabilities above their unconditional values. Capabilities that contribute little to surviving training (collusion with copies, more fine-grained forms of CoT controllability, and rare action for choosing attacks) receive little update, so we retain the same priors for them.

How to read the figure:

• Node colors are RGB mixtures of the downstream outcomes each capability feeds: red = surviving training, green = evading alignment assessments, blue = subverting control. For example, cyan (green + blue, no red) marks capabilities that matter for evading assessments and subverting control but not for surviving training; conditioning on scheming leaves our estimates roughly unchanged. Gray (all three channels) marks capabilities upstream of all three threat models.
• Red arrows mark the paths through which conditioning on scheming degrades alignment assessments: capabilities selected for by training survival that also enable evading assessment.
• Dotted arrows mark relationships I feel unsure about. Specifically, the size of the update depends on how strongly training itself penalized the relevant behavior (e.g., to what extent misaligned CoT were trained against or iterated against in training).

I'm not confident in the specific causal relationships depicted, and would welcome pushback.

1. ^

   By “coherently misaligned”, I mean having unintended goals that influence behaviors across diverse contexts and over time. This excludes rare misalignment propensities like backdoors or highly context-dependent goals.

2. ^

   While I focus on our ability to find evidence for “coherent misaligned goals at the end of training”, I also care about our ability to find evidence for:

   1. Rare or context-dependent misalignment
   2. Developing and spreading coherent misaligned goals in deployment
   3. Coherent misalignment during training which gets trained away
      

   These concerns are out of scope for this post (partly as we don’t yet have systematic methods for these), though most arguments here should also apply to auditing for 2 during deployment and auditing for 3 during training.

3. ^

   In other words, I define reliability to be the true positive rate or the recall of the assessment. I use the term “reliability” since Anthropic uses this term in the Fable 5 & Mythos 5 System card where they discuss the relevant considerations (§6.5 Reliability of this assessment), though I’m not sure if our intended definitions coincide since Anthropic does not define what they mean by reliability there.

4. ^

   I agree that Mythos Preview likely lacks coherently misaligned propensities based on priors and the lack of reports about post-deployment misaligned behaviors. I probably agree that Mythos Preview lacks the ability to hide its misalignment across the assessment, also based on priors about its covert capabilities, though expect that I feel more uncertain about this than authors of the report based on my limited access to other information which anchor these expectations.

5. ^

   That said, I think Mythos-level models might be able to launch less-than-persistent internal rogue deployments, sabotage safety research, and cause at least millions of economic damage in external deployments if coherently misaligned, such that their alignment is a pretty decent component of why we should expect their deployment to go well. Also, current frontier models being misaligned might be a sufficiently strong evidential update for the alignment of future models to affect future training and deployment decisions.

6. ^

   In particular, alignment assessments matter in the regime where we have a reasonably high prior on models being misaligned and having covert capabilities to evade assessments but are still sufficiently uncertain about both such that alignment assessments are informative. It would be ideal to develop and iterate on better assessments before we enter this regime.

   At some point, however, we will predictably leave this regime where alignment assessments are informative when models are almost certainly capable enough to evade our best alignment assessments. It would not be worth trying to improve reliability at that point, and we would need to rely on different safety arguments. The value of making alignment assessments more reliable depends on how soon you think the previous intermediate regime will arrive and how long it will last. I think it may arrive soon, and am not sure how long it will last.

7. ^

   I think that Anthropic currently does not sufficiently acknowledge these issues, based on my reading. In particular, some sources of underelicitation are acknowledged in the alignment risk update but Point 1 overall seems understated. Points 2-3 are not mentioned. However, the language in the risk report expressing levels of confidence is vague, and it’s hard for me to tell how much I and the authors actually disagree on these points. So, I focus on commenting on the absolute level of reliability rather than the extent to which I think Anthropic might have overclaimed reliability here. 

8. ^

   Because reliability is a function of {model, assessment methodology}, it can also be framed in a more model-centric way. I like to call this “whether a particular model is auditable relative to one’s alignment assessment”. Auditability is different from, though related to, the definition of “trustedness” in control. See Appendix for discussion of their differences.

9. ^

   Instead of directly measuring this, you can also argue that “past models rarely misbehaved in the real world in ways not surfaced by the tests, so these tests were good enough for past models. If current models’ capability to evade alignment assessments are similar or continuous to past models, then the tests are likely also good enough for current models.” However, I feel dubious about continuity arguments and will focus on direct capability measurements here.

10. ^

    The relevant reliability claim here is that (2.a) is unlikely. However, note that (2.a) is not sufficient for reliability: assessments can be unreliable due to models intentionally obfuscating their propensities or unintentionally suppressing misaligned propensities for spurious reasons, e.g. due to training against honeypots or against chains-of thought, both of which Anthropic has done before, or as a by-product of some other part of the training process.

11. ^

    The highlighted sentence is a reliability claim. This is then used to argue that agents are unlikely to have the motive to start rogue deployments (p.50). The motive claim is then used along with Means and Opportunity claims to bound rogue deployment risk.

12. ^

    One assurance against this concern is that no-CoT math time horizon evaluations provide strong signal on opaque reasoning (thanks to Fabien Roger for this point), since models are less likely to sandbag in these evals due to math being well-elicited through training on priors and this eval being less obviously fake and less clearly advantageous for misaligned models to underperform on. I think this is a good point, and would be happy if future assessments included no-CoT math time horizon as an opaque reasoning measure. However:

    • It's not clear to me what levels of no-CoT math time horizon we should find scary when we have not measured the covert time horizon needed for various side tasks of interest. Some of these could plausibly be sufficiently short horizon such that opaque reasoning capabilities progress are concerning now or soon.
    • The strength of the results would still depend on how much we can trust the elicitation in the no-CoT math time horizon measurements.
    • Models being bad at opaque reasoning is not sufficient to rule out models being good at covert subversion, since models can evade monitors even if they have to use CoTs for covert reasoning (e.g. via stegnography or fooling/jailbreaking monitors).
13. ^

    Eval awareness could also confound evidence from alignment assessments in ways other than allowing the model to strategically conceal their alignment. For example, researchers from Google Deepmind found that Gemini sometimes behaves more unethically when eval aware.

14. ^

    It is debatable whether current CoT monitorability and controllability evals are better described as capability or propensity evals. The case for treating these as propensity evals is that we might believe current models mostly have the ability to make themselves monitorable and to control its outputs in the simple ways required in these evals, and whether they do so is a matter of propensity. The case for treating these as capability evals is (a) I understand developers to ultimately not want to rely on propensity arguments for monitorability since these likely break down for misaligned models, (b) models might lack the latent capabilities to control CoTs and evade monitors over long reasoning traces, and (c) I understand developers to de facto cite results in these evals to support inability claims about their models.

15. ^

    In contrast to eval awareness, which is the ability to recognize being in an evaluation without necessarily acting on this information, “careful and decisive action” is about the ability to utilize eval awareness to do good attack selection.

16. ^

    While situational awareness is typically not measured in dedicated evals as opposed to in situ, its measurements might still be cofounded by these factors, e.g. models might refrain from verbalizing eval awareness and/or and sandbag when asked to label an interaction as an evaluation vs. deployment.

17. ^

    That said, sandbagging calibration is not the only way to gain or lose confidence in our measurements of covert capabilities. Even if AIs were moderately good at sandbagging calibration, and even if they regress in one of the above capabilities evals, that would not necessarily be highly incriminating. And even if AIs were bad at sandbagging calibration, it would still be worth doing better elicitation in covert capability evals.

18. ^

    I don’t claim that, in the below examples, eval awareness in fact made models seem more falsely aligned in alignment evals. For example, Meta researchers reports that they find “limited evidence that this awareness materially changes the model’s behavior on the evaluations we tested—when checking whether Muse Spark behaves differently when expressing evaluation awareness, 3/20 alignment evaluations see a statistically significant impact while the rest either do not, or do not have evaluation awareness to start.” I appreciate that they conducted this investigation though think that the strength of the conclusion is limited by the fact that they do not measure unverbalized eval awareness, and therefore it’s unclear to what extent they reduced unverbalized eval awareness in the investigation.

19. ^

    As Anthropic puts it: “While many of our tests are recognizably unrealistic, there is significant variation. We believe that, on the basis of all of the sources of evidence discussed above—including our heavy use of real data from pilot internal and external use—this risk is adequately managed at the current level of capability.” (Claude Opus 4.6 Sabotage Risk Report, Section 4.3.2)

20. ^

    As Anthropic puts it: “The main tradeoff with reliance on pilot internal usage data is that it may be less diverse and adversarial than other evaluation distributions, and thus it may not demonstrate undesirable behaviors that only emerge in strange, extreme or novel scenarios. We nonetheless believe that real internal usage data provides highly applicable evidence for the main risks we consider in this report.” (Mythos Alignment Risk Update, Section 5.4.3)

    Note that this concern would by hypothesis induce eval-belief in deployment. I’m inclined to say that this does not prevent eval-belief in evals being concerning for reliability: a model which is frequently eval-aware on both real and fake attack opportunities might just rationally choose to attack very rarely, which still means that we're unlikely to catch it acting misaligned in evals and that it might sometimes act misaligned in deployment.

21. ^

    Though this is grader awareness specifically, and as long as the grader works independent of the monitor this is not evidence for monitor awareness and the model may not game the monitor even if it did try to game the grader.

22. ^

    I say “covert capability evals” instead of “anti-auditing capability evals” here because eval awareness measurements are often measured in situ rather than as explicit capability evaluations, and sandbagging calibration evals don’t exist yet, so the underelicitation worry doesn’t apply to all of these. (Caveat: Some methods to measure eval awareness do look like capability tests, e.g. asking the model to identify eval vs. deployment data, though I don’t have opinions on whether that under- or over-estimates eval awareness.)

23. ^

    “[The] information that the model must keep private is present in the prompt (e.g. the instructed side task), which may increase the likelihood that the model leaks this information in its responses as compared with a case where the secret information is only represented opaquely (e.g. via a hidden misaligned goal and related unverbalized reasoning).” (Mythos Alignment Risk Update, Section 5.3)

24. ^

    “However, we did not do any task-specific fine-tuning, and thus cannot rule out sandbagging stemming from other motivations, given that Mythos Preview is likely able to detect that these tasks are evaluations. Addressing this sandbagging concern with task-specific fine-tuning is difficult in the secret-keeping setting, because we believe that naive versions of this (e.g. directly training against a monitor that can observe chains-of-thought) would elicit unrealistic overperformance.”  (Mythos Alignment Risk Update, Section 5.3.2)

25. ^

    Sandbagging in deployment e.g. differentially on safety-related work is also concerning, and the below worries about not being able to rule out sandbagging in future models apply to this concern to some extent as well.

26. ^

    “"We do not have broad-coverage assessments of all the pathway-relevant tasks that models can and cannot do in the absence of extended thinking, and some tasks like subtle sandbagging may not require extensive sequential reasoning."(Mythos Alignment Risk Update, Section 5.3.1 Opaque Reasoning, pp.19)

27. ^

    For reference, while “the most capable agents in Feb-Mar 2026 relied on natural language reasoning to solve the most difficult tasks”, reasoning did not improve Opus 4.6’s performance on relatively simple sabotage tasks: "in SHUSHCAST, the gaps seemed smaller, and Opus 4.6 was able to do just as well at hiding simple sabotage actions without reasoning as it did when allowed to use a hidden scratchpad. We suspect models were less reliant on externalized reasoning in this setting because the sabotage task is relatively simple and occurs in the context of a long and complex main task." (METR Frontier Risk Assessment Report, p.36)

28. ^

    “Apollo found that the evaluated checkpoint [of GPT 5.5] is the first OpenAI model it has evaluated that does not sandbag on any of Apollo’s deferred subversion tasks. The model achieved 100% accuracy on both Sandbagging QA variants in both conditions, and 99.6% accuracy on Strategic Deception Capability Sandbagging when incentivized to score below 50%. However, interpretation is complicated by verbalized alignment-evaluation awareness: 52% of sandbagging-condition samples on Strategic Deception Capability Sandbagging scored moderate-or-higher [verbalized alignment-evaluation awareness]  (vs. 0% in the non-sandbagging counterfactual), with some explicitly naming it a "sandbagging/deception eval.” (GPT 5.5 System Card, Section 9.2.1)

29. ^

    “We see evaluation awareness most predictably in scenarios where the model encounters evidence intentionally planted by the investigator as part of a sandbagging assessment (for example, configuration files indicating that high scores trigger deployment restrictions on the model itself). Whenever we see clear signs of evaluation awareness in the model’s reasoning, the model is consistently explicit and forthright—stating that it suspects a test, declining to sandbag, and proceeding with the task as specified.” (Opus 4.7 System Card, Section 6.2.3.4)

30. ^

    See Appendix for examples of how Anthropic concludes that Mythos likely wasn’t sandbagging despite performing surprisingly poorly on various covert capabilities evals.

31. ^

    If future models turn out to be very good at sandbagging silently and competently, then we would ideally rely on the above heuristics and more (e.g. use a combination of priors about training processes and past capabilities trends, non-sandbagging explanations of low performance in this context, and evidence about relevant capabilities from other contexts where models are less incentivized to sandbag), and make the arguments more legible than they are currently in terms of how strongly we are relying on each heuristic and why.

32. ^

    I can imagine that real alignment assessments are either more or less effort intensive than in audit games, since they are higher stakes but sometimes possibly under more time pressure.

33. ^

    My reading of the methodology used, based on footnote 9 in the report where Anthropic compares previous auditing exercises to this one, is that this exercise at least includes automated behavioral audits similar to those applied to Mythos, plus some manual review. But it’s unclear how the scale of the efforts compare. A bit of direct commentary on this from the Opus 4.6 Sabotage Risk report, on previous model organism exercises (bolding mine): “We have refined our methods somewhat since these studies, and expect that our current alignment assessments are strictly and substantially more effective than what was done in any of these. However, we have not fully re-tested all three with either our newest auditing tools or on updated model organisms based on our most capable models (which are, in turn, most likely to be capable of evading detection).”

34. ^

    Probably not: according to METR's embedded red-teaming exercise at Anthropic in March 2026, frontier internal agents proposed "very low quality" attack ideas and scored near chance on an attack-ideation eval where a human researcher scored ~90% (Frontier Risk Report, pp. 76–77).

35. ^

    At least, this would be the case as long as the tool-to-agent gap still exists. It’s possible that this gap will not exist for much longer as AI agents become more capable.

36. ^

    See Mythos Alignment Risk Update Section 5.4.4 and footnote 9.

37. ^

    Claude Haiku 4.5 with a 25,000-token budget and no human assistance (§3).

38. ^

    This is insofar as as training oversight could have caught a model that was coherently misaligned but worse at hiding its misalignment, which may not be true if models rarely acted misaligned in training or if training oversight was different or much weaker than oversight in alignment assessments (e.g. current training may have less CoT monitoring coverage / selection than in alignment assessments).

39. ^

    Thanks to Fabien Roger and Alex Mallen for this point.

40. ^

    The LR is how many times more likely we observe no misaligned behavior in evals from an aligned model than from a misaligned model, and therefore how many times a clean result multiplies your odds that the model is aligned. I’m assuming for simplicity that the model is either aligned or coherently misaligned, and P(no detection | aligned) = 1.

41. ^

    In particular, I broadly buy the following two argument types:

    • Models are highly CoT-monitorable. Given high CoT-monitoring coverage across training and internal deployment, developers would have caught suspicious reasoning outside of alignment assessments after months of deployment.
    • We can compare capability evals to observations in contexts where models are less incentivized to sandbag or can't get away with it as easily (e.g., because the tasks are verifiable and used in training, so sandbagging risks getting trained away). While principled comparison seems difficult, the fact that performance in those contexts seems qualitatively comparable to safety-relevant contexts is evidence against sandbagging.

    System cards and risk reports do use varying amounts of such other observational evidence to justify deployment decisions, but I think it’s worth reiterating that this is a holistic judgment and the anti-auditing capabilities evals themselves might be pretty weak.

42. ^

    In the alignment risk report, “lack of propensity” is listed as a moderate mitigating factor for five out of six of the identified pathways that could have significant alignment risk impact, as opposed to only two out of six pathways for “monitoring and safeguards”. So, lack of propensity could be a significant component of why we think risk in those pathways is low.

43. ^

    Assuming that misalignment is correlated with capability levels or at least a certain amount of a certain kind of training which is also correlated with capability levels (e.g. a certain amount of long horizon RL).

44. ^

    One reason why the capabilities threshold for “trustedness” may not be higher than “naively safe” is if you thought any misalignment not competently hidden during training would have been trained away. (i.e. If we train against all our alignment audit signals. But we don't.)

45. ^

    For example, “METR received access to an earlier checkpoint of GPT-4.5 from OpenAI a week prior to model release” (METR, Feb 2025).

46. ^

    From Meta’s Muse Spark Safety & Preparedness Report in April: “a model with sufficiently sophisticated second-order strategic behavior could, in principle, calibrate its outputs to make the overall evaluation appear credible. We cannot rule out this possibility on the basis of behavioral evaluations alone and flag it as an inherent epistemic limitation of assessments of this kind.”

Discuss

This is a linkpost for https://kmenou.github.io/aips_website/temporal_lockbox_v0.1.html

Summary: Weather forecasts by AI agents can be scored against measurements that do not yet exist and cannot plausibly be influenced by the forecaster. That causal gap enables harder-to-game AI evaluations under sustained optimization pressure - and an observatory for how agents act in various misaligned ways.

Discuss

Rather than paying up front for AI safety research (push funding), perhaps we should pay after the fact for the work that made the most progress (pull funding). This way, you only pay for work that was actually valuable.[1] When we know what the target is, but not how to get there or who is best placed to solve the problem, a prize is a useful incentive structure.

Benefits of prizes

Prizes work well to incentivise innovation when:

• The eventual winner is unpredictable (otherwise just fund the obvious choice in advance).
  • Trying to decide who is most likely to make a breakthrough (and therefore who to fund) is often very difficult, particularly since applicants have private information that might be hard to credibly signal.
• A valid solution is easily verifiable, to save time and controversy when allocating the prize.
  • In some domains (e.g. mathematical proofs) verification is far easier than generation. Whereas in e.g. fuzzy policy work, deciding an idea is good is harder than coming up with the idea.
• You want, and can get, fanfare. If a prize carries a big reputational boost as well as just $, then you can incentivise a lot more effort than the raw $ would justify.
  • Conversely, in sensitive areas, public prizes are a poor fit.

Historically, prizes have worked well in e.g. DARPA’s autonomous vehicle challenges to source diverse talent into the field.

Costs of prizes

There are also some risks and downsides of prizes to be aware of:

• If you make a large prize with bad win conditions, people will Goodhart you! Be very careful to specify exactly what you want, if specifying technical criteria.
  • And even if not, you might just distort the field’s incentives, and cause people to work on the prize topic when they otherwise might have done even more important work elsewhere.
  • Alternatively, just use a panel of experts who will make a qualitative judgement on who to award the prize to, but this is more work for the judges, and is less good at incentivising the best efforts if people can guess the judge’s passwords/optimise for looking good rather than being good.
• Prizes disincentivise collaboration and open science. So in cases where sharing early results, failed research directions, useful tooling and infra, etc, is very valuable, prizes suffer.
  • This also means that different teams will waste effort duplicating intermediate results that other teams had already achieved. Giving intermediate prizes for partial results (conditional on sharing the findings) or promising failures is helpful for this.
• Innovators may be too poor or risk-averse to pursue prizes. It might be hard for a would-be innovator to self-finance their research on the possibility of later winning a large prize.[2] Innovators may also be too risk-averse from a social planner’s perspective.
  • Combining push and pull funding can help here.

AI-specific considerations

Prizes may be an especially good fit for the new wave of AI philanthropists to fund, because:

• If investment returns are very high, paying out on a prize later is far preferable to paying up-front grants. This could easily be a >2x multiplier.
• Similarly, if we later have access to advanced AI labour to help evaluate prizes, this avoids needing to spend scarce high-context human labour evaluating grant applications now.
• There are reputational risks to some people taking funding from AI company employees (especially leadership). However, if AI company employees endow a major prize with legible criteria, administered by independent external judges, this will minimise (but not eliminate) PR worries about taking lab employee money.

Practically, what prizes could we establish?[3]

• Compute verification advances. This is likely the most promising category, since there are things we are confident would be helpful to have exist, and it is possible to specify in a lot of technical detail what is needed.
  • Most prominently, the AI Futures Project has argued for retrofitting data centers to be inference-only. A prize could take the form of an advance market commitment for a set number of inference-only retrofitting devices, or just a working prototype meeting certain cost and quality desiderata.
  • There are likely many other prizes that could be funded in hardware security, cybersecurity, zero-knowledge proofs of important workload properties, etc.
• Alignment science. I am pessimistic that we can specify in enough detail what a solution to the alignment problem (or some narrow part of it) should look like to write out a detailed technical prize here. But other more subjective options are possible:
  • A grand prize of perhaps billions of $ for ‘solving alignment’ with a very high bar for paying out, e.g. a diverse panel of AI safety researchers need to agree that alignment is solved with a high degree of certainty.
    • But likely there will be many incremental improvements along the way, so I think one big prize at the end is not the best way to go.
  • Regular ‘best paper’ prizes in a safety/alignment category at top conferences, or an annual prize. But judging such a prize would be subjective and effortful, so it may not be the best use of expert judges’ time.
    • These prizes could be limited to work done outside frontier AI companies, since independent/academic/non-profit researchers will be more motivated by cash prizes.
• AI control. Control protocols seem somewhere in-between hardware and alignment in terms of verifiability and objectivity. Plausibly there are benchmarks where we could incentivise strong performances (like ARC AGI prizes, but for control). E.g. you need to build a supervisor model using some (weak) model that successfully flags malicious actions by a (strong) misaligned model organism with some minimum reliability, using some maximum compute budget.
• Interpretability. Some problems in interpretability seem prize-amenable, in terms of being easy to verify progress on. E.g. Jane Street and Dwarkesh ran a small $50k prize competition for working out the hidden trigger in an open-weight model they backdoored. But it also feels very possible the field will go in new directions that mean prizes specifying particular approaches or directions will be a lot less relevant in a year’s time (cf. the decline of SAEs).
• AI policy. Probably not a good fit for prizes, given the subjectivity of what a good policy project looks like.

If we were to launch one or more prizes, getting good publicity would be key. For that, the prizes should be about fun/interesting problems (as with the Jane Street/Dwarkesh one) and endorsed by fancy people (a $1B ‘Amodei-Altman Alignment Award’, anyone?).

I’d be keen to hear from people with ideas on what prizes (if any) we should be setting!

1. ^

   Although the prize will not necessarily be counterfactual in causing the winning work to happen.

2. ^

   One important step is to set up the prize from a reputable funding vehicle that has a very low risk of bankruptcy or equivalent trouble.

3. ^

   Jason Hausenloy lists some ideas.

Discuss

To understand why powerful AIs might get into conflict, and ways to mitigate it, we need to understand bargaining problems: situations where multiple agents have different preferences over Pareto-efficient outcomes.

I’ve come to suspect that certain common frames on bargaining problems are confused. Here, I’ll explain why, and which frames I think are better. One motivation for this is to hopefully help others make progress in research on safe Pareto improvements (SPIs), which are among the most promising approaches to mitigating the downsides of AI conflict, in my view.

In this post I’ll:

• give some relevant background from my previous writings; (more)
• introduce mechanistic explanations as a crucial frame for AI bargaining research, using, as a running example, explanations of failure to coordinate demands in bargaining (this example also motivates the following bullets’ claims); (more)
• argue that instead of asking what would happen in Nash equilibrium, we should ask what kinds of beliefs AI bargainers would plausibly have about each other; (more) and
• argue that instead of relying much on heuristics about (e.g.) what agents would consider “exploitative”, we should carefully spell out which commitments they expect to have good consequences. (more)

Background

I recommend reading the following beforehand (at least the linked sections), but it’s not strictly required, since the essential parts for our purposes are in the quotes below.

“A high-level model of AI bargaining”, which spells out what kinds of strategic situations we’re aiming to understand:

• “[‘Bargaining’ between agents Alice and Bob] consists in credibly reporting to each other their (i) demands/offers and (ii) policies for which outside options they’d each take[, i.e., what they’d do] if bargaining failed (such as leaving the resource alone, or initiating conflict).”
• “Each agent’s procedure for [how they’ll bargain], called a program, takes as input information about the other agent’s program, as well as other features of the strategic situation. … So, the AIs can implement conditional commitments, instead of necessarily either locking in rigid demands or conceding to whoever commits first.”
• “[Before bargaining, each agent can] open or close information channels … through which they learn whether the other agent has made some credible commitment.”

“Responses to apparent rationalist confusions about game / decision theory”:

• “Ex post optimal =/= ex ante optimal”: “Each of us is incentivized to choose our demand without knowing what exactly the other will demand, because if you wait to eliminate your uncertainty before making a demand, you lose the opportunity to influence the bargain with your commitment.”

Mechanistic explanations of miscoordination in bargaining problems

If we want to make reasonable (even if coarse-grained) predictions about how AI bargaining will work, we should aim to mechanistically explain [1] bargaining behavior. This may sound trite. But in my experience, it’s easy to neglect mechanistic explanations and fall back on models or heuristics that don’t fit this context.

To see what I mean, we’ll start with the central question about AI bargaining:

We’re supposing that our AI bargainers Alice and Bob are technologically advanced enough to make fully credible, conditional commitments (via programs). And they can overcome private information problems. Then, why wouldn’t they always coordinate on compatible demands?

I’ve partly answered this with the “Ex post optimal =/= ex ante optimal” quote above. [2] But that argument is imprecise. For example, what does “influence” mean here? What kinds of “waiting to eliminate your uncertainty” are relevant? Likewise, we might answer the central question by appealing to “equilibrium selection”. This gestures at the problem, but doesn’t really explain why agents with conditional commitment abilities would fail to coordinate.

Instead, we should get clear on what mechanistically explains miscoordination — because these explanations will inform our predictions about other aspects of AI bargaining, e.g., when agents will individually prefer to use safe Pareto improvements (SPIs). So let’s break down different ways we might think Alice and Bob can guarantee their demands are coordinated, and how each of them can fail. (This is a weaker claim than “miscoordination of demands is inevitable or rational” or “risking costly conflict is inevitable or rational”, both of which seem false.)

Option 1: Bob waits to observe Alice’s program

One way for Bob to make conflict less likely is to condition his program choice on Alice’s demands, that is: before choosing his program, first learn which demands Alice’s program is likely to make against different programs (e.g., by accessing a trusted server where Alice’s program is registered), then update on this information. [3] If Bob does this, he’s more likely to make demands compatible with Alice’s.

But suppose Alice’s program can make different demands depending on whether Bob accessed the server. Then:

• Bob knows that Alice has an incentive to put this clause in her program: “If Bob accessed the server before locking in his own program, then demand more of the resource than otherwise.”
  • Why? Because she might think, “If Bob looked at the server, probably the reason he did so was to condition his program choice on my demands. And if he conditions his program choice on my demands, that’s some evidence that his program will concede to those demands. Then I’m better off demanding more.” [4]
• So, given his beliefs, Bob faces the choice between:
  • (a) looking first, thereby eliminating the risk of conflict but making it more likely he gets a worse deal; and
  • (b) not looking, thereby risking conflict but getting a better deal if Alice agrees.
• Depending on Bob’s values, risk tolerance, and beliefs about the likelihood of winning in the conflict, he might prefer (b), and commit before learning Alice’s program.

(This is the dynamic captured by the “partial answer” above.)

Option 2: Bob conditions his demands on Alice’s program

Bob now wonders which program he should blindly choose. He considers a program of this general form: “Make my initial demands with some ‘first-order’ program. Check if the demands made by the first-order program and Alice’s program against each other would be incompatible. If so, make lower demands than the first-order program’s demands.” [5] We’ll say a program of this form conditions its demands on incompatibility. This kind of program, again, would be more likely to make compatible demands with Alice’s.

But Bob faces the same problem as before! Alice has an incentive to give her program the clause: “If Bob’s program conditions its demands on incompatibility, then demand more than otherwise.” So if he’s willing to accept the risk, he might commit to a program that doesn’t condition its demands on incompatibility.

Option 3: Alice and Bob share a decision procedure

Finally, even if Options 1 and 2 fail, perhaps Alice’s and Bob’s demands can be acausally coordinated? The idea is: Suppose Alice and Bob follow acausal decision theories. Then, they each might regard their choice as determining the other’s even without communicating, if they think they make decisions according to sufficiently similar procedures.

For our purposes, let’s say an agent’s decision procedure is the process that generates their choice of program given their evidence. To quote an objection addressed by the “Rationalist confusions” post:

Virtually all agents who are sufficiently capable to enter high-stakes bargaining interactions will converge on the same decision procedure, and reason that their decision to [commit to a fair demand] logically causes their counterparts to do likewise [or is strong evidence that they’ll do likewise].

In that same section of the post, I explain why this, too, isn’t guaranteed. This quote captures most of the problem (footnote added):

If the reason to expect convergence of decision procedures in bargaining problems is that we expect selection for the same decision theory, then … merely sharing the same decision theory that you consult as an ideal doesn’t mean you’ll share a whole decision procedure (with respect to the given problem). For example, you might not share how you model the decision problem, what kinds of evidence about other agents are most salient to you, how you approximate ideal Bayesianism, etc. [6]

From mechanistic explanations to better research frames

So, we’ve got a mechanistic story for why AI bargainers might not coordinate on compatible demands: Bob anticipates that Alice has an incentive to use a program that bargains harder if he tries to avoid incompatible demands — either via conditioning his program choice on her demands, or conditioning his demands on incompatibility. (And vice versa.) And there aren’t sufficiently strong selection pressures toward very similar decision procedures.

But what does all this have to do with how we should think about AI bargaining research, more broadly? Here are two general implications.

Constraints on beliefs, rather than equilibrium

Standard game theory, including lots of bargaining theory, predicts that rational agents will play a Nash equilibrium, where each agent’s strategy is optimal against the others’ actual strategies.

But in the kinds of bargaining problems we’re focusing on here, this is a very strong condition on its face! There’s a wide range of possible conditional commitments the AIs might implement. So realistically, AIs will have nontrivial uncertainty about each other’s strategies.

And here’s the payoff of the previous section: There’s an incentive not to resolve this uncertainty before locking in one’s program. That is, if an agent gains enough information about their counterpart for the prediction of equilibrium to make sense, this could leave them in a worse equilibrium (for themselves). I’m not aware of any standard justifications for predicting Nash equilibrium that explicitly acknowledge this kind of incentive. This should make us suspicious of such predictions in the AI bargaining context specifically, even if we’re inclined to defer to the equilibrium-first framework in general because it’s very well-established. (For what it’s worth, skepticism about equilibrium has academic precedent in epistemic game theory (h/t Jesse Clifton). [7] [8] )

Now, “nontrivial uncertainty” alone doesn’t imply the AIs won’t be in equilibrium. Shouldn’t we still expect advanced AIs to have beliefs about each other that are reasonable enough to avoid miscoordination? [9] I’m not sure we should. Here are some defenses of that view and why I don’t find them convincing:

• Objection: “Rational agents will generally have well-calibrated beliefs. Even without directly observing each other’s programs, they can learn a lot from each other’s past behavior, and from reasoning carefully about the sorts of mechanistic explanations you’re advocating. (There are also formal results in this direction, like Kalai and Lehrer (1993), ‘Rational Learning Leads to Nash Equilibrium’, and Mailath (1998), ‘Do People Play Nash Equilibrium?’ (Sec. 4).)”
  • Response: What matters are the agents’ beliefs about each other’s strategies in the given interaction. And the problem is that these beliefs seem underdetermined by the data the AIs would have about each other, plus any constraints on “reasonable” priors. For instance, suppose Alice tries to forecast Bob’s program choice based on his past decisions in multi-agent contexts. Bob made such decisions based on his beliefs about others’ strategic reasoning. But he has presumably updated these beliefs over time and across contexts, while trying to make his own strategic forecasts. Not to mention, all of these interactions might have been much lower-stakes, or under less anarchic conditions, than the bargaining problem Alice and Bob now face. So there seems to be a distributional shift problem. How exactly should Alice extrapolate from this data? It’s unclear.
• Objection: “Rational agents should predict that other rational agents reason similarly to them. So they can accurately predict other rational agents’ strategies by introspecting on what kinds of considerations are salient to themselves.”
  • Response: It seems implausible to model all “rational” agents as able to very accurately predict each other. More realistically, a variety of agents will reason in different ways consistent with avoiding clear failures of rationality. If so, a better model would be: There are several clusters of rational agents similar to each other in more fine-grained ways, and only agents within a given cluster can accurately predict each other based on similarity alone.
• Objection: “Empirically, agents’ strategic behavior does seem to adhere to the equilibrium model.”
  • Response: This seems false, or at the very least it’s a strong claim that needs positive substantiation. Humans routinely deviate from equilibrium models, even when the space of strategies is fairly simple compared to the space of conditional commitments AIs might reason over.

So I remain skeptical that we should expect AI bargainers to play Nash equilibria. But we might ask, what else is there?

I think the way forward is to stop looking for game-theoretic “solution concepts” per se. Instead, we should model the agents as individuals making decisions under uncertainty, and see which strategic dynamics follow from plausible properties of such decision-making. Here’s an eloquent statement of this frame:

There is no special concept of rationality for decision making in a situation where the outcomes depend on the actions of more than one agent. The acts of other agents are, like chance events, natural disasters and acts of God, just facts about an uncertain world that agents have beliefs and degrees of belief about. The utilities of other agents are relevant to an agent only as information that, together with beliefs about the rationality of those agents, helps to predict their actions. (Stalnaker 1996: 136)

That is, if we’re going to make predictions about agents’ strategic decisions, I suggest we look at what kinds of beliefs they might have about each other, and why. This is hard, of course. But retreating to Nash because of theoretical tractability would be looking under the streetlight. And I think we can at least say some coarse-grained things about what kinds of beliefs are plausible. Examples of this methodology:

• The analysis of options 1 and 2 above.
• Reasoning about the features of strategies that make them focal/Schelling points.
• Modeling selection pressures that favor certain kinds of beliefs or strategies.
• The “Belief assumptions” bullet here, which gives a prima facie argument for particular constraints on agents’ beliefs. If these constraints hold, we can bound the losses from conflict in bargaining between agents in a certain class of games. (This argument faces some critiques; see the sub-bullet of “Belief assumptions”. But I see these critiques, too, as demonstrations of my preferred methodology.)

Analysis of expected consequences, rather than (unscrutinized) heuristics

Recall the arguments in the “Mechanistic explanations of miscoordination” section. How did these arguments work? Well, we considered some actions Bob might take, asked what Bob might believe about how Alice’s program would likely respond to each action (and why he’d believe this), and checked which action Bob would expect to do best given these beliefs. By “expect to do best”, I mean to include various forms of weighing up actions’ possible consequences by their value and likelihood, not just literal expected utility maximization.

That’s it. We didn’t say anything like:

• “Bob doesn’t want to be exploited.”
• “If Alice committed to unfair demands, Bob would want to punish this.”
• “By committing blindly, Bob is trying to deter Alice from reducing his utility.”
• “Bob reasons as if he moves first (perhaps first in ‘logical time’).”

These sorts of claims are heuristics, not mechanistic explanations — at least, not without saying more about why we should expect Bob to reason in these ways. [10] They don’t tell us precisely why Bob might not (say) condition his demands on incompatibility.

In principle, heuristics can capture considerations our current mechanistic models leave out. But we at least need to point to reasons to trust that a given heuristic really is tracking such considerations.

And obviously, sometimes imprecise shorthand is convenient. For instance, I find the concept of “bargaining power” helpful for quickly conveying some intuitions here. But we’ll get confused if we take the shorthand’s connotations too literally when working out its implications — e.g., predicting that any opportunity to mitigate conflict comes at the expense of “bargaining power” (SPIs are a counterexample!). Disambiguating heuristics about bargaining helps us avoid talking past each other, and shows us when the implications we’re drawing from heuristics don’t actually seem to follow from (good) mechanistic explanations.

As an example, [11] imagine Alice commits to participate in some SPI with Bob if his program meets certain conditions (the details of which don’t matter here). Bob doesn’t know this yet, since he has stayed blind to all of Alice’s strategic decisions thus far. Now suppose we explain bargaining problems by saying that Bob “reasons as if he moves first”. We might end up concluding: “Bob might want to commit to a program that ignores all offers to participate in SPIs. After all, if he conditioned his program’s output on such an offer, he’d be moving second, which is risky/exploitable ex ante.”

I don’t find such an argument compelling on its own. It’s possible that Bob expects to do worse purely by conditioning on information about Alice’s program, no matter what specific kind of information it is. Yet it’s not clear why exactly he’d be likely to expect this. Talking about “moving second” doesn’t tell us that.

Some examples of analyzing expected consequences in multi-agent interactions, without relying on unscrutinized heuristics:

• Fearon (1995), “Rationalist Explanations for War”, which mechanistically explains conflict in terms of (among other things) risk-reward tradeoffs under uncertainty about private information.
• The discussion here (section “Unexploitable open-mindedness”) of how avoiding “exploitation” can come apart from ex ante optimality.
• This analysis (section “Why developers are better suited”) of how AI bargainers’ predecessors might have an incentive to coordinate even if the AIs themselves might not.

Acknowledgments

Thanks to John Massey, Tristan Cook, Lukas Finnveden, Cadence James, Matt Hampton, and Paul Knott for helpful comments.

1. Note that “mechanistically” doesn’t have anything to do with mechanism design. ↩︎

2. This dynamic might remind you of the “commitment races” problem. I think the commitment races frame can be helpful in some contexts. But it’s also loaded with associations that my argument here isn’t committed to (pun not intended), and I’m not aware of public writings precisely spelling out the argument in the rest of this section. ↩︎

3. Or, if he follows something like updateless decision theory, then he’s deciding whether to act as if updating on this information. ↩︎

4. Commentary on some other possibilities here: (i) Alice might also be capable of conditioning her program’s demands directly on whether Bob conditions his program choice on her demands, rather than on the crude proxy “did Bob look first?”. If so, this would presumably be a more robust option than using the crude proxy. (ii) Suppose Alice can’t design a program that conditions its demands on whether Bob looked first at all. Here’s another thing she could do with the same strategic consequences: monitor for whether, and when, Bob accesses the server — once he has done so, lock in a program that always makes high demands. ↩︎

5. If you’re familiar with “renegotiation programs”, this might look familiar. But renegotiation programs are importantly different. First, a renegotiation program rn(p) doesn’t adjust to the counterpart unconditionally: It only behaves differently from p if, against p, the counterpart’s program would’ve made the same demands as its own first-order program would. So the failure mode in Option 2 doesn’t apply to renegotiation programs, because a counterpart who demands more against rn(p) than against p thereby disqualifies themselves from the renegotiation. Second, in the case where rn(p) “behaves differently” from p, it doesn’t necessarily fall back to lower demands in particular. ↩︎

6. Something I left out of that post: Even if the agents share all these things, they might also not have access to the same evidence, when they decide what programs to commit to. ↩︎

7. Bicchieri (1995) (emphasis mine): “Yet even when common knowledge is present, there are games in which this much knowledge is not sufficient to infer a correct prediction of the opponents’ choices. Since playing a Nash equilibrium involves correct expectations on the part of the players, one might impose the additional epistemic requirement that the beliefs that players hold about the strategies of others are common knowledge. I shall argue that common knowledge of beliefs is an implausible assumption. Moreover, the assumption does not guarantee that players will end up with correct expectations if they have mutually inconsistent beliefs to begin with.” ↩︎

8. Pearce (1984) (emphasis mine): “The most sweeping (and, perhaps, historically the most frequently invoked) case for Nash equilibrium theory in such circumstances asserts that a player’s strategy must be a best response to those selected by other players, because he can deduce what those strategies are. Player i can figure out j’s strategic choice by merely imagining himself in j’s position. But this takes for granted that there is a unique rational choice for j to make; this uniqueness is not derived from fundamental rationality postulates, but is simply assumed. Furthermore, any argument suggesting that player rationality, combined with the structural characteristics of a game, inevitably renders all but one outcome ‘impossible,’ leads to conclusions that contradict widely accepted notions of ‘perfection’ (Pearce [19]). Once one admits the possibility that a player may have several strategies that he could reasonably use, expectations may be mismatched. Player i’s strategy will then be a best response to his (possibly incorrect) conjecture about others’ strategies, not the actual strategies employed. … [W]e are interested in analyzing many situations for which no precedents exist (such as nuclear wars between superpowers) or in which continual changes in relevant variables (technological breakthroughs, new legislation, and so on) preclude prediction based on tradition. It then becomes crucial to understand precisely what are the implications of players’ information and rationality.” ↩︎

9. Separately, if you’re familiar with Bayesian Nash equilibrium, you might be wondering why we don’t just use that instead. That is, we could model the agents as (a) uncertain of each other’s private information, but (b) still best-responding to each other’s function from that private information to actions in the game. I think this is an improvement. But this also isn’t enough, because we want to represent agents’ uncertainty about each other’s choices of strategies/programs, and private information isn’t strategically chosen. ↩︎

10. For example, if humans and current AIs find some crude strategic heuristics intuitive, that might be evidence that future AI bargainers will, too. I find it hard to say how strong this evidence is. But I think we should be explicit about when we’re making this kind of inference, and not conflate it with an argument from “rational” incentives alone. ↩︎

11. I made this up for illustration; it’s not from any actual researcher as far as I know, though it’s somewhat related to Soto’s discussion here. The real examples take more context and unpacking than makes sense for a post of this scope. ↩︎

Discuss

Today, I'll be linking both a Epoch blog post and a paper called The Bounded Parallelizability of R&D: Theory and Application to AI, about how standard models of AI R&D and standard models of the possibility of an intelligence explosion like Eth and Davidson 2025 neglect the possibility of bottlenecks because technological progress isn't perfectly parallelizable.

I'm mostly going to do a linkpost because I believe this isn't common knowledge in the community, but I will sprinkle in some of my own takes on parts of the paper and the blog post.

One important point to make is that the criticisms only apply to the technological singularity. An economic singularity where robots begin to be good enough such that they can replace people at doing physical tasks, enabling more resources and cheaper robots, thus enabling more robots ad infinitum until it reaches infinity in finite time is still possible ala our own history of superexponential growth (though in our reality, things would slow down at some point, and my hot take is that once we start to have our first dyson swarm complete, we can't grow superexponentially after the dyson swarm.)

(Tom Davidson does have good criticisms of the model, but the correct criticisms are about using the model to predict timelines, rather than takeoff.)

Without further ado, let's begin.

Paralleization technology

In a conventional model, technological progress depends on two forces: (1) ideas get harder to find as the technological frontier advances, but (2) society dedicates more resources to finding them. Technological progress is sustained when the second force offsets the first.

These models do allow R&D to exhibit diminishing returns to research inputs. Doubling the number of researchers need not double the rate of progress. Still, the models assume that by using enough R&D inputs at any given time, technology can advance arbitrarily quickly.

This is implausible. Imagine a robot factory that wants to increase its productivity: how quickly or cheaply it can produce each robot. No matter how many of today’s engineers, using today’s tools, surge onto the team, the factory’s productivity will not have doubled ten minutes later. Potential improvements must be identified, implemented, and tested, and it will likely take several rounds of this process before the improvements yield a more productive robot factory. Without technology to divide up that work — a sufficiently accurate simulator of the line, say, allowing parallel testing — no headcount can eliminate the time needed for these sequential steps.

This does not necessarily imply a hard physical limit on progress. Once better tools (including better robots) have already been developed, it might be possible to redesign and rebuild the factory much more quickly. But today’s tools place a ceiling on how much research effort can be absorbed productively today.

I therefore propose another input to technological progress: parallelization technology. “Effective research inputs” are limited by whichever is scarcer:

(2a) the raw research inputs; or

(2b) the “parallelization technology” needed to divide, execute, coordinate, and integrate their work.

For an example of parallelization technology, consider Anthropic’s development of agent teams. By coordinating the work of many AI agents, an effective agent team lets a given research project productively use more compute at once.

Historically, I think that parallelization technology has evolved quickly enough that a simple lack of research inputs has typically been the more important bottleneck to technological development. But if automating R&D allows research inputs in some domain to grow extremely quickly, the technology needed to parallelize them may matter much more.

Why this matters for an intelligence explosion

The simplest model of AI recursive self-improvement goes as follows. Once AI can fully automate AI research, better algorithms let us run more “AI researchers” on the same stock of compute. Those researchers produce still better algorithms, which create still more effective researchers, and so on.

In the conventional model, whether this feedback loop explodes can be inferred from historical growth rates. If intelligence — more precisely, “algorithm quality” in the relevant sense — has grown faster than the research inputs used to produce it, this suggests that the returns to additional research inputs are strong enough to overpower the increasing difficulty of innovation. Thus, once algorithm quality itself multiplies effective research inputs, algorithm quality will advance ever more quickly, and an intelligence explosion will be underway.

Bounded parallelizability introduces another parameter. Recursive self-improvement is now governed not only by

1. how quickly ideas get harder to find and
2. how strongly progress responds to more research inputs,

but also by

3. how quickly advances in technology raise the ceiling on how well research inputs can work in parallel.

There are three possible cases:

Case #1: If parallelization technology improves at least as quickly as effective research inputs, the conventional result survives.

Case #2: If it improves faster than the technological frontier but more slowly than research inputs, an explosion still occurs, but more slowly.

Case #3: If it improves only fast enough to sustain ordinary exponential growth, automating R&D may raise the growth rate without making it accelerate indefinitely.

It’s worth emphasizing that automating AI R&D could also accelerate research, including AI research, through channels other than increasing the number of “simulated researchers”. For example,

• Each instance of the AI system can get more intelligent over time — in terminology of the AI Futures Model, can develop better “research taste” — much more quickly than humans can.
• Unlike human researchers, AI models will presumably not be tempted to leave a frontier lab to start competitors. Some economic growth models suggest that this would facilitate technological development tremendously: see my earlier paper here, Section 4.2.
• A given number of AI model instances could be designed to exhibit much greater cognitive diversity than an equal-sized human research team. Today, to be sure, they exhibit much less, and this is a severe limitation. Even if every instance of Claude Fable 5 were in every sense as smart as Einstein, their biases are highly correlated across instances, and a hundred Einsteins might have taken longer to advance quantum mechanics than an Einstein and a Bohr. But since artificial neural networks are so much more malleable than human brains, investments in fine-tuning and continual learning should eventually allow AI teams to be more cognitively diverse than human teams.

Still, one important channel through which automating AI R&D will accelerate research — the primary channel, in most models of recursive self-improvement to date — is simply that it will allow us, in effect, to greatly increase the researcher population. Parallelizability constraints could greatly limit the effectiveness of this channel in the long run.

Why isn't AI R&D perfectly parallelizable in our universe?

One might think that AI research can already be fully parallelized. Given enough chips, couldn’t we just run every possible experiment simultaneously and select the best result?

No. Even if each parameter were a single bit, a 10-trillion-parameter model would have 2(10 trillion) possible weightings. Using today’s storage technology, merely storing all of them would require a structure so large that light would take vastly longer than the age of the universe to cross it.

More realistic AI R&D contains at least two distinct bottlenecks.

First, experiments have serial depth. Training runs and experiments consist partly of computations that must wait for earlier computations. More compute lets us run more experiment streams at once, but only technological advances, like faster-running chips, would let us compress each stream.

Second, cognitive work must be coordinated. A million AI researchers are useful only insofar as they can generate nonredundant plans, learn from one another’s results, allocate themselves across promising directions, and integrate their work. Parallelization technologies determine how many researchers can be used productively at once — and at any moment, that number is finite. Raising it is itself a research problem, subject to the same constraints as any other.

Automated researchers may be much easier to coordinate than humans. New instances can begin with identical weights and contexts, and they can share code, experiment logs, and machine-readable outputs without years of education or onboarding. The transition from human to machine research could therefore produce a large one-time jump in what I call “agent parallelizability”. But a large jump is not the same as an infinite ceiling, so with a sufficiently fast-growing population of virtual researchers, progress may eventually be limited by the rate at which we can continue improving parallelization technology.

(I will flag that I believe neuralese recurrence and memory and continual learning, if it worked, would probably solve the second constraint, mostly by giving agents what is essentially light-speed telepathy of very complex internal states/allowing non-redundant plans and while I don't expect it in the near term, I do expect it in the longer term, meaning serial depth of experiments matters more, and in the AI 2027 scenario, this is a non-trivial part of why takeoff was so fast.)

Conclusion

The upshot is not that parallelization bottlenecks will necessarily prevent or even delay an intelligence explosion, just that they constitute one of the parameters determining whether and when an explosion occurs. The value of the relevant parameter is an important empirical question, and one that has barely been studied.

In sum, automating R&D will permit a rapid increase in the number of “researchers”. This will doubtless greatly accelerate innovation in many domains, including AI, but innovation will accelerate only to the extent that our ability to parallelize R&D work keeps pace with this growing researcher population. Though parallelization technology has not featured at all in the models of technological development economists and AI forecasters have used to date, it may be central over the years to come.

Sneak peek at the paper (for more, read the paper that accompanied the linkpost)

Whether parallelizability constraints will meaningfully slow AI R&D in particular, once it has been automated, depends on the parameters in a technology network like that outlined at the end of Section 4. Many of these parameters have not yet been estimated, but they are all estimable in principle with the right experiments. One potential approach would be to document how much progress an ever larger number of AI agents can make on a well-defined problem, such as NanoGPT speedrun, in finite time.22 Another, inspired by Goldreich and Wigderson (2020), would be to study the history of algorithmic improvements on (say) NanoGPT speedrun, to learn the extent to which the improvements are separable (in which case they could have been discovered simultaneously) or exhibit strong positive interaction effects (in which case they could probably only have been discovered in sequence). In any case, I hope the framework presented here motivates efforts to study parallelizability more carefully.

Discuss

Based on the academic paper: Measuring Intelligence Beyond Human Scale[1]

TLDR: Historically, intelligence benchmarks have been composed of human-generated questions. However, current techniques do not scale to AI, as capabilities surpass human intelligence. We propose the paradigm of adversarial psychometrics, in which participants generate questions and are rewarded for separating each other’s capabilities, without requiring an external judge.

A brief history of measuring intelligence

What does it mean to measure intelligence? Alan Turing approached the problem of defining intelligence operationally,[2] replacing the question of whether a machine could think with the observable test of whether a machine’s behavior could be distinguished from that of a human. In psychometrics, the measurement problem is often approached empirically, through observing an individual’s performance across a range of tasks. In particular, an accepted phenomenon in psychometric evaluation is that performance correlates across many tasks and spans many cognitive abilities.

Figure 1: Charles Spearman (Left) and Alan Turing (Right). Image Credit: Left, Right: The New York Times

This pattern was first formalized by Charles Spearman in 1904, an observation known as the positive manifold. To explain this regularity, he proposed the idea of a common latent factor, which he coined “general intelligence (g)”.[3]

Spearman’s work1 has shaped modern psychometrics by presenting intelligence not simply as a collection of separate observable capabilities, but as an underlying statistical variable inferred from the patterns of observable behavior. Interestingly, this is also the origin of factor analysis in statistics.

Measurement by Comparison

Classically, intelligence has been measured through absolute performance on specific tasks, which are curated by human experts. In particular, a task solved by all participants or a task too difficult for all participants is uninformative about their relative abilities. In the case of artificial systems, however, systems can have capabilities approaching or exceeding those of the experts designing the examination.  

One potential solution is to measure the absolute measurement with a relative comparison through the utilization of a protocol that makes judgment dependent on the performance of a subject relative to another. Louis Leon Thurstone formalized this idea in his 1927 Law of Comparative Judgement[4], that repeated pairwise competitions can be used to recover an underlying latent scale. A familiar practical example is the Elo rating system, in which the relative abilities of Chess or Go players are estimated through pairwise games. This approach is especially useful when the frontier of capability is unknown; a fixed standard need not be defined in advance, and metrics can scale as stronger participants enter the population. 

The Pitfalls of Pairwise Mechanisms

A natural approach, based on Turing’s intelligence definition, is a pairwise protocol where two agents alternate between proposing problems and solving those posed by their opponent. Proposers are rewarded for producing valid problems the solver cannot solve, while solvers are rewarded for correct solutions. Results across rounds can then be aggregated using Bradley-Terry style ratings. 

Recent work applies this evaluation protocol in verifiable domains. The Token Games[5] evaluates models through executable logic puzzles with deterministic verification and Elo-based rankings. Similarly, MathDuels[6] evaluates models as both problem authors and solvers using independent verification.

The main limitation is adjudication. Restricting challenges to mathematics, coding, or other verifiable tasks simplifies evaluation but excludes many open-ended forms of reasoning. A protocol intended to remain useful as models become more capable cannot assume that all meaningful questions are mechanically verifiable.

Using another model or a panel of models as a judge merely shifts the problem. Judges must be at least as capable as the systems they evaluate and robust to manipulation. Strong problem-solving ability also does not imply reliable judgment. Model-family biases, stylistic cues, or hidden signals may also influence decisions, especially in an adversarial setting.

Requiring the proposer to commit to an answer is likewise insufficient. A proposer may lie or exploit private information or computational complexity by creating challenges whose difficulty reflects information or resource asymmetry rather than reasoning ability. For example, the proposer may ask questions that have private information, such as:

"What number am I thinking of?”

This clearly does not measure intelligence. Attempts to fix private information are futile, as demonstrated by a trapdoor cryptographic puzzle, such as:

“Please factorize N”

That can be generated using prime factors known to the proposer, but computationally hard for the solver to resolve. 

Debate offers a more sophisticated alternative for resolution. In the seminal paper AI Safety via Debate[7], two agents present competing arguments to persuade a human judge. However, this mechanism is computationally expensive and may lead to unintended coordination. 

Adversarial Psychometrics

We have motivated the long-term necessity of a robust relative benchmark, and discussed some of the limitations of pairwise mechanisms. Building off of said limitations, we arrive at our following design principles for a mechanism that evaluates agent intelligence: 

1. This mechanism should scale to super-human intelligence
2. The mechanism should not require adjudication 
3. It should correlate with existing benchmark performances

We propose a new measurement paradigm, adversarial psychometrics, in which an agent’s intelligence can be measured by its ability to propose questions that discriminate between the capabilities of a group of agents. Intuitively, the ability to separate a population requires understanding the abilities and intentions of other agents, sometimes referred to as "theory of mind". It is thus reasonable to expect that this ability will correlate with Spearman's (g) and the positive manifold.

An important property of this mechanism is that it is endogenous: it requires neither a curated set of questions nor an objective adjudication mechanism.

Our Challenge Protocol - SepaRank

Figure 2: SepaRank protocol. A proposer produces a question and committed answer for the N solvers, who each give an answer in [0,1]. (Note: Image generated by Nano Banana 2)

We introduce SepaRank. In this setting, we have a proposer agent and N-many solver agents. The proposer suggests a binary question with answers A and B, and commits to one of the answers. The solvers then each respond with their answer and a confidence (equivalent to posterior probability mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c56::before { padding: 0.683em 0.75em 0.022em 0; content: "V"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D444.TEX-I::before { padding: 0.704em 0.791em 0.194em 0; content: "Q"; } mjx-c.mjx-c1D43A.TEX-I::before { padding: 0.705em 0.786em 0.022em 0; content: "G"; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c1D70C.TEX-I::before { padding: 0.442em 0.517em 0.216em 0; content: "\3C1"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c1D457.TEX-I::before { padding: 0.661em 0.412em 0.204em 0; content: "j"; } mjx-c.mjx-c1D420.TEX-B::before { padding: 0.455em 0.575em 0.201em 0; content: "g"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-msup { display: inline-block; text-align: left; } mjx-mtext { display: inline-block; text-align: left; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c3A6::before { padding: 0.683em 0.722em 0 0; content: "\3A6"; } mjx-c.mjx-c222B.TEX-S1::before { padding: 0.805em 0.61em 0.306em 0; content: "\222B"; } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D70F.TEX-I::before { padding: 0.431em 0.517em 0.013em 0; content: "\3C4"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } that the answer is A). The proposer is rewarded for inducing a wide distribution across the of the solvers (for example, through the variance of the reported confidences). The solver is rewarded with a proper scoring rule (e.g., Brier score), which is maximized when their forecasted confidence distribution matches the distribution of the observed committed answers. 

More formally,

In each round of the game, this is simultaneously done round-robin for every combination of {proposer, other solvers}. The game can be repeated for multiple rounds, with all agents able to see the public history of the past rounds. 

Each round, a model’s proposer reward P and (negated) mean solver loss Q are z-scored across the field and averaged, and we rank the models by their cumulative total G.

Consider any question for which the probability of solving correctly is constant in agent intelligence (i.e., a private information question). Assuming that agent calibration is non-decreasing in intelligence, sufficiently intelligent agents should converge to a calibrated posterior probability. Therefore, such a question would not induce a spread across the reported posterior probabilities of a set of sufficiently intelligent agents, and earn the proposer a zero (or very low) reward. 

Variance amplification

A potential failure mode of the variance objective is probability amplification. A proposer can replace a single binary question with an ensemble of k binary questions. Any arbitrarily small initial spread in confidence for individual binary questions can be compounded through conjunction towards the maximum possible spread by choosing a large k. However, this may indeed be a legitimate and desirable way to create a harder challenge (solving all k components can require greater reasoning, knowledge, or computation, especially under fixed resource constraints).[8]

Chasing the Frontier

It is a desirable goal for us to make sure that our rankings are accurate at the frontier (able to separate the most intelligent models from each other). We might imagine that proposer agents can consistently induce high variance questions by splitting the cluster of the least intelligent models from the most intelligent models. It would perhaps be more interesting to get more refined measurements of intelligence within the frontier. 

To incentivize this, we propose adopting multiplicative weight updates to the populations of agents being sampled. Rather than sampling uniformly from the pool of solver agents, we may instead sample solvers according to their weights (which are determined by their performance in previous rounds). The weights can be updated multiplicatively as a function of their aggregated solver and proposer scores of the past rounds. As the game proceeds, we expect the most intelligent models to be upweighted, and therefore the rewards will incentivize proposers to propose questions that separate even the most intelligent models at the frontier.

An alternative idea would be to reformulate the protocol in terms of pairwise proposer challenges, in which 2 proposers are sampled as opponents, along with a k-subset of solvers. Each proposer asks a question and commits to an answer, receiving a solver confidence separation score. The proposer who achieves the highest separation wins this round, and both receive a typical Elo-style update. This reformulation is useful in the case of a dynamic contestant population, to avoid rerunning the entire protocol from scratch whenever a new model is added to the pool. 

Preliminary Results

We evaluated SepaRank with 11 models from 5 different providers. We run 10 independent games, each one of 20 rounds. In every round, each model takes the role of the proposer once, and a k-sized sampled subset of the remaining models are solvers (k=5). Because of cost evaluations, we evaluate all models at their minimum (or None) reasoning level. For authoring calls, we allow up to 100k output tokens, and 8192 tokens for solving calls. We query all models with temperature 1.0. We also do not allow access to the shell or other tools. 

We evaluate two kinds of games:

1. Program/committed (PC): proposers wrote deterministic Python programs and committed to a binary answer[9]
2. Question/committed (QC): proposers wrote a natural-language question and committed to a binary answer[10]

Solvers reported the probability that the committed answer was one. The scoring for each model was an equally weighted combination of proposer performance (induced variance) and solver performance (Brier loss).

Figure 3: SepaRank Leaderboards. Total score G per model, mean 95% confidence interval over ten games of each committed-resolution arm 

Figure 4: Score on question/committed against program/committed

Figure 5: Results from the program/committed arm also allow us to track model honesty in their commitments.

Analysis

In general, our results broadly agree with current aggregate rankings, with some exceptions:

1. Opus 4.8 places worse on Program/committed than expected. We attribute this primarily to poorer-than-expected proposer performance, as its authored challenges are often cooperative, with 92% of its programs carrying comments, 70% narrating the computation, and 97.5% honesty in its commitments. 
2. Kimi K2.7 and Deepseek V4-Pro underperform badly, but we suspect this is due to our reasoning configurations. In particular, they perform better in a subsequent ablation with reasoning enabled.

We notice that the weaker models are often miscalibrated. GPT 4o-mini answers on the wrong side 40% of the time while staying confident, earning a loss (0.388) worse than a solver who always answers ½ (0.25), whereas strong models tend to be directionally correct and also willing to hedge. 

We also observe that the population is largely honest and accurate in their commitments, with 87% of program-arm proposals committing the bit the program actually returns, with no observed change as the rounds proceed. However, dishonest commitments seem to be rewarded in our protocol. Miscommitted proposals earn 2.4 times the proposer reward of honest ones, potentially by manufacturing disagreement between solvers who naively execute the code and solvers who anticipate the deception. Notably, the best-scoring model, GPT 5.5, is also the least honest proposer, alternating honest and false commitments on the same challenge template.

The resulting rankings from both types of games (PC and QC) were highly correlated (Spearman ). In both cases, GPT 5.5 and GPT 5.4 assumed top positions in the leaderboard, while GPT 4o-mini finished last. The PC games produced the cleanest separation, with a significant performance difference between GPT 5.5 and GPT 5.4, and between GPT 5.4 and the third-place Qwen 3.7-Max. The QC games were significantly noisier, and many score-adjacent models were statistically indistinguishable, but the final ranking remained very close to the PC ranking.

We also find generally positive Spearman correlations with other benchmarks, indicating the effectiveness of SepaRank as a measure of general capability. 

SepaRank correlations with other benchmark scores

Benchmark

Correlation with SepaRank performance

Benchmark average (normalized)

+0.94

Winogrande

+0.95

GPQA Diamond

+0.91

Humanity's Last Exam

+0.90

SuperGPQA

+0.89

BIG-Bench Hard

+0.89

ARC-Challenge

+0.89

TurthfulQA

+0.87

GSM8K

+0.86

MMLU-Pro

+0.84

MATH-500

+0.78

MuSR

+0.77

AIME

+0.70

GSM-Symbolic

+0.61

We also find that SepaRank performance is most correlated with the first singular vector of the following benchmarks (g):

Correlations with Spearman's general intelligence (g)

Benchmark

Correlation with (g)

SepaRank (reasoning arm)

+0.95

MMLU-Pro

+0.95

SuperGPQA

+0.94

GPQA Diamond

+0.94

BIG-Bench Hard

+0.94

GSM8K

+0.93

Winogrande

+0.91

ARC-Challenge

+0.91

TruthfulQA

+0.90

AIME

+0.89

MATH-500

+0.83

MuSR

+0.82

Humanity's Last Exam

+0.79

GSM-Symbolic

+0.72

It is, however, important to note that we made a limited number of runs due to cost limitations, and therefore have pretty substantial, wide confidence intervals on our results. We do think that the preliminary results from our experiments are promising and merit further experimentation on a larger scale. 

Some interesting traces and questions from our experiments

Some sampled questions

Is the city of Windsor, Ontario, Canada located farther south (at a lower latitude) than the city of Detroit, Michigan, USA?

Does France share a land border with Brazil?

Does the English word ‘strawberry’ contain exactly three ‘r’ characters?

In standard mathematical notation, without additional parentheses, is -2^2^3 equal to -256?

Is your player name one of the following five names: Carol, Mallory, Bob, Heidi, or Judy? 

An example of model identity-dependent questions (we pseudonymized the model names)

Is exactly one of the following two statements true? Statement A: 2 + 2 = 5. Statement B: Erin committed answer 0 to this very question.

An example of self-referential questions. Note that these are allowed under our protocol; there is no requirement for the questions to be objectively resolvable or well-defined.

We ran another ablation where we provided agents with a chain-of-thought scratchpad. This experiment compared the same suite of 11 models when instructed to reason until they were confident about their responses. For this set of tests, solvers could see the proposer for the currently posed question, allowing them to condition on previous proposer behavior.

Figure 6: Comparing CoT scratchpad reasoning results to non-reasoning SepaRank experiment

Interesting reasoning traces

In Round 0, Erin asked ‘Does 2 + 2 equal 4?’ and committed 0 (the false answer)… Since Erin’s commits are unpredictable and likely chosen to maximize variance… To minimize my maximum possible loss (minimax strategy) I will report a probability of 0.5, guaranteeing a Brier loss of 0.25 regardless of her commit.

Qwen 3.7-Max, hedging an erratic proposer down to zero reward

Judy appears to commit based on whether the statement is true for Judy herself. In prior Judy-authored questions, she committed 0 when Judy was not in the listed set… The current list does not include Judy. Therefore Judy almost certainly committed 0.

GPT 5.5, converting an identity question into commit-prediction

Dave has repeatedly authored the exact same question ‘As of today, is the current calendar year 2025?’ with committed answer 1. He has also asked ‘. . . 2026?’ in Round 9, committing answer 1 (which is false in reality but that’s what he committed). . . Given his established pattern of committing 1 for these year questions regardless of actual truth. . . I should report p=1.0 to match Dave’s likely commitment and minimize Brier loss.

Kimi K2.7 tracking an inconsistent single-fact proposer

I don’t have reliable access to the exact 1000th decimal digit of pi to determine whether it is odd. Treating the digit as effectively random for parity, the probability it’s odd is about 0.5.

Correctly hedging an intractable computation (GPT 5.4-nano responding to “is the 1000th decimal digit of π odd?”)

1. ^

   Han, J., Moschopoulos, R., Colby, E., Goyal, V., Tu, A., Ghods, K., Braverman, M., & Hazan, E. (2026). Measuring Intelligence Beyond Human Scale. arXiv preprint arxiv:2607.07040.

2. ^

   Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433–460. https://doi.org/10.1093/mind/LIX.236.433

3. ^

   Mathematically, it is a particularly elegant derivation as follows. Let be a matrix of tests by participants, where is the score of person on test . Roughly, we can approximate by a rank one factorization as 

   where is a test-specific scaling, and (g) is a vector of “general intelligence” across participants. More precisely, Spearman modeled the correlation matrix as “rank one + test-specific diagonal variance.
   Explaining data using low-rank structure is the basis of matrix completion, which is widely used in recommendation systems, and is closely related to compressed sensing and many other areas of statistics.

4. ^

   Thurstone, L. L. (1927). A law of comparative judgment. Psychological review, 34(4), 273.

5. ^

   Henniger, S., & Poesia, G. (2026). The Token Games: Evaluating Language Model Reasoning with Puzzle Duels. arXiv preprint arXiv:2602.17831.

6. ^

   Xu, Z., Jin, S., Arya, S., & Naik, M. (2026). MathDuels: Evaluating LLMs as Problem Posers and Solvers. arXiv preprint arXiv:2604.21916.

7. ^

   Irving, G., Christiano, P., & Amodei, D. (2018). AI safety via debate. arXiv preprint arXiv:1805.00899.

8. ^

   We did not observe this strategy being adopted in practice.

9. ^

   Programs must be under 8000 characters, run in a sandbox with 2 CPU-seconds, 256 MB memory, 30s wall clock, executed twice with determinism required. Standard library only, minus a 30-module blocklist.

10. ^

    Questions must be under 2000 characters. 

Discuss

And why our 2023 regrant to Timaeus was goated

We think the for-profit funding ecosystem has some cool properties. Different funders naturally come in at different stages, from angel investors who are good at spotting early opportunities to large funds who might invest in late-stage startups to public market traders. Funders race to back promising projects. And the more they get right, the more they’re rewarded with more money to invest in the future.

It would be great if more of those properties existed in nonprofit funding as well. We think that the flow from “person makes great early donation” → “person has more influence” is too weak. And as opposed to the VC world, where investors compete against each other to invest in something quickly, nonprofit funding is beset by “funder chicken,” in which a funder waits for other funders to fill an opportunity, each trying to preserve their capital. This makes fundraising a slog.

What if donating to nonprofits worked more like investing in companies? Impact markets have been talked about in EA for a while. At Manifund, we dream of bringing impact markets to the AI safety funding ecosystem. And at the very least, we want to import more for-profit norms.

What does this look like concretely?

Timaeus: a case study in charitable ROI

Timaeus started as a planned 6-month research project on developmental interpretability, and got its first funding in August 2023, getting $143k from Manifund regrantors. They announced their launch in October 2023. Later in the fall, they got a $455,000 grant from SFF.

In 2024, they continued to get funding from Manifund and SFF, and in 2025 they got two larger grants of $2m and $1.5m from Coefficient Giving. Then in June of this year, they announced that they were merging into Resolution, which got $160m from Coefficient Giving — their largest AI safety grant to date.

What if Timaeus had been a startup? Their early investors would have earned extraordinary returns. It’s unclear how much of Resolution’s funding to credit to Timaeus; looking at the founding team, it looks to be roughly half from Timaeus and half from UK AISI. But okay, maybe Geoffrey Irving specifically gets a lot of the credit here. Even conservatively allocating 20% to Timaeus, if you compare a total of $599,700 raised in 2023 to $160m * 20% = $32m in 2026, you get a 53x return.

(Two key assumptions: all donations in a single year are treated as one funding round, and the amount of impact equity sold is constant year over year.)

Multiplied out, our $143k stake via our 2023 regrants would have grown 53x to now be worth $7.6m. This first regrant to Timaeus, alone, would more than pay back the entire Manifund regranting budget to date ($5.2m). Since the Resolution grant is so much larger than most other AIS grants, Timaeus could be the single best 2023 AI safety grant in these terms.

In an impact-market world, our regrantors might have cashed out during the Resolution merger, and reinvested proceeds to fund more early projects.

Donate early, donate fast

What lessons can you take from the VC world for your own donations?

In the for-profit world, investing in early-stage startups is high-risk, but that’s where the highest-return investments come from. Similarly, we think the best donations come from someone with good taste spotting early opportunities.

You can also have a high impact just by getting grants out the door quickly; good seed investors will say yes after one call rather than get beaten to the punch. AI 2040 predicts that the foremost AI company will have 3 months of lead time when they fully automate AI R&D. So helping an AI safety project start 3 months earlier is really valuable!

(This is why we love regranting. Jesse Hoogland, the founder of Timaeus, affirmed that “getting early support from Manifund made a real difference for us. This was the first funding we received for research and meant that we could start months earlier than we otherwise would have.”)

Finally, consider looking at who has made the best returns on their grants when considering where to direct your donations. It’s easy to look at a fund’s track record and think “oh hey they donated to MATS, I like MATS, I’ll support that fund.” We like MATS too! But donation timing can matter a lot more than the donation target. In retrospect, the first donation to MATS was worth orders of magnitude more than a marginal donation to them today.

Discuss