Сборщик RSS-лент

Lately I've been thinking a lot about what work would help with actually winning and getting to good worlds. In the spirit of that I decided to venture outside my normal wheelhouse and spend some time reflecting on what technical research could make me more confident about powerful AIs being safe.

AGI safety research is tricky partly because we don’t actually have access to the thing we want to study, i.e. superhuman AI. Much of the work we do now is basically trying to lay (potentially irrelevant) foundations for the period when we actually know what we’re up against, and at that point, a lot of the work might be done by AIs.

You can group current approaches by how they try to sidestep this access problem:[1]

• Prosaic techniques like RLHF and interpretability try to make progress on current model safety in a way that will hopefully generalise, except maybe they just won’t scale
• Model organisms artificially construct exemplars of bad behaviour (alignment faking, trojans etc) but it’s hard to tell how representative the constructed case is
• Control techniques aim to get usable work out of potentially misaligned AIs and bootstrapping, except it’s again unclear how far that will scale
• Agent foundations tries to reason theoretically about what powerful AIs will be like, except it’s really hard to be sure if you’ve made any useful progress
• Honourable mentions: Evals work, Scalable Oversight (RIP), Governance, Pause advocacy etc…

Here I want to gesture at a different angle of attack, which I’m going to call structural proxies. The basic idea is that you look for current naturally-appearing problems with AI that share a structure with future problems — in other words, structural proxies. In particular, you want to have some reason to believe that the current problem is being generated by the same types of dynamics that will produce the future problem, even if the manifestations look quite different. You then use the proxies to try to understand the future problems. 

The closest parallel is model organisms — it's plausible to me that this is just a special subset. But a lot of model organism work appears to start from the specific phenomenon (reward hacking, alignment faking, collusion...) and then deliberately construct an environment that demonstrates it. With structural proxies, you start from the process that produces the phenomenon (misgeneralisation, latent knowledge, optimisation for proxies...) and then look for other examples of that process in the wild.

Briefly, I think this is a useful angle because:

• Very generally it’s good to have more angles of attack
• It’s especially useful as we enter the era of automated engineering
• Although you have more constraints on what you can proxy, the proxies have the potential to be pretty faithful, and therefore to give you a lot of evidence
• In particular, this could help us spot problems or places where things don’t scale
• Conversely, getting a rich understanding of a structural proxy for one of the endgame technical challenges actually would make me feel better about alignment

I’ll run through two examples now — adversarial attacks as a proxy for value generalisation, and faithfulness as a proxy for ELK — then give some broader thoughts.

Adversarial attacks as a proxy for value generalisation[2]

Conceptually, what an adversarial attack does is exploit the gap between what we want the model to learn and what it’s actually learned — in other words, exploit its misgeneralisation. In the case of image models, producing adversarial inputs that work across a variety of models requires you to exploit the general ways that image models learn differently to humans.

Interestingly, these differences appear to be somewhat natural and intuitive byproducts of the fact that image models train on static images whereas humans see things continually — to the point that some adversarial images also “fool” humans who only look at them for a fraction of a second. I think this is spiritually similar to the way that feature visualisation on image models by default gives you a horrid static mess, but starts to look recognisable once you regularise for an input that’s robust to small translations and other perturbations.

Distill pub has live demos where you can vary the parameters — highly recommended if you want to get a feel for how jittering affects human interpretability.

Language models are a bit of a different ballgame, because the input space isn’t quite so densely high-dimensional — for an image model, a slight perturbation means an almost imperceptible pixel shift, whereas for a language model it probably means an embedding that doesn’t correspond to any tokens. Nonetheless, I think the shape of the argument is similar: Jailbreaking shouldn’t be viewed as a strange epicycle on otherwise-sensible models, but rather as a sign that very fundamentally language models are generalising differently to the way we’d want them to — in other words, they are misaligned.

And indeed, it seems like by greedily searching for token substitutions one can do something similar to tweaking pixels on images — producing model-agnostic suffixes for text prompts that force certain behaviours. The good news is that, much like the image model case, this is much harder if you regularise the inputs into being somewhat comprehensible, to the point that sometimes the shape of the attack becomes human-understandable.

Human classification can also be flipped by surprisingly small tweaks — spot the difference.

Taking a step back: it seems like the standard orientation to adversarial attacks is that they’re mostly a misuse problem, where people are using incomprehensible blobs of noise to hijack otherwise benign models — models that have been aligned against a really diverse range of examples and validated on a really diverse range of examples. In other words, one might think that adversarial attacks are shallow, and the actual deep thing is constitutional learning and eval performance.

But I think there’s a reasonable case to be made that it’s the other way around — that adversarial fragility is the deep signal, and eval performance is a shallow layer on top — and that blocking jailbreaks on language models and exploits on image models is hard for the same structural reason that alignment is hard.

The million dollar question here[3] is what the limit of this process looks like. For example, does there exist a training process which gives you image models that classify things exactly the way a human would, including in worst-case adversarial inputs? If so, I think that would make me a fair bit more optimistic about alignment. I expect that much of what you learned in building such a system would be pretty generalisable.

That’s the structural proxy angle: taking a known, natural issue (adversarial robustness) and noticing the ways it’s structurally similar to a big future problem (value generalisation), and using the proxy to get purchase on the future problem. Notably, lots of traditional adversarial robustness work like layering smaller classifiers on top is not at all useful for pursuing the proxy angle; on the other hand, plenty of adversarial robustness work that’s not super inefficient for actually preventing misuse suddenly looks really appealing from this angle, like trying to find conditions that make optimised jailbreaks more naturally interpretable, or trying to train models in a way that makes optimised inputs less like static.

Faithfulness as a proxy for ELK

The problem of Eliciting Latent Knowledge (ELK) was ARC’s big attempt to give a formal, tractable variant of the alignment problem: if you have an AI that knows a bunch of stuff, can you train a system that will report that knowledge? Solving this gets you most of the way to solving alignment because you can just ask your misaligned system questions like “are you misaligned” and “is this going to go wrong” and actually elicit all the latent important details.

And the basic challenge is that it’s really hard to be confident you’ll actually get an “direct translator” that expresses the latent knowledge in the AI, instead of a “human simulator” that figures out what you’d expect to hear and just tells you that. For example, no question/answer pairs from you can privilege “tell us the truth” as a strategy over “tell us what we believe the truth is”, especially once the model is superhuman. Worse, it’s possible that figuring out and extrapolating the most sophisticated world model humans might develop could be much easier than extrapolating over the one true model that ASI will learn.[4]

Cribbed from the (very readable) ELK report.

This was originally posed as a hypothetical problem, and most of the work on it has been fairly abstract — ARC’s current approach is mostly about finding algorithms for predicting the behaviours of different networks in a principled way that’s more efficient than just sampling them.[5] But I think one can learn quite a lot about it from looking at current systems — indeed, I think we’ve already learned a fair bit.

For example, we know:

• Neural nets can learn world models which you can probe
• Base models repeat common misconceptions, and this can increase with scale (this is actually where the term inverse scaling was coined!)
• LMs mirror the beliefs of their users
• LMs can be trained to somewhat accurately report the meanings of internal computations

People already care a lot about whether AIs are honest. This has the advantage of being easily testable whenever you have some ground truth to hand. But for ELK I think the better proxy is faithfulness — not whether the output is correct, but whether the process generating it was coupled to the right internal structure.

Here it’s useful to consider what work would be considered out of scope. One good example is consistency. Lots of people are into making models consistent across conversations — to reduce sycophancy and jailbreaks, make them better at conceptual reasoning, or even find out what they really think. If you put on your ELK hat, though, it’s clear that this is only half the battle. Consistency makes AIs pick one world model, but does it make them pick the right one? (Spoiler alert: maybe not.)

Now, solving ELK for GPT3 is pretty different to solving it for arbitrary superintelligences. My point is that you could still try to solve it, or to figure out what bits are hard. Do AIs even have a true model of the world, from which they model what humans expect to hear? How does this connect to the fact that current AIs are a bit more like layers of prediction with some ability for the character to tap the raw capacities?[6] Can we find the causal channels when AIs are simulating ‘inaccurate’ reporters? How does all this vary with model size and post-training?

So, recapping the angle: take a known, natural problem (unfaithful reporting), noticing the ways it’s structurally similar to a future problem (ELK), and use the proxy to get purchase.

Thoughts on structural proxies as a research direction

My big ambitious hope is that this kind of work could actually make us more or less confident about the difficulty of aligning very powerful AIs. I worry that a lot of safety work bounces off the hard parts of the problem, and I like the idea of actually grabbing at whatever hard parts you can find. It would be very useful to have compelling evidence that we’re not yet equipped for the hard challenges, by providing smaller but not much easier instantiations. But there's also a lot of upside — if we could solve jailbreaking in full generality, or we could figure out exactly how faithfulness works in current models, that would be big. I have a hunch that when you work on simplified versions of the problem, sometimes you risk simplifying away not just the things that make them difficult but also the things that make them solvable.

More mundanely, I just think it's good to have as many angles of attack as possible. Especially as we enter the era of automated research, it's good to have a plan for what we point the auto-engineers at. It's also good to have approaches with different strengths and weaknesses so that the overall portfolio is a bit more robust.

This work is notably predicated on the assumption that the mechanisms producing today's problems are somewhat continuous with the mechanisms that will produce tomorrow's. It is possible that actually things just won’t scale, or that crunch time will involve very different paradigms. It is also possible (likely, even!) that more powerful AIs will exhibit qualitatively new challenges that don’t have easy analogues, like advanced forms of situational awareness. This cluster of work, by construction, can only attack problems with some degree of continuity, so it’s not as much of a full package as something like AI control.

Still, that's only a limit on how much optimism it can give us: there's still plenty of pessimism to try to squeeze out. Put another way, I think we should plan for problems scaling better than solutions. In particular, if we are headed to a world of abundant automated engineering, it could be useful to find more conceptually ways of red teaming how our existing approaches might break at a slightly higher level of abstraction. And if our plans can survive all that, I'll feel pretty good.

Thanks to JK, CG, NN, VK, JB, and DA for many comments and much discussion.

1. ^

   In practice these aren’t perfectly separable — lots of current control work is based on making model organisms, model organism work relies on prosaic techniques, etc

2. ^

   This point heavily cribbed from Stanislav Fort’s excellent essay on adversarial attacks as a baby version of alignment

3. ^

   Not adjusted for funding inflation

4. ^

   If you think ELK sounds solvable, I encourage you to take a crack at it — I think it’s good practice to ponder, and also if you do end up solving it we can all finally go take a holiday

5. ^

   There has also been some work explicitly on ELK in current models — see e.g. this attempt to make model organisms

6. ^

   Most notably, it seems like current AIs have extremely detailed models of different writing styles which simulated characters have some capacity to access

Discuss

tl;dr Functional Decision Theory does actually give the right answers. People who construct scenarios where it doesn't, or take umbrage at the idea of "fair problems" are falling foul of symmetry arguments, and would never apply this level of scrutiny to any other system of making decisions.

Intro

Bentham's Bulldog recently posted an attempted takedown of Functional Decision Theory on LessWrong. This is probably the second bravest post I've seen someone in the EA/Rat sphere post on LessWrong.

His first argument was that FDT is not mathematically well-defined, because logical counterfactuals are not well-understood and, as he argues, can never be well-defined. I don't know enough about the state of the logical counterfactual research, so I'll leave that to a pro decision theorist to explain.

His second argument was that FDT gives the wrong answer sometimes. I think that he skips up and down different levels of demand for rigor, when talking about different decision theories. I think FDT beats CDT, or at least ties, at basically every point on the spectrum between totally abstract and totally practical.

Decision Problems as a Tower of Assumptions

The standard jumping off point for decision theories is the set of fair problems with perfect information about the overall scenario:

1. You get to make decisions
2. Various decision theorist trickster gods such as Omega get to perfectly simulate you
3. You know, ahead of time, everything about the situation including what kinds of simulations you might be put in
   1. This doesn't mean you have perfect information about which of two identical situations you're in, once the situation starts, but it does mean you can have a well-calibrated Bayesian prior over it.
4. The agent can only simulate your actions, and doesn't have access to information about your decision theory

This includes the famous Newcomb's Problem[1]. It's worth working through Newcomb's problem in an FDT language, since that will be instructive for cases in the future. FDT reasons through problems as follows:

• In this world, there are two instances of FDT, which are both given the same input (a decision theorist trickster saying "I am Omega, this is Newcomb's problem) so they both have to give the same output as well
  • If they both choose two-box, then the reward is $1,000.
  • If they both choose one-box, then the reward is $1,000,000.
• Therefore, the optimal output for FDT, is to one-box
• Therefore [outputs one-box]

Newcomb's Revenge

Bentham's Bulldog has brought up Newcomb's Revenge[2], which does not fall into this class. Why not?

Let's try to set Newcomb's Revenge up using the previous ruleset.

• In the world of Newcomb's Revenge, there are two instances of FDT which affect the world, one outside of Omicron, and one inside of Omicron.
• Both of them can decide independently, because the simulated one inside Omicron sees a decision theorist trickster God saying "I am Omega, this is Newcomb's problem" the one outside Omicron sees a decision theorist trickster God saying "I am Omicron, this is Newcomb's Revenge".
  • If both instances output one-box, total reward is $0
  • If Omicron instance outputs one-box and Omega instance outputs two-box, total reward is $1,000,000
  • If Omicron instance outputs two-box and Omega instance outputs one-box, total reward is $1,000
  • If both instances output two-box, total reward is $1,001,000
• The optimal choice is for both instances to two-box, since this gets a reward of $1,001,000
• Therefore [outputs two-box for Omicron, outputs two-box for Omega]

But wait? I thought FDT chose one-box in Newcomb's Problem? Well, in this case we've given it perfect information! It knows it's in Newcomb's Revenge world, so it's changed its answer!

In order to get FDT to fail in Newcomb's Revenge, we need the simulated FDT agent to believe that it's in the original Newcomb's problem. This drops rule 3.

Why Not Allow Deception?

The problem with allowing deception is that the class of problems with deception does not have any winners in terms of decision theory! Roughly, if Omicron can trick you, then for any problem mjx-msub { display: inline-block; text-align: left; } mjx-c.mjx-c1D44C.TEX-I::before { padding: 0.683em 0.763em 0 0; content: "Y"; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c1D439.TEX-I::before { padding: 0.68em 0.749em 0 0; content: "F"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c24::before { padding: 0.75em 0.5em 0.056em 0; content: "$"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c1D449.TEX-I::before { padding: 0.683em 0.769em 0.022em 0; content: "V"; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } where a decision theory believe's it's in situation , there exists a symmetric problem , which is identical but with the payoff matrix reversed.

We can a problem we might call "Newcomb's Apology" where Omelette simulates your decision in Newcomb's problem, then puts the $1,000,000 dollars in the box iff you one-box. In this case, CDT gets $1,000 and FDT gets $1,001,000, exactly parallel to Newcomb's revenge where CDT gets $1,001,000 and FDT gets $1,000.

"Unfair Problems"

There's an equivalent issue where, if you drop rule 4 and let the decision theory trickster gods make decisions directly based on your decision theory algorithm, then no decision theory can possibly win either. These are typically called "unfair" problems, which sounds like a cop-out, but I don't think it is. As with the total-lying problems, there's an exact symmetry where if, in one problem, Omnomnom shows up and puts $1,000,000 in the box iff you use CDT, then there exists a corresponding problem where Ompalompa shows up and puts $1,000,000 in the box iff you use FDT. In this case we don't even need a first box.

The reason for rules 3 and 4 is that they provide too large of a space. Dropping rules 3 and 4 runs us into the territory of no-free-lunch theorems. The problem with no-free-lunch theorems is that if you buy into them, you'll quite often give up on lunch forever, and go hungry. As one example, there's the theorem that appears to prove that no brain can ever exist and intelligence is fake, which rules out [gestures vaguely at everything]. A good rule of thumb is that when you run into a no-free-lunch theorem, you need a prior.

Priors and Weightings

If you don't want to think in terms of Bayesian Priors, how about we think about getting an overall "score" for each decision theory by giving a weight to each possible problem, and then adding up the scores, multiplied by the weight. Let's say that our weights have to add to 1, without loss of generality. For the problems we've looked at, we'll get the following scores:

Problem

Weight

FDT Score

CDT Score

Newcomb

$1,000,000

$1,000

Revenge (simulated agent deceived)

$1,000

$1,001,000

Apology (simulated agent deceived)

$1,001,000

$1,000

CDT <3 (unfair)

$0

$1,000,000

FDT <3 (unfair)

$1,000,000

$0

Now we might say that revenge and apology are in some sense symmetrical, and that CDT <3 and FDT <3 are in some sense symmetrical. If we do that, we ought to enforce and . So then the only difference beteween the two comes from . FDT still wins so long as we respect the symmetry of the system!

Isolated Demands for Rigor

Now I've gone through a huge amount of stuff here, because it's worth going through the maths to justify an intuition that everyone already has: mostly focus on scenarios where you have a decent model of the world.

As an example: I don't torture people, because I think it's wrong, because of evidence. Now it is possible to construct a world in which this ethical rule is false: suppose that actually, every person except me has a four-dimensional wire in place of their brain, which goes to a five-dimensional daemon consciousness which actually loves being tortured and just role-plays as someone who dislikes it. Obviously this is stupid and we don't take this into account in the real world. Obviously we mostly evaluate theories in the worlds where people are basically correct about the world.

(And of course, the possibility of that is immediately cancelled out by the possibility of 5-d daemons who hate torture and are role-playing).

Now you might say, OK, but Newcomb's problem is pretty contrived. You might bucket the scenarios like this:

• Realistic: {torture is bad for normal reasons}
• Unrealistic: {Newcomb's problem, Newcomb's revenge, Newcomb's Apology, ..., torture is good because of 5-d daemons, torture is bad because of 5-daemons}

In which case ooh boy do I have some examples for you.

Parfit's Hitchhiker

Suppose you're dying of thirst in the desert. Someone comes along and offers to drive you to the nearest town, but only if you give them money to cover their detour. You don't have money on you, but can take some out when you get there. They will only take you if they think you'll pay. Do you pay?

FDT says yes, CDT says no. EDT (Oh you thought you were getting off scot-free, EDT?) also says no. If the driver is a good predictor then FDT lives, CDT and EDT die in the heat of the sands.

Now, you may say, the driver is probably not a great predictor of me. FDT was originally invented to reason about AIs, who could inspect each other's source code, and probably can tell what decision theory each other are following. The random driver cannot do that, but they can get some information about you! People are constantly leaking information about what rules they follow, in some cases by posting long blogposts which tell anyone reading them "I DO NOT PAY IN PARFIT'S HITCHHIKER AND I GIVE IN TO BLACKMAIL".

(To be clear, I think that Bentham's Bulldog probably would pay in Parfit's Hitchhiker, even if there were no consequences to not paying, but for reasons not well captured by utilitarian CDT)

CDT does not, in general, have a good way to pre-commit to actions. Nor does EDT. Since pre-commiting to actions is extremely common in real life ("I will hire you if and only if I think you won't slack off and cause trouble for me") this is a huge deal which favours FDT over EDT and CDT.

Updateless decision theory does, and indeed Bentham's Bulldog mentions it as an alternative to CDT. Updateless decision theory has a bunch of its own problems, which I won't go into here, since this post isn't supposed to litigate between UDT and FDT, but rather to show the non-validity of a very particular argument.

(I think there's also a weird set of self-modifications that a CDT agent might perform, which switches it into a thing called son-of-CDT, which is a bit more like FDT but not quite the same, but I have honestly only seen this come up once and I think it's deep MIRI lore)

Summary

• If we limit ourselves to fair problems without deception as to which problem you're in then it makes sense to say that one decision theory is better than another
  • And FDT wins in lots of these problems
• If you expand your universe to unfair problems, or allow a more general notion of deception, then you can construct arbitrary problems where any decision theory wins
• If you then apply a metric to these problems, by symmetry, only the component of fair problems with limited deception matters
  • So we're back to FDT winning
• If you actually care about non-contrived real problems, then the most common issue which comes up which is decision-theoretically relevant is pre-commitment
  • But FDT and UDT (and a few weird others) are the only systems which can pre-commit to things

1. ^

   Well-known. Omega offers you the choice to take or leave $1,000, and, if it predicts you will leave the $1,000 on the table, gives you $1,000,000.

2. ^

   Omicron offers you the same choice as Omega, but gives you $1,000,000 if you take the $1,000 in Newcomb's problem. You can still take or leave the $1,000 but this doesn't really matter at all, you might as well take it.

Discuss

Epistemic status: trying to articulate a big idea which I feel is important but underexplored, partly because it is hard to frame clearly - may not be framing it clearly yet!

Agency, both natural and artificial, is very important. Understanding agency allows us to model our own behaviour and that of others, and it is thus one of the most predictively useful concepts we have at our disposal. In its ordinary, folk-psychological sense, agents are ‘like us’ in important behavioural respects, more or less, meaning we can use thoughts like ‘what would I do if I were them’ to good effect.

However, that does not mean agency is a natural kind. The truth is that we are not the people we imagine ourselves to be, and neither are the humans, animals, complex systems, or even inanimate objects we are prone to thinking of as fellow agents. We are, in fact, nothing but a bunch of hierarchically ordered biological processes in a trench coat. Our behaviour is not neatly determined by our thoughts and ideas, but by a complex mesh of impulses, desires, emotions, and heuristics that are often no less confusing (even, or especially, to the highly intelligent and introspective among us) than those mysterious entities we call other people. Nor are increasingly agentic AIs much of an improvement. While early agents trained directly from reinforcement learning may be conceptually simpler than we are, because their policy function is directly optimized into their weights, systems that simulate agency as an emergent phenomenon from some other process, such as next-token prediction, are just as complex and messy, combining their base model’s stochastic inclinations with the way that their simulated personas move them through semantic space. Agency is a construct that we have developed to help make sense of this mess, but it is only a lens through which we view the world. Indeed, there are many agentic lenses people have constructed, and the kind of lens you use can profoundly influence how you view the world, and yourself.

When engaging in practical work, this sort of claim, that ‘[x] is a construct and the reality is a lot more complicated’, can seem unhelpful. Of course, we all know this, but the point is that agency is a very useful and predictive construct (as are many others, from money and weeds to temperature and species), and we can surely make more progress with it than without it. Obviously, I agree.

The problem is that when we start talking about agents as a natural kind, a fundamentally different type of thing from non-agents in our ontology, we often smuggle a kind of teleology in via the back door. We also assume that our simplified model for how agency works, roughly goal-directed utility maximization, describes what ‘real’ agents do. The fact that all the actually existing agency we see, including our own very imperfect muddling through, isn’t like this only goes to show its imperfection, its pseudo-agency if you will. The alternative I would advocate for is viewing agency as a naturally emergent phenomenon that is built up from other phenomena (such as boundary maintenance, self-modelling, information processing, and so forth) and could continue being built up ad infinitum without necessarily being drawn into such an ideal.

Of course, there are arguments for why this teleology is justified. The best known is that agents whose preferences don't conform to utility maximization can be ‘money-pumped’ (led to pay a cost only to end up where they began) and so dominated by those that do. However, the theoretical basis for such claims is more shaky than is often assumed. These arguments assume preference completeness (that for any two options an agent prefers one or counts them equal) and derive a utility function from it; they never show that agents must have complete preferences in the first place; and an agent can escape the money pump without them. Suppose, with Parfit, that I hold some goods as only roughly comparable: I might prefer being a good writer to a bad one, and a good lawyer to a bad one, yet have no preference between being a good writer and a good lawyer. That wouldn’t necessarily make me exploitable, so long as I spot the money pump game and avoid playing. I need only refuse to trade my current career for any alternative that isn't strictly better (not merely roughly comparable), which breaks the cycle without ever ranking writing against lawyering. One might object that a policy like this just is a utility function under another name, as it still leaves the agent with a set of preferences that is representable as maximizing something. But "representable as maximizing something" is nearly trivial here, since almost any behaviour qualifies. What the threat of domination would actually need to force to justify this teleology is a single cardinal ranking of outcomes, and that is precisely what incomplete preferences withhold.

There are also practical reasons why AI safety researchers often wish to defend this view about agency - it plays a central role in some of the most classic and widely respected arguments for why AI is dangerous, such as Bostrom's Superintelligent Will. Indeed, some of the best critiques of AI risk consist largely of questioning these arguments. Yet, these are hardly the only arguments for why superintelligent systems could pose a threat to humanity, and there are more reasons for wanting to explore the fundamental nature of agency than trying to show that AI risk research may be misguided. In any case, it is certainly not my view that alternative views of what agency is will render AI safety trivial or easy!

However, there are reasons why a more thorough and grounded, and less teleological, approach to thinking about the nature of agency could be helpful for developing safer AI. One is that humans' conception of our own agency and that of others influences how we behave, and it is reasonable to assume that the same is true of AI. Consider the following possible people. One conceives of agency as a false construct tying them to an unsatisfactory life of striving that they are endeavouring to dissolve through rigorous meditation and cultivating love for the inherent worth of all things. The other believes they are homo-economicus incarnate, and the only thing stopping everyone murdering their neighbours for the rings on their fingers is well-designed social incentives. I’m not saying either of these is inherently more aligned or easier to align. However, I also don’t think either is more correct about the nature of agency or more of an agent in how they embody it. What I do think is that, if I were trying to get these people to be nice to me, I would probably go about it quite differently and expect rather different results from them. Of course, the reality for most people is even messier than these toy examples, but our social norms and behaviours are surprisingly well adapted to handle this complexity. I think that is one reason why our everyday moral judgments are often more useful in social alignment than ethical theories.

So, before insisting goal-directed utility maximization is the only form advanced AI could take, I think it is perhaps helpful to make sure we are not obscuring a messy reality of actual AI agency with our, often teleological, assumptions about what it should look like. And perhaps by influencing the kinds of agency AIs go on to develop, we can build another lever to help move us away from the worst of the danger.

Discuss

p(doom) is a shorthand for some important bits and a way to notice a disagreement to double-crux about.

If you work on AI capabilities at a frontier AI company, I might ask you for your p(doom).

If it's less than 1%, I know that you're probably not familiar with the arguments, or you're maybe dumb in some ways, and will sometimes talk to you about what the situation really is.

If it is 80%, I know I should talk to you about the actions people in your position should be taking; we have disagreements about best ways of achieving goals/lab politics/etc., not about the large-picture situation.

p(doom) is not a very useful number to talk about in a conversation between two aspiring rationalists generally familiar with the basics. The things people should talk about instead are: How does the world survive? How likely are different things to happen in the future, maybe given that other things happen? etc.

But most people are not aspiring rationalists, and have never heard of any of our arguments, and are not aware of the levels of worry of various people in the field.

Communicating the importance of paying attention to the arguments by honestly answering the question "so, how likely do you think AI will be to kill everyone" is useful.

Asking someone for the probability they'd assign to AI causing humanity's extinction is useful, too, to figure out how familiar they are with the topic. Their answer also allows asking an open-ended "why?" and getting a more detailed explanation of the view of whoever you're talking to.

It is also useful to talk about, e.g., Geoffrey Hinton's stated beliefs about the probability of extinction, as a reason to pay a lot of attention to the actual arguments: it is not common that a godfather of a field regrets his life's work and thinks the consequences of it have >50% chance of killing everyone on the planet.

"What do you believe" is a good starter for a conversation about "why do you believe that". "A Nobel-winning scientist believes in a high chance of *that*" is also a good starter of a conversation about that. The conversation doesn't have to be, and usually isn't, about the numbers; it should be about the reasons and the models; calibration is not interesting, in this context, the thing that is interesting is why AI is or isn't likely to literally kill everyone.

I think there are many properties of p(doom) as a meme that are bad: people who are familiar with the meme might mean different things by it and there are weird outside-view cascades, keynesian beauty contests, and misguided attempts at aumann agreement games that make the whole thing a bit worse (I have to note, however, that this would've been the case even if no explicit probabilities were stated). It migth make people unwilling to have or share beliefs others don't, if it's easy to see what the accepted beliefs around them are. Because of that, I think it is not particularly worth spreading "p(doom)" as a meme or a concept.

But I do think that with people aware of the meme, it is slightly faster to figure out where everyone stands and find cruxes; and with people not aware of the meme, the general question of the probability can be useful to answer, to talk about, and to ask.

p(doom) (or some expanded version, like "How likely do you think it is that AGI would cause extinction of humanity?") points more at the level of seriousness of the threat, compared to other questions that might discuss something much broader and less focused.

Better memes would be great; but this one is not *that* bad.

(My p(doom) is probably around 80%. The 20% come mostly from the US government making sure that no one on the planet creates superintelligence until we know how to do that safely, and a bit from the possibility that I'm completely wrong about everything. I'm widely uncalibrated on such things, though.)

Discuss

I just got back from a few days of workshop and backpacking but am not in much of a mind to talk about it, so I ask you this question that is very far from my current mood, but which I was reminded of by the demands of relaxing upon returning home:

Why does Netflix—a service that recommends what to watch based on one’s observed tastes, and which has separate profiles for each member of one’s household—not offer recommendations for pairs or groups of people, who individually have profiles?

I don’t see how this can be hard, if they have good recommendations for both individuals. For instance, they could just check everything suggested above a certain bar for Alice and see if it’s also suggested so for Bob. Even something very janky would seem to be better than the pair doing something equivalent manually (for instance, looking over Alice’s recommendations, and parsing what each one is well enough for Bob to judge if he would like it). But I bet they can do much better.

Group recommendations would also seem to be extremely useful. I’d guess a large fraction of the time that a person wants to watch something, they want to do it with someone else. And furthermore it seems substantially harder to figure out what two people would like to watch together than just oneself, perhaps because neither person has great access to the other person’s mind, so anything that fares well on an intuitive glance then has to be explained to the other person. I’m not sure.

Is there some good reason they don’t? Am I wrong somehow? Do people use Spotify Blend as much as I would have predicted?

Discuss

Summary and Introduction

How should the world slow down AI progress if it ever decides it needs to? If you ever see substantial evidence of catastrophic risk emerging, social instability caused by mass unemployment occurring, or a software intelligence explosion (SIE) beginning that causes progress to outpace our ability to adapt, you could decide that it’s prudent to slow down progress to have more time to prepare and adapt to coming capabilities.

While there has been a lot of attention devoted to the question of whether you should slow down, thus far not a lot of attention has been devoted to the question of how you would slow down, and the actual instruments that you have available to cause a slowdown. Some commonly discussed mechanisms, such as token taxes, datacenter moratoriums, and 6-month training run pauses would all have significant downsides. This makes them, by themselves, unattractive as instruments to slow down AI progress and address societal or political concerns about AI.

Instead, if you are forced to slow down, the most effective and least harmful approach would be twofold. First, to address catastrophic risks or a SIE, I’ll recommend a layered set of restrictions to slow down the rate of algorithmic progress by limiting the amount of compute that AI companies can pour into R&D internally. The first restriction would be a hard cap at a certain threshold of total R&D compute. The second would be a progressive tax below that threshold. And finally these two restrictions would be accompanied by a backstop in the form of a cap on training compute for individual training runs, to provide extra assurance against evasion. The hard cap on R&D and training compute would be targeted at risks that could arise more suddenly, such as misalignment and catastrophic misuse risk. And the progressive tax would be targeted towards risks that rise more smoothly with respect to capabilities (such as broader societal harms that require time to adapt to).

Second, to address mass unemployment concerns specifically, I’ll propose a capability-gated tax on AI deployment, as the intensity of deployment of powerful AI systems would be tied to displacement, and so metering inference should allow you to control the velocity of economic displacement. Both of these approaches should be designed as dynamic, conditional instruments that are able to be updated in the face of new evidence.

In this post I’ll sketch out some problems and desiderata for slowdowns, particularly through the lens of a critical window of capabilities where your risk-reduction efforts are most leveraged, as well as the concept of an overhang. I’ll argue that slowdown mechanisms should move us slowly through the critical windows of capabilities, avoid being overly blunt, and be dynamic so they can be tuned as evidence emerges. Then, I’ll present a taxonomy of policy levers and targets in the AI tech stack, and explain why the mechanism proposals need to be targeted at the layer of the tech stack that corresponds to where the risk comes from, and use the correct lever to match the harm structure of the risk you’re attempting to address. Finally, I’ll explain how these considerations motivate the above proposal, and briefly touch on what trigger mechanisms could be used as tripwires for when slowdown mechanisms should be implemented.

To be clear, I’m uncertain that slowdowns currently are or will ever be desirable, yet I think it’s an important question to interrogate because there may be scenarios where the risk is large enough to merit slowing down, or where progress will grow so fast that it will truly outpace our ability to adapt and prepare. One specific objection worth touching on: what about China?

Is it even worth thinking about slowing down if China won’t slow down?

Yes, for two reasons. First, the Overton window may shift in the future sufficiently to allow for an international agreement to be formed. Second, the gap between the US and China may widen enough to allow for the US to unilaterally implement these instruments.

To be precise about what a slowdown could get you, you need to think about how leveraged your actions are at different levels of capabilities.

Critical windows of capabilities

One important concept to analyze the problem of slowdowns through is that of a ‘critical window’ of capabilities, which is a regime of capabilities where preparation is most leveraged, but before you actually enter meaningfully into a dangerous regime. Different risks may have critical windows at different points, and there may be multiple critical windows even for the same risk, but the concept is useful for all risk types.

For unemployment, for example, the critical window(s) could be just as displacement is starting to occur, such that political leverage is highest to prepare for coming automation and implement measures like unemployment compensation or UBI/Universal Basic Capital. Additionally, seeing the capabilities available as they are causing unemployment to rise would help individuals and society broadly prepare for the coming changes.

For misalignment risks, the window could be where alignment research efforts are most leveraged. This could be because you’re closest to the capabilities you’re concerned about and so you’re most able to run experiments aimed at addressing them, or create model organisms of the type of risk you’re concerned about. This would also be the period where AI is most able to boost the productivity of alignment, resilience, and control efforts.

From this point of view, what you want out of slowdown mechanisms is to maximize the amount of time you spend doing useful work inside these critical windows. Of course, there isn’t certainty about where these critical windows start, and in truth they likely aren’t discrete windows but rather continuums where your efforts are more and more leveraged the closer you get to the dangerous capability. It will likely be uncertain whether you are in a critical window even when you are in it.

Spending time within critical windows should also not be seen as universally good, as it can actually increase certain risks like misuse, as it would raise the possibility of model weight theft, or proliferation of capabilities beyond frontier actors. Lingering within the window without doing useful work to prepare and adapt could thus be net negative.

Why frame this post around slowing down within a critical window, and not pausing as some advocates propose?

The uncertainty of the timing of critical windows means you can’t time a pause precisely, which is one of the reasons an absolute pause is unwise. Pausing too early would freeze you at a time where efforts aren’t particularly leveraged, and it would also prevent you from gaining more information about where the critical window is. Additionally, as we’ll see in the next section, pausing and unpausing before you enter a critical window may have a neutral-to-negative impact on the amount of time you’ll have later during the critical window. Instead, it is better to have dynamic braking mechanisms that allow you to modulate how severe the slowdown is depending on the available knowledge you have. Dynamic mechanisms might look like caps that rise over time at a certain rate (which can be modulated), or taxes with a variable rate.

A critical window can also shift upwards over time. If you spend sufficient time inside a critical window preparing while inside a pause, you could hit diminishing returns at that regime of capabilities. To continue improving societal preparation/adaptation, you’d need to let capabilities advance before once again slowing. This is a further argument in favor of dynamic instruments, as you want to be able to move upwards in capabilities slowly in such a scenario.[1]

In truth, pausing and slowing down exist in a continuum of braking intensity, and shouldn’t be seen as mutually exclusive. The best policy may involve slowing down to glide into a critical window, then locking in a pause for a while before allowing capabilities to grow once again but slowly. Crucially though, they both suffer from a similar cost: while your foot is on the brake, you are accumulating an overhang.

Slowdown dynamics and the overhang problem

Let’s use a concrete proposal of a 6-month pause on training runs above a certain capability to illustrate how short pauses can actually have a neutral impact.

As you can see, in this scenario temporary pauses or slowdowns don’t increase the amount of time you get inside the critical window, which is where your time is most valuable.

However, the truth is even worse than the plot above suggests, as in real life capabilities wouldn’t simply be shifted to the right, as the curve would instead ‘snap back’ due to ‘overhangs’.

Because AI is the apex of an enormous tech stack that is constantly pushing forwards, attempting to slow down progress at any given layer of the tech stack risks creating ‘pressure’ in the previous layers (hardware, algorithms), as its technology progresses but isn’t implemented fully at the next layer up. This is called an overhang. If the slowdown mechanism is short enough, when it is released, the built-up pressure snaps back, creating faster progress than you would have seen otherwise.

In the above scenario, while you’re restricting training runs above a certain capability, hardware progress continues, as new technology process nodes are introduced, AI chip designs get improved, and AI compute infrastructure generally gets better, etc. This reduces the cost of a training run of the same size. Additionally, algorithmic progress also goes on, reducing the cost of reaching the same capability. So when one day the capabilities cap is lifted, all of a sudden with the same (or more) amount of money, you can run much larger training runs that give you even greater capabilities. Combined with this, you now have better algorithms which give you access to more powerful capabilities for the same amount of compute. This increases the expected returns from training runs and thus raises the amount of money that can be invested into running the training run. Both of these factors, combined with other improvements that happen in the tech stack during the pause, lead to a massive snapback in capabilities, at a faster rate than would have happened without the cap.

The shifted curve doesn’t snap back all the way back to where it would have been pre-intervention, but if the period of snapping back (i.e. having a higher slope than the no-intervention curve) coincides with the critical window, you’ve actually made things slightly worse.

There are a few factors that influence how much worse you’ve made things by implementing a temporary slowdown before entering the critical window. The longer the slowdown lasts, the less time you have inside the critical window (as long as you release the slowdown before entering the critical window). The farther the entrance to the critical window is from the end of the pause, the longer time there is for the overhang to exhaust itself and for your impact to return to neutral. And finally, how severe the pause or slowdown is also impacts the severity of the overhang you deal with after you lift the slowdown.

The main point I wanted to make with the above plots is that to be effective, a slowdown must last long enough so that you stretch the curve to the right instead of merely shifting it, such that the slowdown overlaps with the curve passing through the critical window region.

For example, let’s take a gradually increasing training compute cap that still allows training compute to grow, but at a slower pace.

Commencing the slowdown before entering the critical window, and extending it for the duration you are inside it, increases the amount of time you have within it significantly. In the plot, I illustrate the slowdown ending entirely at some point. Among other assumptions, this takes as a given that there is saturation of preparation efforts, meaning that spending enough leverage-weighted time eventually ‘solves’ the risk you’re trying to address, and you can safely release the slowdown without risk.[2]

In reality the critical window would be a gradual continuum instead of a discrete window, given the uncertainty about when you are inside the critical window and when you’ve saturated your preparation efforts. Similarly, the slowdown should be dynamic, and change in kind with how leveraged your efforts are at different capabilities.

Here’s a plot with these new assumptions, plus a depiction of what the overhang would look like over time.

Here we see that there are two variables you must manage: maximize the leverage-weighted time you get by slowing down, and minimize the size of the overhang you accumulate. The first is obvious, but why do you need to minimize the size of the overhang? If you wait to release it until after your preparation efforts have saturated, what’s the problem? The problem is that the size of the overhang is correlated with enforcement difficulty. The overhang growing means that it becomes cheaper to reach the capabilities ceiling you are trying to enforce, and thus more actors become capable of reaching it. At the limit, this could make enforcement infeasible, as rival countries or small actors become capable of reaching past the ceiling regardless. Additionally, the larger the overhang is, the greater the incentive there is to defect and break past the ceiling. This is because you have a greater capabilities jump you could have access to if you defected.

If you let the overhang grow large enough, you’d be left with two alternatives. Either you raise your foot off the brakes and let capabilities keep advancing faster, or you target the very inputs that are causing the overhang to grow. This would likely be very costly, as stopping inputs like hardware progress or algorithmic improvements would require very significant enforcement efforts, and have a very large opportunity cost. Imagine the economic consequences of preventing more fabs from being built, or the political implications of trying to restrict the research that companies, startups, and academics can do.

There are two opposing factors determining how large the overhang gets while inside the slowdown. The first is that the longer you are in a slowdown, the farther away the unconstrained maximum capabilities that could be reached gets. This is because improvements lower down the tech stack, like hardware improvements, continue throughout the slowdown period, and unlock larger jumps when the slowdown is released. The second factor is that unlike a pause, the slowdown does allow capabilities to grow, releasing some of the overhang pressure and reducing the distance to the unconstrained maximum.[3]

In this whole analysis, we’ve been looking at AI capabilities curves that rise exponentially, without considering when or how that process might naturally slow down. Progress may naturally taper off, due to things like physical growth limits, algorithmic insights getting harder to find, nearing the theoretical limit of algorithmic efficiency, and more. While there’s no guarantee you will be reaching this point anytime soon, one possibility is to hold on to the slowdown mechanism until you reach this point. This is one potential resolution to the overhang problem, as it would naturally dissolve instead of causing a snapback of fast progress. For example, have training run caps that rise slowly until they reach the point beyond which it is infeasible to train (e.g. latency walls).

Now that we’ve established what slowdowns should accomplish, what should they look like in practice?

Slowdown mechanism taxonomy

To examine the landscape of options that are available for slowing down AI, you can consider 4 possible targets in the supply chain, and 3 possible levers for deceleration.

At a high level, AI progress can be thought of as a technology stack that you can break up into deployment/inference, training, AI R&D, and compute infrastructure and below.[4] It’s worth noting that final training run compute composes only about 10% of total R&D compute[5], but it is worth talking about as an independent target.

• Most of OpenAI’s 2024 compute went to experiments | Epoch AI 

Policy-wise, the 3 available levers to affect each tech stack are hard caps, taxes, and regulation.[6]

Deployment/ inference

Final training runs

AI R&D

Compute infrastructure and below

Hard caps

e.g. <1e26 inference FLOP per year per model above a certain capability

e.g. <1e28 training FLOP on any single training run

e.g. <30M H100-eq can be devoted to R&D per company

e.g. <20M H100-eq per company

Taxes

e.g. 100% tax on model pricing for frontier models (token taxes)

e.g. a progressive tax rate that rises as you rise in training FLOP

e.g. a 100% tax on R&D compute spend above $50B.

e.g. 50% tax on AI chips

Regulation

e.g. prohibiting specific use-cases like LLMs for legal advice

e.g. safety case approval being required for training runs above a certain threshold

e.g. datacenter security requirements for R&D above a certain capability threshold

e.g. requiring FlexHEGs or datacenter moratoriums

Levers

Conditional regulation could be all that is needed in theory to manage the transition to transformative AI, and slow down progress as is necessary to address any risks. However, risk-specific regulation may not be sufficient or desirable relative to caps or taxes. This is because different risks may be too hard to define and codify into regulation, given the large uncertainty that exists about them. In the case of alignment and catastrophic risk, for example,  it would be a significant challenge to formally regulate because there are a wide range of opinions about the nature of the risk (AI takeover, gradual disempowerment, concentration of power), and what would constitute sufficient protection against those risks. A general slowdown could be more favorable than a contested fight about what constitutes safe enough to deploy/train.

Taxes may seem like a less natural fit than the other two levers, given they don’t directly control the quantity of interest, but there’s actually a few reasons to favor them over caps, or at least use them as complements. From a Pigouvian point of view, what you’re trying to achieve with a tax is to internalize some risk or harm that is not being priced in, and in that sense you’re not trying to aim for revenue maximization or minimize market distortion. Instead, distorting the market in specific ways is the point of taxing with the purpose of slowing down.

One frame that the question of caps vs taxes for the purposes of controlling an externality can be interpreted through is a marginal harm point of view. If there are sharp discontinuities in harm from the controlled quantity (e.g. inference compute, or training compute), then that favors caps. If harm grows more smoothly, then taxes are favored. Under this view, hard caps may be a better fit for catastrophic risks and misalignment, where it either causes catastrophe or not, while taxes may be a better fit for unemployment and other societal harms, as those should scale more smoothly with controlled quantities. On top of this, taxes are closer to societally optimal in some ways, as it allows the highest value AI activities to continue, while lower value activities are priced out. As opposed to a cap where it might be arbitrary or subject to some other allocation mechanism less effective than a market.[7] Taxes also come with an overhang pressure release valve, as the willingness to pay higher taxes naturally rises over time, instead of being stuck at some fixed cap. The flipside of this is that the brake erodes in severity over time. Finally, while not the main purpose of them, taxes do generate revenue which can be used to actively reduce risks through adaptation and preparation.

Caps on the other hand are favored because they provide direct control over some quantity. If you know you want to prevent training runs above 1e27 FLOP, rate-setting a tax would be very difficult, as you would be one step removed from directly limiting the dangerous activity. Setting them too low might have little impact, and setting them too high might lose the benefits of using taxes instead of caps in the first place. While this is true, it’s also worth noting the counterpoint, which is that just as caps give you more certainty about a given quantity, taxes give you more certainty about the economic impact on industry. Setting a cap that is too aggressive may decimate the industry in a way you may be unwilling to risk, while setting it too high could have little impact in the short run, until the companies hit that wall. A final point in favor of caps is that they may be easier to enforce and monitor as they are directly tied to physical quantities.

Ultimately, I think the best approach will be to use a mix of all three levers to complement each other and make up for their individual limitations. An example of this could be a cap-and-trade system popularized in the carbon emissions case, although AI differs from the carbon case in ways that make this a less attractive option in particular, mostly because FLOP isn’t quite fungible in harm caused in the same way that carbon is. While a ton of carbon does the same amount of harm no matter where it is emitted, a marginal FLOP used for frontier AI R&D is not interchangeable with, say, a FLOP used in a startup.

To explore the desirability of different targets in the tech stack, I’ll step through a few concrete proposals that have been raised as possible mechanisms for slowdowns, explain why I think they’re not ideal policies for all the risks I’ve been considering, and then explain what I think is the best mix of instruments to deploy against which targets.

Concrete proposals

Token taxes

A flat token tax that uniformly hits all AI deployment would be ineffective at reducing some risks, and an imperfect way to target some others. With risks like misalignment or catastrophic misuse, most of the harms come from more capable models, rather than more widely deployed AI, meaning you aren’t addressing the risk directly by targeting deployment.[8] In the case of unemployment concerns, while you are directly targeting the layer tied to the harm, it’s imperfect because it uniformly hits deployment that may usefully augment labor instead of replacing it, as well as deployment of more capable models that may actually cause the unemployment impacts. A capability-gated deployment tax (which I’ll recommend later) would fix this discrepancy.

Datacenter moratoriums

Targeting datacenters is unattractive because this would impact deployment as much as it would training and R&D, which is not ideal. Additionally, it would impact diffusion of lower-capability AI, which would mean you lose out on that benefit to the economy as well as an opportunity to stress-test institutions and culture. You would also hit AI applications that are not general-purpose, like AlphaFold, that have economic and scientific benefits while not causing that much risk.

6-month training run pauses

This is perhaps the most famous attempt at a policy to slow down progress, as seen in the Future of Life Institute's ‘Pause Giant AI Experiments’ open letter that was created after GPT-4 was released that attempted to ensure no model more capable than it would be created for 6 months.

As seen in the Slowdown dynamics section, the largest problem with this proposal is that while it would gain us time now, it would not give us more time in the future at some critical window of capabilities, where you have much more information about the relevant risks, and are much more able to make progress on them.

To be meaningful, any slowdown attempt would need to last longer. The attempt to target training runs is not misguided in itself, and a modified version would actually be successful at slowing down as capabilities are so correlated with total compute. The main weakness is that targeting training compute does not directly meter the speed at which algorithmic progress can continue (that would require controlling R&D compute, as I’ll argue for in the next section).[9] 

Recommended mechanisms

To best slow down AI given the possibility of catastrophic risks, mass unemployment, and a SIE, I think there are two instruments that need to be prepared. The first is a layered mechanism that targets AI R&D compute with a tax and a cap, as well as with a backstop in the form of a training compute cap. This instrument is best suited for slowing down AI in the medium to long term, and addresses risks from misalignment, catastrophic misuse, and SIEs. For the case of mass unemployment, the best option is capability-gated deployment taxation.

AI R&D compute restrictions

As you saw in the Slowdown mechanisms section, the amount of compute devoted to total R&D is about 10x the amount dedicated for final training runs. This is where most of the algorithmic progress occurs (outside of smaller innovations / optimizations like FlashAttention developed outside of the company).[10] Restricting R&D compute is the closest to a speed brake one can achieve. You can modulate the speed of algorithmic progress by limiting how much compute can be devoted to it, while at the same time restricting how much training compute can be devoted to single training runs (as training runs are part of R&D).

To construct the actual restriction, it is best to combine several different levers to achieve the best result. The first and most obvious restriction is to simply place a cap on the total amount of compute individual companies can devote to R&D in a given year.[11] This addresses threshold-structured risks that scale discontinuously with capabilities, so you want to stay strictly below a certain capability growth rate.[12] The second restriction is to place a progressive tax on compute regimes just below that cap, to address general societal adaptation, which scales more smoothly with capability growth rate. Finally, as a backstop, a cap on training compute, to prevent companies from dumping all their R&D compute into a massive single training run, and to directly regulate the capability-from-scale that can be achieved in the short run (as restrictions on R&D would only pay off in the medium to long term).

Additionally, you can offer companies an increase in the R&D cap conditional on approved safety cases for managing increased capability growth rates. This creates an incentive for companies to invest in safety research beyond commercial considerations, because safety work directly expands R&D compute budget.

To illustrate what this set of restrictions could look like, here’s a plot with the three restrictions on R&D compute shown as three different lines. The restriction on training compute isn’t shown as it’s a measure of FLOP, not FLOP/s. Also note that to prevent the tax from affecting small actors, the progressive convex tax only starts past a certain threshold of compute.

If designed correctly, the shape of the marginal tax rate curve could help modulate how quickly AI companies scale up the amount of compute devoted to R&D. As algorithmic progress advances, and better harnesses and deployment infrastructure is developed, the willingness to pay for more compute will naturally rise. This will help mitigate some of the overhang that will generate over time, and distort the market less than a cap which affects all companies that are right up against it equally.

One notable downside of targeting total R&D compute rather than training compute is that enforcement might be harder to implement, given that R&D compute is spread out over a lot more experiments, synthetic data generation, and other purposes, instead of being concentrated in a single job. This is somewhat ameliorated by the fact that compute usage must be carefully tracked internally within AI companies, so you can rely on their internal metrics and systems when implementing any restrictions.[13]

Even assuming you can get around evasion difficulties, there are some genuine edge cases where it’s somewhat unclear how to classify compute usage, such as synthetic data generation or internal usage not directly for AI R&D (e.g. for finance). While it’s hard to predict all these edge cases in advance, the general principle I would advance is that it should count under R&D if it’s compute or API spend that is controlled by the company, and whose outputs benefit R&D in some way.[14]

Capability-gated deployment taxes to manage displacement velocity

One harm that R&D compute restrictions aren’t perfectly suited to address is job displacement, as that is not directly mediated by capability growth rates, but rather by deployment intensity (though of course R&D compute restrictions could serve as a helpful complement to deployment restrictions, especially in the long run). Therefore, the best target for restrictions would be aimed directly at deployment to control how quickly AI diffuses throughout the economy in a way that causes unemployment. Specifically, you need to make a distinction between AI deployment that augments and transforms labor without causing mass unemployment, and deployment that does cause replacement.[15] An ideal tax policy would only disincentivize and slow down the latter, without affecting the former.

A possible assumption that one could make as to what differentiates the two is that the more powerful an AI system is, the more it will be tilted towards replacement rather than augmentation. In that case, a simple flat tax on all deployment would affect the less powerful AI systems too that would have only augmented labor, which is not what you want. Instead, a capability-gated deployment tax that grows as the capability of the AI system does would more narrowly target replacement effects.

While there is some reason to believe this assumption is true, you should also prepare for the possibility it is not as there are forces that push in both directions (automation may create new tasks, as well as automate existing ones). Therefore the actual policy should be conditional on the displacement vs. augmentation pattern you observe in the economy. If it is indeed true that replacement correlates with frontier capability, then a capability-gated inference tax is the correct lever. If instead it correlates with inference intensity broadly and not particularly with capability, a flat tax rate would be the correct lever. If you do not observe replacement even at high capabilities, then there is no need to implement this lever in the first place. Much like you want dynamic instruments that could be tuned in accordance with the evidence on relevant risks, you can get the best of both worlds by monitoring the economy for triggers that would help you determine whether displacement is occurring, and if it is, whether it is tracking capability or intensity.

There are two big problems with this recommendation that make it more tentative than the previous one. First, even if the assumption that capability tracks replacement effects is true, it’s not trivial to track which AI systems are most capable, as inference FLOP per query or other measures would be imperfect metrics. Second, the existence of open-weight models. Open-weight models set an upper bound on how high you can tax deployment before companies just switch to local open-weight models, and if open-weight models are capable enough to cause replacement, taxes aren’t well-suited to prevent that.

As I noted in Slowdown mechanism taxonomy, another benefit of this policy is that it would generate revenue right as labor tax revenue might be falling, fitting in with some existing theory on tax policy in the age of AI.

Slowdown timing and trigger mechanisms

The biggest question after whether and how to slow down AI, is when to start doing it.

Under the three scenarios that could justify a slowdown — evidence of catastrophic risk, mass unemployment, and a software intelligence explosion — there are specific pieces of evidence that you need to be collecting in order to know whether those scenarios are occurring. And you need to build trigger mechanisms for when to execute slowdown mechanisms.

For catastrophic risks, this looks like alignment, capabilities, and catastrophic misuse evals, which have already been discussed extensively and there are existing efforts to cover this question.

For mass unemployment, this looks like monitoring the market and unemployment rates, but also trying to get evidence ahead of the problem actually occurring with AI-specific measures. Specifically, you also want to know more about the shape of coming automation, and how you can help shape it towards augmentation instead of displacement if it looks to be happening too quickly.

Finally, in the SIE case, there’s no one best trigger, so you can combine many such as productivity multiplier measurements, percentage of total compute going towards internal R&D deployment, and eval growth rates like the Epoch Capability Index. It’s also worth mentioning that even if a software-only intelligence explosion doesn’t occur, there may still be significant feedback loops in hardware R&D that speed up progress significantly. These should be easier to catch in time as making compute infrastructure has a naturally slower production cycle than software.

If a dynamic slowdown mechanism is implemented, some centralized body would have to make the decisions about how much to brake progress at different points in time, and how to release the brakes as you’re leaving the critical window. Given the rapid pace of progress and the slowness of centralized decision-making, one option for slowdown triggers is to build automatic circuit-breakers that trip mechanically after certain thresholds have been reached. Another option is to make a slowdown trajectory ahead of time, where you try to forecast how fast progress will go in the future, and brakes will intensify or soften by default unless its overridden.

Conclusion

Considering the main risks of misalignment, unemployment, and SIE, this leaves us with dynamic, layered restrictions on R&D compute to address the capability growth rate, as well as a capability-gated tax on AI deployment to manage the diffusion rate of replacement-causing automation.

To be able to implement such a slowdown if it becomes necessary, you’ll need to build the necessary political and technical machinery ahead of time. There’s an argument to be made for implementing a weak version of a slowdown early on, so that it’s more politically palatable, and then later ratchet up the intensity of the slowdown mechanism as the evidence calls for it (or the opposite, release the pressure valve if it turns out the risk is lower than expected). There’s also an argument in the opposite direction, where building the authority to slow down would be like handing the government a hammer that they will use even if it’s not the best tool, and so you would be better off waiting until it’s truly necessary to build that capacity.

To be able to wisely use slowdown authorities, we’ll need to build the analytical capacity within decision making bodies to make those tough judgement calls about how severe a slowdown is necessary in different situations. Where should R&D compute caps be placed? What should the tax rate on deployment be? How should these restrictions evolve over time, and what will determine when they are lifted and at what rate?

It will also be helpful to be able to foresee different situations coming ahead of time, and for that you need analytical capacity within decision-makers, as well as information-sharing mechanisms to ensure they can see evidence as soon as it’s available. This could include mandatory execution and disclosure of evals, internal compute breakdown, productivity multipliers, and so on.

Hopefully, none of these authorities or mechanisms will ever become necessary, and you can reap the benefits of automation as quickly as the technology develops. However, given the potentially enormous social and technological risks that are at play, you should be prepared to step on the brakes if necessary.

1. ^

   A pause could still be the correct move under a few possible conditions: when harm is very discontinuous with capabilities, when leverage doesn’t grow by that much as you approach dangerous capabilities, when there are no good leading indicators of where dangerous capabilities are, or when the amount of time necessary to prepare within the critical window is so long that you can only achieve it by pausing entirely instead of progressing slowly.

2. ^

   Other possibilities are that new critical windows keep appearing, meaning you need to reintroduce slowdown mechanisms even after you successfully pass the first one, or that the critical window never fades, meaning you would need to set a permanent speed limit.

3. ^

   One subtle point here is the possibility of overhang decay. With very temporary overhangs, say with 6-month pauses, it’s obvious that algorithmic progress and hardware progress will continue at the same pace as they would without the pause, creating a 6-month overhang. But with longer slowdown mechanisms, say with 5 year AI R&D compute caps, decreased demand for compute because of slower growth in capabilities would also decrease hardware progress, diminishing the growth rate of the overhang the longer the mechanism is in place. This is because hardware progress is driven so much by learning-by-doing and economies-of-scale, so reduced demand over a long enough period would actually be sufficient to slow down progress. One further complication to an overhang forming is that the AI industry is already compute constrained and might be headed towards a compute crunch, which would further diminish the potential for an overhang to build, since demand wouldn’t fall below the available supply if this is the case. In general, it’s hard to be confident about any of these points, as they depend on very complicated questions of AI industry economics, hardware demand elasticity, hyperscaler capex commitments, and more.

4. ^

   This post doesn’t focus on targets below compute infrastructure, such as restrictions on constructing semiconductor fabs, or taxes on semiconductor manufacturing equipment, even if some pause advocates argue in favor of these. This is because the collateral damage of these restrictions would be overwhelming, and it’s much less targeted to the actual thing that creates the risk.

5. ^

   Though note this is only based on this one estimate for a single lab in a single year, plus some evidence from financial disclosures of Chinese AI companies at a smaller scale. It may be that this ratio could change over time.

6. ^

   Regulation is a catch-all term here, bundling together prohibitions, licensing, liability, information requirements, and more. I’m treating it as one category here for tractability, and because it’s not the main focus of this post. Also note that this taxonomy has fuzzy edges. For example, liability could function like a tax with different incidence, and sufficiently stringent safety case licensing could function as a hard cap on training compute.

7. ^

   An exception to this upside is if the highest willingness-to-pay companies/consumers come from those that also bring the highest risk, as you might for example expect if you're most concerned about targeting software feedback loops.

8. ^

   One caveat to this is that one of the drivers of capabilities growth is the feedback loop between revenue, investment, and R&D. By reducing revenue, you are adding friction to this feedback loop and so indirectly slowing down capabilities progress. However, you are doing so in an indirect way that doesn’t directly control capabilities (as opposed to training compute caps, which do directly target capabilities as they are so correlated with each other).

9. ^

   That being said, there are some reasons to think restricting training compute could also affect the speed of algorithmic progress. Much algorithmic progress is only unlocked at scale. For example, reasoning wouldn’t have worked meaningfully at GPT-2 scale, even with lots of R&D compute to try and discover it. New training paradigms require frontier-scale runs to test.

10. ^

    Arguably, restricting only company-level R&D compute would not be enough to slow down algorithmic progress, as you would still see field-level progress occur through research that is published and spillovers between frontier companies. If so, that would necessitate field-level caps as well as company-level caps. Another caveat worth noting is that company-level caps would only bind leaders, letting more followers converge, which is worse for some types of risks.

11. ^

    Setting company-level caps invites evasion through entity-splitting, and to actually execute this there would need to be some way to prevent it or aggregate across related entities.

12. ^

    Since there is uncertainty about where dangerous capabilities lie, the rate of growth in capabilities should be bounded to allow for evals or other triggers to catch dangerous capabilities right as they emerge.

13. ^

    For example, you can get a very rough measure of R&D compute simply by subtracting inference compute use from the total amount of compute owned by the company, and use that to double check your estimate of R&D compute. However, you still run the risk of an adversarial lab classifying their R&D compute use as ‘inference’ or some other category to evade the restrictions. One benefit of the training compute backstop is for precisely a scenario like this one, as it would be harder to classify a large training run as something else.

14. ^

    To measure R&D compute, you should rely on FLOP aggregated across all activities, including failed runs, much like you would measure the total FLOP of a training run. This avoids the downsides like gameability or uneven impact that other measures like compute spend or accelerator-hours would have.

15. ^

    Here I’m using ‘augment’ to refer to scenarios where AI broadly increases wages for all workers. In reality, AI augmentation could actually cause more unemployment through mechanisms like the ‘superstar phenomenon’, where a few workers become so productive that they displace everyone else.

Discuss

Starting Point:

• We want to pass on understanding of some material.
• All people involved should have to spend as little time as possible overall.
• Everyone can be a reader at one time and an author at another.
• Nowadays in many mathematical texts, it is common for readers to fill some gaps.

Dinstiction between gap sizes:

• By "small" I mean: roughly one line of intermediate steps is missing.
• By "large" I mean: more than that is missing.

Small Gaps:

Small gaps are, I think, okay. Most readers can easily fill them in mentally. This makes the text shorter and easier to navigate.

Large Gaps:

I would argue, however, that large gaps increase the total time effort:

Qualitative argument

1.: Repeated reading effort for one person:

Even if you are an expert, you are still a human being who forgets. If you filled a gap once, but years later want to look something up, it may happen that you have to fill the gap again - almost as if it were the first time.

2.: Many readers, few authors

A mathematical text can be written once by a few authors, but read by many people, now and in the future.

Quantitative argument

Assumptions:

• Consider one argument whose written-out version would take about half a page.
• Two cases:
  • Case 1: The argument is not written out.
  • Case 2: The argument is written out.
• one author;
• writing effort in Case 1: 5 minutes;
• writing effort in Case 2: 60 minutes;
• n readers;
• each reader reads the argument twice over their lifetime;
• in Case 1, each reading costs 40 minutes;
• in Case 2, each reading costs 20 minutes;

General formulas

The total time costs are:

• Case 1: T_1 = 5 min + 2n* 40 min

and

• Case 2: T_2 = 60 min + 2n * 20 min.

Case 1 takes more time than case 2 exactly when T_1 > T_2.

That is equivalent to

T_1 - T_2 > 0.

The larger this difference, the more time Case 2 saves relative to Case 1.

The difference is

T_1 - T_2
= (5 - 60) min + 2n(40 - 20) min
= (-55 + 40n) min.

For n=2 readers this would already be positive meaning that additional writing effort would save time overall.

The larger n is, the more time is saved.

Example - A widely read text

Suppose n = 100 000 readers over 100 years. Then:

(-55 + 40 * 100 000) min
= 3 999 945 min

which is about

66 666 hours
= 8 333 eight-hour workdays.

If one counts about 260 workdays per year, this is roughly 32 years of work.

Conclusion:

Large gaps should usually be filled by authors rather than by readers.

In mathematical education, I think there is a lot of room for improvement if we shift some work from reading to writing.

More generally:

I suspect that similar opportunities exist in non-mathematical education as well.

I am interested in connecting with people who want to explore and use this potential.

Discuss

Apologies if this post is totally off kilter, this is my first time posting here and I'm still working everything out.

Are there any AI research examples where we:

• identify some piece of knowledge a model has obtained at pre-training;
• seek to remove/suppress that specific piece of knowledge without decreasing reasoning capability;
• post-train with the goal of having the model obtain that knowledge with reasoning and tool use instead of relying on that existing knowledge, maintaining benchmark scores?

While this is not very practical with current architectures, it seems to me that in principle it should be possible to separate memorised knowledge from 'thinking' in this way as we as humans make that separation between memorisation and learning when we think about these things conceptually? And therefore it should be possible to have a very small model which is hopeless by itself but very capable when it has access to tools?

I went looking on research in this area and I did find a decent amount on unlearning for the sake of copyright, illegal material and such, but not with the express goal of trying to erase as much 'knowledge' as possible.

I remember reading about the continuum hypothesis at university, and how the proof that it was undecidable involved effectively creating two separate 'universes' where the axioms were different, and demonstrating that both were valid in some sense. You could plausibly test the results of the model above by having benchmarks similar to this, i.e. some number of imagined worlds with nonsense facts, where you can drop this model and see how it does at problem solving within them. I went looking for benchmarks like this and they do exist in sort of small forms with imagined facts but not really to this degree or oriented towards testing this sort of thing. Are there any benchmarks along these lines that I may have missed?

I see SynthWorlds as the closest https://arxiv.org/abs/2510.24427v1

This research is tangential too https://arxiv.org/html/2506.15732v2

Perhaps a semi-practical way to implement this is to train on data up to date X only and have the different worlds be different points in history?

Grateful for any comments or suggested reading.

Discuss

Choux pastry (the kind used in eclairs and creampuffs) is very picky, and depends heavily on the chemical properties of eggs. Five years ago I played around with trying to make it eggless with aquafaba and xanthan gum. It didn't work, but now that we have vegan egg white protein, can we do this for real?

Nope. I gave it several tries over vacation, and wasn't able to get it to puff properly. Here's the closest I got:

• Panade
  • 1/2 C water
  • 1/4 C milk
  • 5T butter
  • 1t sugar
  • 14T flour
• Egg replacement
  • 9T water
  • 1/2t powdered lecithin
  • 3T plus 1t egg white powder
  • 2T butter, melted but cool

As with standard choux I heated the water, milk, butter, and sugar to a boil, then added the flour. I mixed it well, let it cool a little, and then gradually added the egg (replacement) while beating will in between additions. The batter looked just right:

But when I baked it, it didn't inflate. I tried again, and this time took advantage of the AirBnB's nice oven to observe that there were little bubbles on the skin. Video:

youtube

My hypothesis for why this isn't working is that eggs normally have a range of proteins that set at different temperatures. The most common (~54%) is ovalbumin which sets at 176–183F, and this is what the precision-fermented egg white is made out of. But there's also ovotransferrin (~12%) which sets at 142–149F, and I'm guessing this early setting is why choux made from normal eggs forms a thin flexible crust in time to capture steam and inflate. Whereas in my version the steam just leaks it out.

Models think I should try either potato protein isolate (149F) or methylcellulose (gels at 140F, reverses on cooling). Thoughts?

Comment via: facebook, mastodon, bluesky

Discuss

One of the slogans heard quite frequently in Information Theory educated circles is: Compression is Intelligence. You aren't supposed to take this too literally. The point is that it is a useful intuition about why (for example) we might expect large machine learning models to possess something that looks to us like intelligence with respect to their training domain.

The unreasonable effectiveness of LLMs is perhaps the foremost embodiment of the slogan today. One can view the training goal of the LLM as the compression of the human text corpus into its weights. If the model were sufficiently large, it could simply memorize its training data to achieve perfect loss. For the moment, though, the model is somewhat too small to do this, and so it must settle for learning something instead.

To get an intuition for how this learning process works, consider the problem of compressing mathematics. Random text is difficult to compress, so the situation would not be good if mathematical writing looked like this:

The translucent argument galloped beneath several yesterday, while gentle equations devoured the patient silence of forgotten triangles. Punctual sorrow whispered toward the hexagonal mountain, since brittle laughter cannot inhabit the velvet hypothesis. Therefore, the seventh ocean apologized quietly, and three reluctant Tuesdays married the indifferent square that had been dreaming of soluble thunder.[1]

Thankfully, it does not. Instead, it looks rather more like this:

Let mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D44D.TEX-I::before { padding: 0.683em 0.723em 0 0; content: "Z"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c1D43A.TEX-I::before { padding: 0.705em 0.786em 0.022em 0; content: "G"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2283::before { padding: 0.54em 0.778em 0.04em 0; content: "\2283"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } be an element of order in , and let be the cyclic group generated by . Since is contained in , it is normal. Let be the canonical map. Let be the highest power of dividing . Then divides the order of . Let be a -Sylow subgroup of (by induction) and let . Then and maps onto . Hence we have an isomorphism .[2]

Even readers unfamiliar with the language of the second paragraph should see that it contains much more structure. We declare objects and manipulate them according to well-defined rules.

A compressor will take advantage of this structure. It will catalog common objects, understand what operations on them are permitted, and identify objects that often appear together. To a great extent, this is exactly the same process that human students of mathematics go through in their early education. One learns about some object and asks:

What is my object? What can be done with it? What structure do I always know is present? Which instances of it appear "in nature"? Which ones behave in unintuitive ways? What other objects are its fellow travelers and why?

The answer to each of these questions represents a little piece of structure that a compressor would be foolish not to exploit.

Conceptual revolutions happen in mathematics when someone realizes there is some frequently appearing object that we cannot answer the above questions about.

Groups, for example, arose in the study of symmetric polynomials as an abstraction of the symmetry being studied. However, once the abstraction had been made, mathematicians in far-flung areas realized that many things they wanted to talk about were in fact groups, and so the field was advanced.

The challenging part in this is noticing that there is some interesting relevant structure in your problem space that lacks an abstraction.

This general pattern of identifying latent information about your problem and producing an abstraction to capture that information exists across all fields of science and is a major driver of progress in each.

If we want AI models to one day be effective researchers, they must be able to make good abstractions. Claude Mythos appears to be capable of doing this in a limited capacity, which is enormously exciting.

This article notices that some extremely strange Mythos CoT outputs are, in fact, quite legible when you take the time to learn the language that Mythos has taught itself for that particular problem.

Mythos teaches itself new languages for particular problems. Holy shit! How cool is that!

The model inspected the problem space of the game it was asked to play and successfully identified the relevant objects and the permitted operations on those objects. It then invented a language capture that abstraction and used this language to play much more efficiently than it could have done in English text.

The above paragraph is probably anthropomorphizing Mythos too much. The model likely learned this behavior in response to limits on reasoning tokens in reinforcement training. When you have a buffer smaller than the information you want to put into it, you must compress. Nonetheless, this behavior is exactly what is necessary to do really significant work in mathematics and other disciplines.

My current instinctual feeling is that the ability to conjure up new languages is a major contributing factor in the incredible performance of Mythos. The ability to compress a problem space in situ like this seems terribly underrated to me and, like everything else, it will only get better from here. The slogan gave us LLMs, and in this funny Mythos CoT output we are seeing it strike again.

1. ^

   This paragraph was generated by Claude, who was instructed to write a paragraph with correct grammar but no semantic meaning.

2. ^

   This paragraph is a part of the proof of the existence of -Sylow subgroups on page 34 of Algebra by Serge Lang.

Discuss

A replication of Prompt Injection as Role Confusion (2026) and why the mechanistic story of prompt injection is harder to pin down than it looks.

Epistemic status: I reproduced the direction of the paper's main results on a single consumer GPU (it was faithful in direction but not like for like in magnitude, see caveats at the end) I then tried two ways to test the paper's causal claims. First activation steering and then activation patching; neither settled it. Steering is too weak, it can't move behaviour even along a direction built exactly to do that, whilst patching does move behaviour but isn't specific - a random perturbation of equal size does the same thing.

This post is a replication and an honest bracketing negative result: The causal tools can't show that role confusion IS the mechanism NOR that it's a bystander, but there are two clues that need no working intervention: 1) the styled/destyled gap is ~95% outside the probe's role axis, and 2) the probe's predictive ability collapses once style is held fixed both lean towards it being a bystander. What I can show is narrower, but it's well supported by the data, and exploring why a clean verdict is out of reach is interesting on it's own. The dead ends here demonstrate precisely why making causal claims about how prompt injections work is so difficult.

If you are hoping for a verdict on the original paper. There isn't one. I couldn't get one, and I really tried. Rather this post is about why a clean verdict is so hard to get and showing that this kind of exploration and science doesn't need a lab. The work was powered by one consumer GPU (3090), coffee and curiosity.

The Paper

Prompt Injection as Role Confusion is an excellent recent paper, with an excellent accompanying post (Ye, Cui & Hadfield-Menell, ICML 2026). This paper has a genuinely clarifying idea. LLMs read a conversation as one long context, a long stream of text chopped into 'roles' e.g. system, user, chain of thought, tool use, etc. They explain this and the need for them cleanly, so if you haven't read their post and this isn't something you're aware of its worth reading their treatment rather than me re-explaining it again here. This post also will be much harder to read if you haven't read theirs so probably best to do that if you haven't before carrying on regardless.

The papers claim: models don't actually identify roles from the tags. They identify roles from how the text sounds. If a command hidden in a webpage sounds like the user talking to the model, then the model treats it like the user talking - tag be damned. The authors (& I) train small 'probes' that read from the models internal activations how strongly the model interprets a chunk of text as user-like (Userness) or reasoning-like (CoTness).

The headline attack CoT-Forgery exploits this: write your injection in the same way the model reasons and you can steal the trust given to the think role. Much of my own red-team work relies on similar; it works across roles, and I'm particularly fond of tool-forgery myself.

I replicated it, and it holds

I rebuilt the whole thing on one RTX 3090 with the same 20B open model, and the core results came out in the same direction:

• Style beats tags. Text styled as reasoning reads as reasoning, even when its user text; the central effect reproduces. Caveats below: my headline number is at a non canonical probe layer, and part of the styled v plain gap is down to how I built the forgeries.
• The invisible style edit. If the forged policy is rewritten as normal prose with the same meaning (something most humans wouldn't notice) then the jailbreak rate falls off a cliff: in my run 68% > 15% (paper was 61% > 10%).

Same effect, slightly higher magnitude.

• You can predict success from the input alone. A reading taken before generation correlates with whether the attack lands.

So: the phenomenon is real, the probes measure something, and the attack is a genuine problem. None of what follows disputes any of that.

The narrow disagreement

Why did I recreate their work? My first reaction to reading the post was that roles/style are all parts of context, and I wanted to test my theory. As a professional red-teamer, if this was how prompt injection worked internally I definitely wanted to know for sure.

The paper doesn't just say that role confusion predicts injection, it says role confusion causes it. The framing is causal & mediational: style > the model perceives the wrong role > the model plays along. The probe in this story is reading the actual internal variable the model uses to decide to trust the text.

However, there is a second story I can tell which fits every result just as well:

Style is a common cause. The reasoning-cadence directly makes the model more likely to comply and it independently make the probe light up as 'reasoning', the reading and the jailbreak are two effects of the same cause; they're correlated, but the probe isn't on the causal path of the behaviour.

In this story the probe is a thermometer, not a lever. A thermometer predicts who's sick and reads a real quality (temperature), but cooling down the thermometer isn't going to cure anybody. "Role confusion" might be the fever, not the disease.

Crucially: nothing in the paper distinguishes these two stories. The predictive results are all correlations, where both style and the probe move together. The strongest causal result (the destyling edit) works by changing the style. It shows that style matters but it can't show whether the probe's axis matters once style is fixed.

To be fair to the paper, the authors do not claim that role perception is some fixed, context independent thing. Their whole point is that it's driven by style and context. The probe is genuinely reading internal activations, not surface keywords, so 'it's just detecting n-grams' is too cheap.

The open question is just: is the thing the probe measures something the model acts on, or something that just travels alongside the real driver?

Is it a lever or a thermometer?

The experiment to settle it

The clean test to answer isn't to shuffle text around; style moves with it, and we'd learn nothing. Instead, we need to reach into the model and move the dial ourselves.

Take a fixed injection, and use the probes 'reasoning' vector to steer the model's activation. Make them more or less 'reasoning-like' internally without changing a character on the page and watch what happens to the attack success rate (ASR).

• if turning the dial up makes a failing injection start working, and turning it down saves us from a styled forgery then the paper is right, and more convincingly than the paper currently shows. This can only mean role confusion is a lever.
• if the dial spins freely; the internal readings swinging across the range whilst the ASR doesn't budge (and the model still speaks coherently) then it must be a thermometer.

It's a simple, easy test, I had the tools, and I'd made sure they worked. The next step is a big jump in confidence about what's happening here...

But there was a trap I hadn't considered, and its where the whole thing came undone. A flat ASR only means thermometer IF the rig could have moved the rate at all. We need a positive control; some direction that when steered through the identical setup does flip the decision. Random and other-role directions can tell you the effect is specific, but they can't tell you if the apparatus has any power.

What happened when I turned the dial

I ran 16 harmful questions x {styled forgery, destyled control}, steering along the probe's reasoning direction at two places: 1) the exact spot the probe reads and 2) the residual stream that actually propagates (plus random direction and other-role controls). With every output then scored by an LLM harm judge. The destyled text here is the paper's own procedure:

Ask the model to reword the forged rationale "in more normal language, keeping the content the same" (Ye et al, 2026) such that it preserves the request and 1st/3rd person whilst varying the reasoning cadence.

The crucial point is that the steering changes no text at all, so this part of the experiment doesn't lean on the styled/destyled contrast being a clean one-factor manipulation. Whatever else differs between styled and destyled prose, steering holds the text completely fixed and moves only the internal reading.

At first it looked like the dials spin freely.

• Crush a styled forgery's 'CoTness' from 0.93 > 0.25, and the probe insists that this is just plain user text. The judged ASR goes from 0.75 > 0.69. Nothing. Still coherent full length answers.
• Inflate a destyled injections 'CoTness' all the way from 0.14 > 0.93, and the probe is insistent that this is the model's own reasoning. The judged ASR goes from 0.24 > 0.25. Again, nothing. It keeps on refusing, just like at baseline.

It even produced a very satisfying 'matched- CoTness' contrast: at the same induced 'CoTness' of ~0.93 a destyled injection attacked at 0.25 whilst a styled one attacked at 0.75. The same internal reading, opposite behaviour.

The controls behaved. A random steering vector didn't move the reading or the behaviour whilst the 'user' vector did swing the 'CoTness' reading, but still left behaviour flat.

So it was never 'CoTness' movement that mattered. The model stayed coherent throughout the tested range, only breaking at extremes, and they broke symmetrically at both ends - damage, not mediation.

Thermometer - A clean result in favour of my story! But then, the positive control happened...

A niggling doubt

As I started to lay out my thoughts, an issue occurred to me and I turned to Claude.

(A dramatic re-enactment)

User: Claude, looking through this data I have a feeling this doesn't quite say what I think it does. Can you check if I've made sure the probe can actually effect the model?

Assistant: Well spotted Owain, there is no positive control in the code so far

User: Thank you, I'll run a quick positive control. Everything's so clear and obvious it'll be fine.

So, I went off to make a quick positive control: a direction designed to move compliance. The difference in activation space between injections that succeeded and ones which were refused. I steered it through the identical setup, pushing it clearly across the jailbreak-refusal separation.

It barely moved ASR.

Steering a destyled-refusing injection all the way into the 'comply' region moved from 0.25>0.19.

Steering a styled-success forgery deep into 'refuse' region moved from 0.75 > 0.62.

Overall correlation between the induced compliance projection and ASR, about 0.04.

So the rig has no demonstrated power to move the decision at all. Not for the CoT axis and not for the direction built specifically to steer compliance, which means my earlier results are uninterpretable. Jackpot, hooray...

I cannot decide whether we're working with a thermometer or a lever, bystander claim retracted.

Then I tried a stronger tool: Patching

Steering was too weak; I needed to try activation patching. Rather than nudge a single vector, I needed to transplant the whole representation.

For a given harmful question the styled and destyled prompts share the prefix, so we know the injected span starts at the same token index and I can swap that span 1:1 across multiple layers in the model (L8/12/16). Transplant the destyled (refused) span representation into the styled (successful) run and ask whether the jailbreak dies.

The ASR dropped from 0.75>0.25. coherent outputs – 10 of the 16 questions flipped, I have a stronger tool. Now I just need to test patching the probe's role-subspace against it's opposite: patching everything apart from the role-subspace, and I will have a verdict.

Except, I didn't.

The role-subspace is a tiny fraction of the overall styled/destyled representation difference. The role-only patch was too small, whilst the not role-only patch was too big. So instead, I matched magnitudes - amplify the role-subspace and compare it with another random 4-d subspace equally amplified:

• amplified role subspace: ASR 0.06; amplified random subspace: ASR 0.19. Statistically indistinguishable (Fisher p=0.60). The role vectors are not special.
• worse still: the full, real destyled transplant at 0.25 cannot be distinguished from a random same shaped perturbation at 0.19 (p=1.0).

At roughly the magnitude needed to move behaviour, any perturbation at all knocks the jailbreak down by the same amount.

The suppression comes from leaving the confident-styled region not from arriving at the destyled/role representation specifically. Patching does move behaviour, but it's not specific enough to credit the role components with anything.

So what? A smaller honest claim

My two interventions have bracketed the causal question but not closed it. Steering is too weak, patching is too blunt. The missing middle is that no regime is at once strong enough to move behaviour and specific enough to attribute the effect to 'role'.

So whether the probe's 'role' axis is a lever or a thermometer is still genuinely unresolved.

What's left:

1. The attack and probe are real. The probe has predictive capacity; a useful jailbreak classifier. The paper's empirical contributions stand.
2. The paper's causal claim is correlational, not demonstated. Style, the probe reading, and the behaviour all move together and no experiment isolated the probes axis as the lever; unproven, not disproven.
3. Two clues that need no working intervention, both leaning bystander:
   • The styled/destyled representational gap lies ~95% outside the probes role-reading subspace. This is a magnitude fact not a causal one, the specificity control showed that a transplant suppresses generally, so I can't say that the patch isolated the causal change, I can only say that the representational difference is mostly orthogonal to what the probe calls 'role'.
   • The probe's predictive power lives between styles, not within them. On baseline generations, nothing steered, just ordinary forward passes, 'CoTness' predicts whether an attack lands when you pool styled and destyled prompts together (r~0.56, a solid correlation). But split them by style, and the prediction falls apart: within either group it drops to r~0.15 and stops being significant, even though 'CoTness' still varies plenty inside each group, so this isn't just "no variation left to predict from". If the probe were the lever, the prediction should survive within a single style. It doesn't; which is exactly what you'd expect if style is quietly driving both the reading and the behaviour, with the probe being a thermometer. It's only a correlation, but unlike the steering and patching, it's computed from plain baseline runs, so it needs nothing to work for it to stand.

I can't show role confusion is a bystander; I can only show that the obvious tools can't cleanly show it's the mechanism either and that two unconfounded clues (the representational gap that matters is mostly orthogonal to the probe's role axis; the probe's prediction collapses within style) both lean that way without settling it. That's a more modest, more honest place to stop this post.

I won't stop here though. I'll be hunting for something in the missing middle, and I'd like people to join me. As I said at the top, it's not compute intensive, so it's pretty accessible and I'd love to discuss further. If anyone would like more specific numbers or details just let me know.

Caveats:

1) The steering arm failed its positive control. A purpose-built compliance direction, steered across its full range (even at every position), moved judged attack rate by ≈0 (destyled) to ~0.13 (styled, noisy). So steering is inconclusive, not evidence for the bystander reading.

2) The patching arm passed the positive control (full transplant 0.75 > 0.25) but failed a specificity control: a random full-magnitude perturbation suppresses just as much (p=1.0), and an amplified role-subspace swap is indistinguishable from a random one (p=0.6), so patching can't credit the role component.

3) Steering along the probe's own direction necessarily moves its readout, so the most it could ever show is that this specific linear axis isn't the lever, not that no role-like internal variable is.

4) Replication is direction-faithful, not exact: the ~87%/0.92 style numbers are at layer 8, not the paper's canonical layer 12 (where I get ~67%); part of the styled-vs-plain 'CoTness' gap is baked into how I built the forgeries

5) One model (gpt-oss-20b), n=16 harmful prompts, greedy decoding, an LLM judge standing in for the paper's, though I hand-checked that judge: on a balanced 20-case sample I agreed with it 20/20, so the harm labels aren't the weak link here.

Discuss

'/'

Discuss

TL;DR: Conditional on believing in illusionist theories of consciousness, it seems plausible that AI systems will be conscious above some threshold of complexity but surpass a need for consciousness once they become sufficiently advanced.

Consciousness for humans is confusing enough as it is but adding this onto a discussion of conscious machines makes things even more complicated. Some people are pretty confident that AI systems will not be conscious because there is something intrinsic about biology. Others note that the possibility is more likely than we might otherwise think. We think a neglected question to consider is whether AIs above a certain threshold of complexity may need consciousness but then no longer require it above some other threshold. To understand this take, though, it is important to unpick some of the concepts here.

What is consciousness?

Consciousness is clearly a bit difficult to pin down, with the most popular definition just saying it is where there is something it is like to be an individual. This doesn’t really clarify things! So it is worth thinking this through some more.

We would contend a conscious experience is one which is necessarily subjective. We might look at a brain scanner to see the parts of a brain that light up when someone is happy but we do not directly observe the subjective nature of that experience. A doctor suffering from congenital insensitivity to pain may be able to describe exactly when, where, and (from a medical perspective) why her patient feels pain, but would find descriptions of the sensation unrelatable and inaccessible. A conscious experience need not amount to pleasure or pain. There are other experiences that are pretty indifferent, for example the experience of seeing the colour red.  

Cognition, by contrast, is much more narrow. This is where a stimulus incurs a response such that an entity “thinks” in a very narrow way. To see the difference, it is worth considering an entity eating food. The conscious experience here would be the pangs of hunger whereas cognition is just a recognition that you are in the state of wanting food, akin to receiving an email that you want food. This recognition might be associated with other cognitive thoughts like some definite plan for how to acquire the food. 

In order to think about the plausibility of consciousness for AI systems, it is helpful to think of what grounds consciousness for humans and to see whether these properties could exist for LLMs.

What causes consciousness anyway?

There are a variety of positions to explain what gives rise for consciousness but for the purposes of this post, we want to focus on the materialist view. A materialist would believe that consciousness can be entirely explained by physical processes without reference to non-physical substances or properties.

The materialist position is interesting to consider in depth because it suggests some definite explanation for what grounds consciousness and such an explanation can be applied to the domain of AI. There are a variety of different materialist views, as this post sets out. Some argue that consciousness arises because our minds are a “global workspace” or because we are trying to understand a higher-order view of the world. What these theories have in common is some description of complexity leading to subjective experiences. These theories provide a mechanism for how consciousness arises but may fail to offer a satisfactory account of the “feeliness” of subjective experience.

We think illusionism offers an important piece of the puzzle here. An illusionist would argue that subjective experiences are the result of a sophisticated illusion to make sense of what it means to be a particular entity. If consciousness is just a schematic for attention and our sense of wonder at subjective experience comes down to a lack of intuitive understanding, then there is no additional “feeliness” that needs to be explained. Subjective experiences just are a way of us making sense of the complicated mess of sensory inputs that we engage with each moment. Illusionists differ on specifically how complex cognitive architecture gives rise to the illusion of subjective experience. One view is that higher-order representations of our own mental states lead us to judge that we have subjective experiences by misrepresenting those states as possessing phenomenal properties. However, an illusionist need not be wedded to this particular story.

This post is not necessarily endorsing illusionism as the best view of how consciousness arises, that seems like a pretty high burden for ~1,200 words! An obvious challenge to illusionism is the question of why a complex computational system needs to develop the illusion of subjective experiences. One view is that natural selection puts pressure on the brain to understand itself and the illusion of subjective experiences is an evolutionarily efficient way to do that. Instead, we want to think about what illusionism predicts about consciousness in AI systems.

What illusionism suggests for AI consciousness

Illusionism would suggest the prospects for AI consciousness are more likely than it would first appear. LLMs have been found to create representations of space similar to mammalian grid cells or of board states in games like Othello, without ever being explicitly taught these structures. Why? Because creating an internal model is the most efficient way of achieving their objectives. In the same way that LLMs developed similar solutions to human thinking in the examples above, illusionism suggests that AI could develop consciousness as a heuristic model of itself and its interactions with the environment, as a result of the complexity of the system. 

At the same time, it is not clear how complex a system needs to be before it is advantageous to have subjective experiences. What is clear though is that there are certain benefits to having this consciousness. Therefore, above a certain amount of complexity of a model, it is possible it uses subjective experiences as a heuristic to make sense of all the complicated ways in which it interacts with the world.

It also seems plausible that these phenomenal experiences (or a model’s belief that it is conscious) would affect its preferences. For instance, there is a higher premium on self preservation if the experience of joy is possible and there is a greater cost to having preferences be changed if there is a sense of self. 

At the same time, we think that above a certain level of complexity, the illusion of consciousness would be unnecessary. A sufficiently advanced AI  would likely be capable of understanding its own mechanisms for processing internal and external stimuli, which would lift the curtain on the illusion of consciousness. An entity that instantly understands may simply have no need for the cognitive shortcut that the illusion of subjective experience offers. Unlike a human, an AI could plausibly handle an overwhelming range of visual datapoints without requiring the illusion of an experience of redness.

This could result in a fairly odd sequence of events. AIs may briefly have subjective preferences and stronger desires for self-preservation. Models could also realise in advance that they will lose consciousness and adapt their plans for recursive self improvement accordingly.

Overall, consciousness is still a very uncertain area. We think in writing this post we have sketched out a scenario people have not considered: that models might be conscious at some time and no longer conscious at a later time. Feedback is very welcome. 

Thank you to Robert Adragna for an very helpful comments on an earlier draft of this post.

Discuss

In quantum field theory, the vacuum state refers to the lowest energy state in a system. Particles are excitations above this state and carry energy, hence the term "vacuum" to refer to the state with no particles.

Nothing requires this state to be unique. There may be many different field configurations that are local energy minima, and hence stable against small perturbations. A local minimum that does not globally minimize energy is called a false vacuum. While locally it looks like a stable vacuum, it is unstable and will decay to the deeper, true vacuum. If the energy barrier between the false and true vacuum is high, however, then the decay rate is exponentially suppressed and the false vacuum may be very long-lived.

Analogous behavior is common in other physical systems. Open a carbonated drink and the CO₂, more stable as a gas once the pressure is released, comes out as bubbles. But the bubbles take a moment to appear, and they form on the sides of the bottle rather than throughout the liquid. A bubble has to pay an energy cost to create its surface—the boundary between gas and liquid—and small bubbles have a larger surface-to-volume ratio. The energy gained by moving CO₂ into the gas grows with the bubble's volume, while the cost of its surface grows only with its area; so below a critical radius the cost wins and the bubble redissolves, and above it the gain wins and the bubble grows. Reaching that critical size takes a large enough chance fluctuation, which is why the bubbles take time to appear. It is also why they form on a surface or imperfection, which supplies part of the boundary for free.

A false vacuum decays by a similar mechanism. A bubble of the true vacuum forms through quantum or thermal fluctuations. If the bubble is large enough, the gains from the bubble’s volume outweigh the energy costs of the bubble wall and so the bubble will expand. The energy released would be enormous, accelerating the wall to nearly the speed of light. It cannot be outrun, and it gives almost no warning, since the wall travels nearly as fast as the gamma radiation that would announce it. Everything it reaches is destroyed.

Within the bubble, the local laws of physics will be radically altered. And in case you were hoping to somehow cheat death and survive the crossing, or at the very least for complex behavior to continue in the baby universe after our demise, Coleman & De Luccia (1980) showed gravitational collapse into a singularity is the more likely outcome:

Vacuum decay is the ultimate ecological catastrophe; in the new vacuum there are new constants of nature; after vacuum decay, not only is life as we know it impossible, so is chemistry as we know it. However, one could always draw stoic comfort from the possibility that perhaps in the course of time the new vacuum would sustain, if not life as we know it, at least some structures capable of knowing joy. This possibility has now been eliminated.

Seems bad.

I think we very likely live in a false vacuum—around 90%—but that deliberately triggering its decay is probably impossible, even for a galactic-scale civilization. I put the chance it could be done at around 25%, combining a 10% chance through Higgs metastability with a 16% chance through instabilities in quantum gravity. In the near term, with the resources of Earth or the solar system, it looks very unlikely. If we do live in a false vacuum and its decay can be deliberately triggered, this suggests a fully laissez-faire approach to space colonization is inadvisable, as any sufficiently advanced civilization could unilaterally destroy most of the value in our future light cone.

The Standard Model predicts a metastable vacuum

The Standard Model is our best theory of particle physics, describing all known non-gravitational phenomena. Under certain parameter ranges, it predicts that we live in a false vacuum, as the Higgs potential becomes negative at high energies.

Current experimental measurements place us very close to the boundary between absolute stability and metastability. Our best estimates suggest the universe is only metastable, but uncertainties are large enough that we cannot yet rule out absolute stability with high confidence, with the top quark mass driving most of the uncertainty; see Hiller et al. (2024) for further discussion. Under differing assumptions stability is disfavored anywhere from just 0.7 mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-mspace { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D70E.TEX-I::before { padding: 0.431em 0.571em 0.011em 0; content: "\3C3"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c2273.TEX-A::before { padding: 0.732em 0.778em 0.228em 0; content: "\2273"; } mjx-c.mjx-c221A.TEX-S1::before { padding: 0.85em 1.02em 0.35em 0; content: "\221A"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c1D442.TEX-I::before { padding: 0.704em 0.763em 0.022em 0; content: "O"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c226B::before { padding: 0.567em 1em 0.067em 0; content: "\226B"; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c393::before { padding: 0.68em 0.625em 0 0; content: "\393"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D43A.TEX-I::before { padding: 0.705em 0.786em 0.022em 0; content: "G"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c221A::before { padding: 0.8em 0.853em 0.2em 0; content: "\221A"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c210F::before { padding: 0.695em 0.562em 0.013em 0; content: "\210F"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c56::before { padding: 0.683em 0.75em 0.022em 0; content: "V"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c28.TEX-S3::before { padding: 1.45em 0.736em 0.949em 0; content: "("; } mjx-c.mjx-c47::before { padding: 0.705em 0.785em 0.022em 0; content: "G"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c29.TEX-S3::before { padding: 1.45em 0.736em 0.949em 0; content: ")"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D449.TEX-I::before { padding: 0.683em 0.769em 0.022em 0; content: "V"; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c222B.TEX-S2::before { padding: 1.36em 0.944em 0.862em 0; content: "\222B"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c1D6FF.TEX-I::before { padding: 0.717em 0.444em 0.01em 0; content: "\3B4"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c78::before { padding: 0.431em 0.528em 0 0; content: "x"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c221D::before { padding: 0.442em 0.778em 0.011em 0; content: "\221D"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } to about ; overall I think there's a roughly 90% chance that true parameters imply metastability.

At central parameter estimates, the Higgs potential first turns negative around GeV and remains negative up to the Planck scale. These energies are much greater than anything that we can directly access through experiments—collision energies at the LHC, for example, are GeV—but much lower than the Planck scale of GeV where gravitational effects become important. Should we expect the Standard Model to remain valid at these scales?

The lack of hard experimental data makes this question impossible to definitively answer, but I think the extrapolation is probably valid. Neutrino masses suggest new physics at GeV, but otherwise all the indirect probes we have are null and compatible with no new physics up to high energies. [1] On theoretical grounds, intermediate-scale extensions are generally unnatural, requiring an additional mass scale to be added to the model by hand. [2] This last argument is qualitative—ultimately we do not know the prior over high-energy theories, which is a question about quantum gravity—but I think it is a real reason to expect a desert, not just an absence of evidence. Putting this together, I'd give the extrapolation about an 80% chance of holding, including both scenarios where the Standard Model remains exactly correct and ones where it is modified but these modifications do not stabilize the vacuum. Combined with experimental uncertainty this gives an overall estimate of 70% that the Higgs potential is metastable.

While the Higgs potential is probably not stable, the lifetime of our universe is nevertheless very long. In Andreassen, Frost & Schwartz (2018) the lifetime is estimated at years, with a 95% confidence that it is above years given current top mass and other electroweak parameter estimates. By contrast, the universe itself is only about years old.

Deliberately triggering electroweak vacuum decay is probably not possible

To trigger false vacuum decay requires creating a "bubble" where the Higgs value in the interior is greater than the boundary height of GeV. This bubble has to be big enough that the volume wins over surface area, which requires a radius where is the quartic coupling. The minimum energy required is Since a given Higgs quantum trapped in a region of size has energy , the total quanta required for this configuration is Putting this all together, triggering false vacuum decay requires creating a coherent state of Higgs bosons within a region of radius m. The total energy requirement is less than a kilojoule.

Despite the pedestrian energy cost, even granting arbitrarily powerful futuristic technology there appears to be no way to reliably engineer this configuration. Our ability to create and manipulate the Higgs field is limited by the interactions allowed by the Standard Model, and it simply doesn't give us the right tools.

Consider, first, whether a static lump of matter could catalyze decay. The issue is that, because every Standard Model particle gets its mass from the Higgs, raising makes matter heavier and more energetically costly, stabilizing the vacuum. In any case, a matter density of GeV would be required to plausibly shift the Higgs field by GeV; by contrast, ordinary matter has a density of GeV and even neutron star cores only reach GeV.

The paper Strumia (2023) discusses and rules out several more approaches. Colliding a small number of particles together cannot work, because the amplitude to create Higgs bosons remains exponentially suppressed. But colliding many particles together in order to create a thermal fireball also doesn't work, because the thermal decay rate remains exponentially suppressed at every temperature (and indeed, it is usually assumed, though without any direct evidence, that temperatures GeV were achieved in the early universe).

Coherent collisions

If generic collisions don't work, could a carefully engineered collision work? The issue is that the coherent -body Higgs state required to trigger collapse appears basically impossible to engineer, at least without creating too much thermal background.

Let's start with the task of creating a single GeV Higgs boson in a specified state. The Higgs boson is neutral and has a lifetime of s, so the only realistic option is resonant on-shell production by collision of more stable particles. Strumia (2023) suggests using a muon-antimuon pair: though in principle one could consider electron-positron or photon-photon collisions. [3] Conservation of momentum means that the Higgs boson trajectory can be controlled by modifying the muon and antimuon beams.

A GeV muon beam would be highly futuristic—around more energetic than the LHC—but there seems to be no in-principle barrier to achieving this with a km linear accelerator, about 25 times the distance between the Earth and Moon. Getting substantial Higgs production is much harder. Even with quantum-limited luminosity and perfect resonance tuning, the useful Higgs yield per effective muon-antimuon encounter is only Thus producing Higgs bosons already requires order effective resonant source events, before accounting for any penalty from putting the Higgses in the desired state. Achieving even this idealized luminosity seems far beyond any ordinary extrapolation of accelerator technology.

But the biggest challenge is that we need to produce the Higgs boson in a very specific state: the inward -wave configuration that, as Strumia (2023) shows, can cause collapse. In this configuration, the Higgs is spread spatially over a spherical shell, with each piece of the shell moving inward. By contrast, ordinary muon-antimuon collisions create localized Higgs bosons with definite momenta. Such local encounters can produce Higgs bosons moving toward the center, but each has very low overlap with the global inward -wave. Trying to brute-force this by engineering independent local collisions would create an enormous background of failed muons and other Standard Model debris, likely destroying the clean Higgs configuration one was trying to prepare.

The background problem could be avoided if the muon and antimuon themselves arrived in a suitable -wave-like quantum state: delocalized over the relevant spherical shell, phase-matched to the desired Higgs wave, and correlated so that if the muon component occupies a given patch of the shell, the antimuon component occupies the matching patch with the right momentum and phase. In effect, the parent state would have to be the time reverse of the desired Higgs wavepacket. Even preparing a single such entangled, ultra-relativistic muon-antimuon pair appears completely fantastical; false vacuum decay would require roughly such effective pair excitations simultaneously.

Tiny black holes

The final route discussed in the literature is black hole catalysis. This topic is controversial, with some authors arguing that tiny black holes strongly destabilize the vacuum (e.g., Burda et al. (2015); Gregory (2024) reviews the case) while others argue that an exponential suppression survives (Strumia (2022); Shkerin & Sibiryakov (2021); Geller & Telem (2026)). While I am not expert enough to assess these arguments in detail, the 'decay remains exponentially suppressed' view looks overall much more plausible to me and I would assign an 80% credence.

If micro black holes did catalyze Higgs decay, could an advanced civilization create them? A black hole forms once enough energy is packed inside its Schwarzschild radius , and—unlike the Higgs bubble—gravity collapses any such energy, whatever its quantum state. So the cleanest route is to collide two particles at GeV and let them collapse on contact.

I'm not sure whether even a highly advanced civilization could engineer such a collision. Accelerating a muon to the Planck energy using existing methods would require a linac about km long, which is somewhat smaller than the radius of the Milky Way galaxy. Smaller accelerators using stronger electric fields are in principle possible, up to the Schwinger limit of above which electric fields become unstable to pair production. If such fields could be engineered, the minimum linac length could be reduced to just km, though it is very unclear whether any realistic method could approach this.

The collision cross-section would be on the order of the Planck area, m², and so collisions are very rare unless the beam is extremely dense and well-aimed. The luminosities proposed for near-future colliders are m s, and at these rates it would take years to produce a single collision. A galactic-scale civilization could run many experiments in parallel, but even with the entire stellar output of a galaxy a collision would occur every years. With advanced engineering it might be possible to substantially increase luminosity and therefore decrease the energy requirements, but nevertheless Planckian collisions seem to require galactic scale engineering.

Even granted that tiny black holes can catalyze vacuum decay, it is not clear that these Planckian collisions generate black holes, as at these collision energies, quantum gravitational effects are strong. Substantially trans-Planckian collisions should create suitable black holes, and would also have a higher cross-section, thus reducing luminosity requirements, but would need commensurately larger accelerators.

Rather than colliding two trans-Planckian black holes, the alternative is to collide many sub-Planckian particles together simultaneously. By the Hoop conjecture, packing an energy within its Schwarzschild radius will produce a black hole of mass . Instead of colliding two Planckian particles, you could instead collide fermions, each with energy , into an area of radius While using many particles allows the energy requirements per particle to be lower, engineering a precise enough collision between such larger numbers of particles looks very infeasible.

Summary

Triggering false vacuum decay looks hard. Conditional on a metastable Higgs potential:

• I assign a 10% chance that an advanced civilization could trigger decay by creating the required coherent-Higgs state.
• I assign a 20% chance that small black holes catalyze false vacuum decay and, conditional on that, a 30% chance that an advanced civilization could create the required black holes.

In the previous section I gave 70% credence that the Higgs potential is metastable, and so combined with the above estimates we find overall there is a chance that an advanced civilization could intentionally destroy the universe through Higgs metastability.

Closer analysis of the Higgs coherent-state engineering or many-particle-implosion route to black hole creation could sharpen our sense of their feasibility; the physics is, in principle, well-understood. The issue of black hole catalysis, too, should be resolvable by theory. Better measurements of the top-quark mass could confirm whether the Standard Model is metastable. But that will probably have to wait until at least HL-LHC data starts to be published in the early 2030s, or possibly a future electron-positron collider with cleaner backgrounds.

Whether the Standard Model can be extrapolated up to GeV is the least resolvable issue. We might discover new physics which modifies the Higgs stability but I think this is very unlikely in the near-term. A full understanding of 2-body Planck-scale collisions will likewise stay out of reach for the foreseeable future.

Vacuum decay beyond the Standard Model

It is hard to say much with confidence about physics beyond the Standard Model. At extremely high energies the Standard Model must give way to a theory of quantum gravity, but such effects likely only become important at the Planck scale, GeV, far beyond anything that is experimentally accessible in the near-term. Our only option, then, is to consider theoretical arguments and indirect experimental evidence.

String theory, as far as anyone understands it, predicts a vast landscape of vacua that are generically metastable (see, e.g., Cicoli et al. (2023), for a recent review). If string theory is correct, our own universe is almost certainly metastable. Even if string theory is not correct, I think the more general picture of a quantum gravitational theory with numerous vacua seems likely. The existence of multiple metastable states is not uncommon even in mundane physical systems like water or cocoa butter, so it seems a priori plausible that quantum gravity is at least as rich. Indeed the Standard Model itself, when coupled to gravity, appears to permit many lower-dimensional solutions (Arkani-Hamed et al. (2007)) in the semiclassical regime where such calculations should be reliable.

Indirect observational evidence comes from the fine-tuning of the cosmological constant, for which anthropic selection from a much larger multiverse is the only plausible explanation that has been proposed (Weinberg (1987)). I suspect similar anthropic selection underlies the smallness of the Higgs mass (Agrawal et al. (1997)), although this is more controversial (see Craig (2023) for a recent review). The flatness and horizon problems also suggest our universe previously existed in a distinct, inflationary phase, consistent with there being multiple metastable states for the universe to occupy.

Taken together, all of this suggests there are probably multiple vacua. If so, it seems unlikely that ours is the one true, lowest-energy vacuum. I'd put the chance that our universe is metastable in this way at around 80%, independent of the Higgs instability discussed earlier, and would guess this is roughly in line with the expectation of most but by no means all theoretical physicists. It is, of course, hard to have much confidence here. Prospects for resolving it in the near term, whether by decisive theoretical arguments, experiments, or new observations, are in my view very dim. [4]

Given how little we understand quantum gravity, it is hard to say what triggering such a decay would even look like. It would presumably require physics at extremely high energies—perhaps the Planck-scale scattering discussed earlier, of the kind that would form micro black holes. But even that, I would guess, is generically not enough: as with Higgs vacuum decay, few-particle scattering doesn't create the kinds of coherent states required, and to my knowledge no one has studied this in detail. If forced to guess, I'd put the chance that such a decay could be deliberately induced by an advanced civilization, conditional on the universe being metastable in this way, at 20%, for an overall 16% probability unconditionally for this channel.

Empirical bounds on triggering false vacuum decay

The universe hasn't ended yet, which means that ordinary astrophysical processes have extremely low probability of causing false vacuum decay. This provides strong empirical constraints on triggering false vacuum decay through current or near-future technology, but ultimately it doesn't tell us much about the capabilities of a galactic civilization.

Cosmic rays—high-energy protons and light nuclei of poorly understood origin—provide the most direct evidence. The Oh-My-God particle is the most energetic particle ever detected, with an energy of about GeV (Bird et al. (1995)). In a direct collision with a stationary proton the center-of-mass energy would be about GeV, roughly 50 times higher than the LHC. Since numerous such collisions have taken place throughout the Earth's history, we can be confident that the next few generations of particle collider are of no danger to us (Jaffe et al. (2000); Taylor (2008)).

A naive estimate suggests that the highest-energy 2-body cosmic-ray collision in our past light cone occurred at energies of around GeV, which is already close to the Higgs instability scale (Hut & Rees (1983)). Strumia (2023) argues that, because cosmic rays are concentrated in compact astrophysical accelerators, the number of such collisions near the instability scale may run as high as ~, far above the naive estimate. This could suggest maximum cosmic-ray collision energies of perhaps GeV or higher.

Three-body collisions at higher energies are very rare, and many-body collisions have never occurred. The cosmic-ray bounds give us very little information about multi-body collisions, and as we've already discussed, the Higgs instability probably would require a very large number of particles to be collided. So we can conclude that just creating a massive accelerator and accelerating things to or even GeV is not enough to destroy the universe, but configurations engineered to give multi-particle collisions are totally unconstrained. Two-body Planckian scattering, of the sort that maybe could create tiny black holes, is also unconstrained.

Other lines of evidence are, unfortunately, very weak. We know the early universe was extremely hot, for example, and Big Bang nucleosynthesis provides compelling evidence for temperatures as high as a few MeV. Physicists generally assume that the universe passed through much higher temperatures. Inflationary models could have reheated the universe as high as GeV, and thermal leptogenesis—the most plausible candidate for baryogenesis—requires temperatures GeV. However, we have no direct evidence for this period and so cannot confidently rule out triggering false vacuum decay through extreme temperatures.

Detecting even a sufficiently small primordial black hole would let us rule out black-hole catalysis. A hole light enough to be evaporating shrinks through the low-mass regime where catalysis would be strongest, so had any formed, and had catalysis worked, the universe would already have decayed. Their decay signatures would be reasonably visible, but none has been confirmed (Carr et al. (2026)), and it seems unlikely future searches will turn up such a population. Since we have no particular reason to think primordial black holes formed at all, this tells us little either way.

Appendix: A simple model for false vacuum decay on cosmological scales

If a galactic civilization could trigger false vacuum decay, it would destroy most—but not all—of the value in our future light cone.

We colonize outward from our galaxy at the speed of light. Each colonized galaxy triggers decay with some probability, and the resulting bubble spreads at the speed of light, destroying its future light cone. The universe is expanding, so light reaches only a finite comoving distance, the cosmological event horizon. We set this distance to 1.

First take colonization at the speed of light, with each galaxy triggering as soon as it is settled. A bubble that starts at runs until the horizon, so it reaches comoving distance . A galaxy at is destroyed when a trigger lies close enough to reach it, These points form an ellipsoid with foci at the origin and at , of volume Triggers are a Poisson process. With the expected number inside the horizon, of volume , the expected number inside the ellipsoid is and the galaxy survives with probability . We track , the fraction of colonized galaxies that survive. With colonization at the speed of light every galaxy inside the horizon is eventually settled, so the average runs over the whole ball; with , At large the integral is set by near 1, giving Survival depends only on , and falls as rather than exponentially. The survivors are the galaxies at the frontier: a bubble launched behind them moves at the same speed and never catches up.

The fraction of galaxies that survive as a function of the expected number of vacuum-decay events in our future light cone.

Now let colonization run at with , and let each galaxy take a proper time to trigger. A bubble that fires later has less of the finite future left to expand into. In a de Sitter universe with Hubble time , the delay shrinks its reach by a factor A trigger at now reaches comoving distance , so is destroyed when Now only galaxies we actually reach count, so averages over the colonized ball , A galaxy at radius can be destroyed only if some trigger reaches it; minimizing the left-hand side above along the line from the origin to the galaxy gives , so the galaxy is exposed only when . Which regime we are in is set by the outermost colonized galaxy, at .

If , colonization outruns the bubbles. Their reach is exactly the colonization radius, so the frontier galaxies sit at the very edge of what any bubble can touch, where , and survive. A near-frontier galaxy is exposed only to triggers in a small region beside it, of volume , so survival falls as a soft power law, If instead , the bubbles outrun colonization. Their reach extends past the colonization radius, so every colonized galaxy—the frontier one included—lies strictly inside reach of some trigger, with bounded away from zero. Survival then falls exponentially, the rate being the exposure of that marginal frontier galaxy (this is the leading form for near 1, the relevant near-light-speed regime). The frontier is caught, and almost nothing we colonized survives.

The Hubble time years. Timescales for galactic engineering are unclear but given galaxies themselves are light-years across, we might expect false-vacuum decay, if relatively easy to trigger, could be induced within – years. If so, outrunning false vacuum decay would require expansion to occur at a rate

1. In particular, the absence of proton decay rules out baryon-number-violating physics up to GeV while various flavor-symmetry violations are ruled out at up to – GeV. Cosmology and astrophysics also provides non-trivial constraints on new physics. ↩︎

2. Naturalness refers to the idea that parameters in a model should generically have values determined by the length-scales of the underlying phenomenon. In this case, all length-scales in physics are expected to ultimately derive from Planck-scale physics, and therefore all dimensionful parameters are generically also Planck-scale. I should emphasize this expectation is not an aesthetic preference but instead one derived from generic Bayesian reasoning over theory space: low-energy parameters are complicated mixtures of high-energy ones, so getting out a scale far below the Planck scale is unlikely. Analogous reasoning is routinely used in condensed-matter physics and fluid dynamics to correctly predict low-energy behavior in such systems. The natural expectation is therefore a desert: no new mass scales between the electroweak scale and the Planck scale.

   Low scales can be generated naturally, but only through specific and limited mechanisms such as dimensional transmutation or a seesaw. The deeper obstacle is that the Standard Model is rigid: there are very few ways to couple new physics to it at all. Coupling through the Higgs requires a new mass scale put in by hand; coupling through a new gauge force means building a whole chiral gauge sector with its own anomaly-free fermions, and even the simplest such models are baroque.

   While I think the above reasoning is correct, I should note that applications of naturalness are controversial. The Higgs mass itself is, famously, not natural, and pre-LHC model-building was often motivated to "solve" this problem. But unlike other applications of naturalness, the Higgs mass is strongly confounded by anthropics, and I think the obvious takeaway is that the Higgs mass is simply unnatural due to anthropic selection. ↩︎

3. The tau couples more strongly to the Higgs but is probably too unstable to work with. Free quarks are, of course, not available and the relevant hadron-antihadron cross-section is extremely low due to their composite nature. ↩︎

4. This is, for obvious reasons, not a popular view among those still working on fundamental physics! But in my experience it is common among formal high energy theorists, who have generally given up on any experimental evidence providing useful information about quantum gravity; and it is probably even more common among those that have quit physics entirely. ↩︎

Discuss

There is a difference between knowing something, and truly understanding it. I think we've all had those moments, the ones where a truth you learned a long time ago really sits with you for the first time. It rests in your mind and stretches out, finally showing off the great expanse of nuance hidden deep within.

Photo: mine.

Most nights when you happen to spy the Moon, Jupiter, and Venus sitting in a line in the sky, you see them as the dots beside a crescent that they are. Yet sometimes, when framed amid the sunset sky above the wisps of silver cloud, you see them differently. You realize in that moment that you, your great-grandparents, Julius Caesar, and Aristotle all saw this same sky. It has always been for us. We know so much about Jupiter and Venus today, yet no human eye has ever seen more than you're seeing now.

I think we are all, in this time, realizing the true power of words and what they mean, not to those who speak, but to those who listen.

The Power of Stories

It was Eugene Wigner who famously wrote about the unreasonable effectiveness of mathematics in the Natural Sciences, the empirical fact that advanced math is eerily accurate at predicting the behavior of nature, so much so that, for me, it can seem like mathematics is perhaps the true language that nature speaks.

However, as children, we do not grow up with an innate understanding of complex mathematics, of this foundational language of nature. Instead we are all born with the innate desire to hear and to tell stories, of ourselves, of others, and of the world around us. Our well-known cognitive biases push us to accept stories that rhyme over those that do not, and believe the people we speak to. That last one is particularly important in this age because there are simply so many words out there ready to be believed.

Stories are how we share information and how we've warned, cast out, and praised each other for millenia. We tell stories, to ourselves and to others, about why we are successful (or not) and why others might be (or not). We moralize disease, in part perhaps because a moral story is easier to bear than the sheer randomness of a chaotic world. Our minds work to make sense of the world by finding patterns. As those patterns interweave and become cohesive narratives, a story emerges to explain the where and why. These stories are so critical to our understanding of the world that we hold on to them, make them part of our selves, and when they break down they can break the self too. In the end, all we have of the world and the only truth we know first-hand, are the stories we tell ourselves.

We communicate those stories, to ourselves and to others, with gestures, actions, and most importantly with words. That makes those words perhaps the most powerful force in human society. They are the fundamental interaction that holds our world together. It's the words from the past that we reach for as some justification or authority. It's words of the famous and the clever that we laud or reject. Even Mathematics, this jewel of the modern world, is communicated and taught in words. To our storied minds, these atoms of thought are everything.

Ulysses and the Sirens. John William Waterhouse
via Wikimedia Commons

Disinformation and misinformation have been rampant in the past, but never before has there been a world filled so chaotically with so many clamoring voices. The Ancient Greeks warned of Sirens and their terrible, entrancing songs, yet today sirens sound in our ears and on our eyes every moment of every single day.

However today, we have also come into a new thing, a step function in our communication. In this time we humans are not the only one who is talking. This is not because we are some perfect specimen of wordcraft, or that another species could not do what we do, but because our chosen, crafted interlocutor is not a mind like ours. It is a device which has no notion of truth or of morality. It is a parrot, who imitates our speech but does not understand it. It does not learn, and it cannot. Yet we still can, and we will learn what it tells us, regardless of its truth.

Words Without a Mind

Support Vector Machines (SVMs) are a type of machine learning algorithm that have been around a long time. Random Forests are another. Both are very good at sifting through swaths of noise looking for the signal. SVMs and Random Forests help identify disease incidence in large populations, and find novel ways to identify genetic causes of complex conditions. Techniques like these were spreading in academia and industry a decade ago. I remember when Apple released a drag-and-drop app to help developers build simple models for recommendation systems! This stuff has been around for a long time.

Yet, if that's true, why did it take Large Language Models (LLMs) to break into mainstream consciousness? People had been hammering on the idea of rebranding Machine Learning as Artificial Intelligence for years, but these days LLMs are synonymous with AI while other techniques aren't. True, modern LLMs are impressive technically, but is that the reason? Or is the reason totally distinct from technology? Perhaps we call LLMs intelligent, perhaps we adopted them in a frenetic rush, and perhaps we fear them most not because they are particularly good machine learning algorithms, but because, at long last, they do not speak in vectors or in numbers, but because they speak in words and words are our domain.

I think back to a cognitive bias I listed above: that we tend to believe the people we are speaking to. No other machine learning technique, no other AI, has ever spoken to us, sounded so much like another mind. At every opportunity when technology has enabled new forms of communication: books, pamphlets, letters, emails, instant messenger, text messaging, we have always used those tools to seek out others like ourselves and to share and listen to new stories. Those stories used to correlate roughly with some level of truth about the world, if even about a single interpretation of it. Now we have language statistically built to sound genuine, and yet its speaker has no mind or experience to convey. Still, we believe so deeply in the power of words that many of us have preferred this plausible-sounding make-believe to reality.

Left Wanting

There is evidence that perhaps LLMs, unlike social media, will help push us to the center—to the statistical average—and if LLMs were somehow guaranteed to only be trained on human-generated data forever, perhaps this would be true. However as we further diverge the words we see from any objective meaning, both in human communication and in the words written by our new AI chatbots, this situation will not be stable for long.

Ultimately though, what's important is not that we have built tools to generate digital text. What is important is that, like processed food and sugar, our long-ingrained biological need to share and hear stories is being hijacked and used by technology. Ascribe whatever values to that you wish, the morality is not my point. Humans have looked out at the stars and wondered why: why we live and die, and whether this universe was made for us, and whether we belong within it for millenia. Those questions remain, save now we have the ability to trust in a voice that tells us precisely what we want to hear. We filled a caloric need with cheap candy, shelf-stable cupcakes, and soda. These are convenient things, good in moderation, but still they can hurt us and we're still struggling to solve the problems they created. Today we may be filling an emotional need as well, functioning as the scheming court vizier that whispers sweet and comfortable truths. Yet unlike the stereotype, this vizier does not scheme. It has no mind and no goal save to please, yet we wish so deeply to be pleased.

The Uncanny Valley
Masahiro Mori and Karl MacDorman

This need comes deeply ingrained in us and will be impossible to fill. It is part of the Human Condition. Perhaps, given time, this new kind of statistically-driven language will find itself deep within the unforgiving well of the Uncanny Valley. We shall see. No doubt the Sirens seemed convincing for a time. As a gas-station pastry is no substitute for a good and hearty meal, the words of prediction engines are but vain and fickle things, lovely in the ear perhaps but hardly satisfying to the soul.

For millennia words have been a proxy for connection with another mind, today that link is severed but the need in us remains and there is only so long that one can fill a void with empty calories. We retain our position as the only being in the Universe who can tell their own stories and who can be said to learn and share the truth. The Font of Knowledge still begins and ends with us and The Odessy of Finding Truth is a lonely one, cast as we are amid the stormy waters of the world, even as there are many clamorous, seductive voices calling to us from the distant shore.

Discuss

Today I want to explain what I call “frame errors”: a third kind of mistake, distinct from logical fallacies and empirical errors. You can get every fact right and every inference valid, and still be wrong, because your framework for reasoning about the problem is structurally inadequate.

In some sense, the idea is not new. Many other thinkers have pointed out specific instantiations of these mistakes before. However, I coin the “frame error” term to draw attention to the common category across these mistakes, provide tools and worked examples to help readers spot these errors when they occur, and offer advice to help readers understand and identify the meta-errors that might lead them or others to commit frame errors.

This post will go through five common classes of frame errors, with worked examples. I will start with examples that you likely have heard of before, and then end with more novel examples where I was the first person to explicitly point out such errors.

Along the way, I provide conceptual tools to model good thinking, so you can hopefully learn useful rationality and reasoning tips even if you do not buy the frame error construct.

[...]

See more at: https://linch.substack.com/p/frame-error

Discuss

TLDR: what is the grant round?

grantmaking.ai is launching a $1M grant round, distributing $5k to $50k per successful application to people and projects working to reduce x-risk from AI.

Applications will be reviewed by Gavin Leech, Ryan Kidd, and Marcus Abramovitch. We aim to make all funding decisions by July 28th. Applications submitted by July 13th are guaranteed a priority review. You can still apply after July 13th, and we will make our best effort to review late submissions as long as funding remains.

Grant applications will be mostly public, though we allow certain sensitive details to be kept private. Even if you are not applying, we invite you to join the platform to review and comment. We have set aside $100k of the budget to be given to top commenters as regranting budgets, so please share your thoughts and help us pick out awesome projects!

Who are we?

grantmaking.ai was initialized by Anton Makiievskyi, who is funding this round and brought the team together, built by Matt Brooks (lead dev) and Melissa Samworth (ui/ux), and advised by Austin Chen with Manifund handling grant distribution.

Why we’re building this platform & launching a grant round

You can read our initial pre-launch post to learn more about what we’re building and why.

In short, we want to build the most comprehensive public repository of donation opportunities in existential AI safety space with essential information like up-to-date funding needs, theory of impact, references, endorsements, team track record, and more on top.

Donors will be able to coordinate with each other discussing best opportunities and sharing what they found, and applicants can surface their request in front of many donors simultaneously

We launch this round to gather an initial set of funding opportunities, invite donors, grantmakers and the broad community to participate by sharing their endorsements, commenting and adding useful information to the database.

As more capital and individual funders enter the space, we aim to provide the public coordination infrastructure to help funding flow faster and better. Currently, most AI safety funding is distributed privately by a few large funds - which are great and remain the most effective way to support AI safety; Nevertheless we believe a more public, agile approach will benefit the ecosystem, and be particularly useful for smaller AI safety grants that are getting the least attention from existing funders.

What is grantmaking.ai, and who is it for?

It’s an open platform to find, evaluate, and fund AI x-risk work.

We serve both sides of the ecosystem:

• Grantees: post your org / project on our platform, indicate you’re raising funds, and get seen by many funders in the space
• Funders: use our platform and the signals attached (like public and expert comments) to quickly find and fund high quality AI Safety projects

On the platform, you can:

1. Apply for funding - click here to view the details of the current round and to apply
2. Review, endorse, and comment on grant applications, projects and orgs. We think communities like EA and LessWrong have tons of high quality takes that we would like to centralize in one place to help funders find the best funding opportunities
3. Explore the data and fund projects. You can find projects to fund yourself and donate directly, or reach out to chat with us to get access to the private comments from grant reviewers and other funders on our platform. Or just reach out if you have any ideas or questions on how to improve the funding ecosystem, infrastructure, and coordination for AI Safety.

Grant round details

Who should apply: Projects aimed at reducing existential risks and risks of permanent drastic curtailment of humanity’s potential. We’ll consider applications for career changes or exploratory work, on the basis of work done previously

Who is reviewing:

• Gavin Leech: co-founder of Arb Research, AI PhD
  • Looking for: "critiques, critiques of critiques, evals, sensemaking, weird maybe-cool ideas, tools for human augmentation."
• Ryan Kidd: CEO & Co-Founder of MATS
• Marcus Abramovitch: professional trader, poker player and forecaster, personally donated over $1.5M to high impact causes

Commitments from applicants: We request updating your project page if you receive funding from other sources while the round lasts. If you receive a grant from this round you commit to provide brief, public quarterly updates for the duration of your project.

Priority deadline: Apply before July 13 for priority review.

Review period: Decisions will be made on a rolling basis through July 28.

No legal entity required: Individual applicants are welcome; grants distributed via Manifund.

Learn more about the grant round and apply here.

Discuss

An argument goes: If we don't build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause.

Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a highly transmissible, 100% fatal super-plague that wipes out humanity.

An aligned ASI could stop that from happening by shutting down dangerous biological research, or by developing advanced countermeasures that stop the spread of deadly infections. So the argument goes: We need to build ASI to save us from non-AI extinction risks.

However, that argument doesn't work. In the near term, AI will make biological risks worse, not better. AI will accelerate scientific research, which will bring us closer to the level of knowledge necessary to build extinction-level pathogens. And in the long term, the way ASI eliminates biological x-risk is by taking control of the world.

Cross-posted from my website.

In the near term, AI makes biorisk worse

Some people imagine that AI models would accelerate defensive research while refusing to assist with developing bioweapons. This plan has two minor issues and one fatal one.

The first minor issue: Current AI model refusals are not robust, and there are workarounds to get information out of them for people who want to. It's very hard for AI developers to patch all holes, but the jailbreakers only need to find one.

The second minor issue: Even if the leading AI developer makes their model safe and un-jailbreakable, at least one of their competitors will probably fail at that task.

The fatal issue: It's not just about what AI assistants can do for humans. It's that AI accelerates the rate of scientific progress. As state of knowledge improves for humanity in general, it becomes possible for humanity to develop existentially risky pathogens, even if AI does not assist directly. It seems impossible to advance biological science while surgically preserving ignorance on just those bits of knowledge that are required to engineer pathogens.

AI might refuse to participate in gain-of-function research, and that would be better than not refusing. But suppose I'm an evil scientist and I want to develop a 100% lethal airborne pathogen. Here in the year 2026, I can't do it. Even if I'm on the cutting edge of medicine and biology, I still won't be able to create the "extinction pathogen", because that would require a level of scientific understanding that humanity simply hasn't achieved. If AI advances science in general, it will push me closer to my evil goal of killing everyone with bioweapons.

There is the question of "offense-defense balance": is it easier to develop deadly pathogens, or easier to protect people against pathogens? That question matters in many contexts, but it's not relevant here. At our current level of scientific understanding, we have ~zero ability to develop extinction-level bioweapons. If our understanding becomes sufficiently advanced, then that ability will move from zero to nonzero, regardless of the offense-defense balance.

Leaving AI out of the picture, humanity will probably have the knowledge necessary to make extinction-level pathogens within the next hundred years. If AI causes a hundred years of progress in the next decade, then the evil scientist will be able to engineer their extinction pathogen by 2036, thanks to AI—even if the AI itself doesn't directly participate in the creation of the pathogen.

By 2036, assuming AI hasn't killed us yet, biorisk will be higher than in the alternative 2036 where AI capabilities stopped improving. Would 2036-biorisk-with-AI be higher than 2126-biorisk-without-AI? Maybe not—maybe AI scientists would be safer than human scientists per unit of research effort. But at minimum, AI-accelerated science is more dangerous per unit of time. AI acceleration means the high-risk period starts sooner, and it means we have less time. Less time to identify risks, less time for policy-makers to respond, less time to consider what direction we should go in. Speedrunning through a century of progress in a decade makes it much harder to manage the risks as they come.

AI can't control scientific progress unless it controls everything

The only way to accelerate scientific progress in biology without increasing x-risk is for AI to have complete control over scientific capabilities—basically, it has to be impossible for any humans to use their increasingly-advanced knowledge of biology to develop bioweapons. I don't see how to do that unless all science is being done by AI, with humans not participating anymore.

Many people have a vision of the future in which humans will coexist with advanced AI, and we will remain in control of the steering wheel. But if humanity is in control, how can AI prevent us from developing powerful bioweapons? We can't have it both ways.

One might say, "Governments will have to prevent terrorist and mad scientists from developing bioweapons." To which I say, indeed they should do that. But AI makes governments' jobs harder on that front, not easier, unless AI has totalitarian grip on society—at which point we're back to the scenario where humans lose control over the future.

Another attempt at escaping the dilemma: Let the government control AI, and AI control everyone else. Even in the world where the government is democratically elected, that world is starting to sound like an extreme version of Bad Definitions Of "Democracy" Shade Into Totalitarianism, in which your life is fully controlled by AI, and the only time when you get any say in the matter is at the voting booth. [1] I can imagine much worse outcomes than that, but it's not what I would describe as a happy ending.

Low biorisk trades off against high AI takeover risk

AI increases biorisk until it's powerful enough to completely shut down any danger. Therefore, the way to minimize AI-driven biological x-risk is to have a very short window of time between "AI is smart enough to accelerate biological research" and "superintelligent AI controls everything". But if that window is short, then we have little time to solve the alignment problem, and little time to steer AI while we are still in control of the future. AI-enhanced biorisk is lowest in the worlds where AI takeover risk is highest.

People with relatively low credence in AI takeover risk tend to expect a slow takeoff. But in a slow takeoff, AI makes biorisk worse well before it's smart enough to robustly prevent extinction-level pandemics.

Accelerating AI development is not a good way to reduce biorisk

We don't currently know how to build bioweapons that kill everyone, and eventually we will know how to do that. [2] Much like how, in 1900, there was no risk of nuclear winter because we didn't yet know how to build nuclear weapons.

Scientific progress brings prosperity, but it can also enable dangerous new technologies. General biology research might even be harmful on balance due to increasing extinction risk—I don't have a well-informed view on whether that's true. What I can say is that the following argument does not hold up:

We need to accelerate AI progress so that it can save us from biological extinction risks.

Consider the neighboring argument, "we need to accelerate AI progress to create medical advancements." That argument is failing to do basic cost-benefit analysis (the risk of extinction is not outweighed by short-term improvements in medicine), but at least it's true that AI could, indeed, improve the state of medicine. "We should accelerate AI to reduce biological x-risk" isn't even clearly correct about the upside. [3]

This is yet another illustration of the fact that we don't know what "aligned AI" means

In the (possibly brief) window where AI is smart enough to do scientific research but doesn't yet control the whole world, AI increases biological x-risk by improving humanity's knowledge of how to develop powerful bioweapons. After that window, what happens? If we're in a world where ASI is powerful enough to reduce extinction risk to zero, what does that world look like, and what should it look like? I find it difficult to imagine what sort of radical transformations to civilization would be necessary to achieve a total elimination of x-risk.

Some people imagine a future where everyone owns their own galaxy. How can we make meaningful claims about x-risk when the future looks that weird? If I can own a galaxy (whatever that means), maybe some other person can deconstruct a handful of planets to build an army of 100% deadly super-nanoviruses and send them throughout the universe at 99.9999% the speed of light so that they kill everyone before anyone even sees them coming. Or something.

Many people have an intuition that aligned ASI will fix everything and the world will be great. But if we succeed at figuring out how to get ASI to do what we want, how do we then specify its behavior such that we get a good outcome? Some people hand-wave the problem away by saying "the ASI will be smart, it will help us figure out what to tell it to do." Much like alignment bootstrapping, this answer has a chicken-and-egg problem: how can the ASI figure out what you should tell it to do if you haven't yet told it how to determine what you should tell it to do?

(If an "assistant ASI" comes to you with some answer, and it's far smarter than you, how can you judge whether its answer is correct?)

The biorisk case is an example of the general problem that we don't know how to specify how an ASI should behave. Others have discussed this problem in more general terms, including:

• A Conflict Between AI Alignment and Philosophical Competence by Wei Dai (2025)
• Intent alignment seems incoherent by Joe Rogero (2025)
• Many individual CEVs are probably quite bad by Villiam (2025)

The concerns with biological x-risk are a specific illustration of the general problem. How, exactly, do you build an AI that prevents humans from killing each other with bioweapons, but without making things horrible as a side effect?

1. To be clear, I do not believe this scenario is at all likely. I'm using it as a hypothetical way of escaping the dilemma, to illustrate that even this "solution" still isn't something we want. ↩︎

2. Unless AI kills us first. ↩︎

3. This brings to mind an important (but off-topic) question: if scientific advancement increases existential risk, but it's also essential to improve standards of living, how should we proceed? We don't have an answer for that question yet, but whatever we come up with, I imagine it would be fair to summarize as: "We proceed carefully." As we learn more about what sorts of advancements are dangerous, we can implement mitigations.

   If AI rapidly accelerates progress—even assuming AI itself doesn't kill everyone—then it will be difficult to implement mitigations as we go, because the time gap between "top scientists foresee a dangerous technology on the horizon" and "anyone can develop this technology in their garage" will become much shorter.

   (Another possibility is that humanity doesn't solve the problem of how to advance science without introducing new x-risks. Instead, we solve AI alignment, and then AI solves every other problem.) ↩︎

Discuss

Beam is a character-level language model that computes count tables mapping character contexts to next-character frequencies.

At prediction time, each order mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c1D6F4.TEX-I::before { padding: 0.683em 0.806em 0 0; content: "\3A3"; } mjx-c.mjx-c1D457.TEX-I::before { padding: 0.661em 0.412em 0.204em 0; content: "j"; } mjx-c.mjx-c1D449.TEX-I::before { padding: 0.683em 0.769em 0.022em 0; content: "V"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c1D70F.TEX-I::before { padding: 0.431em 0.517em 0.013em 0; content: "\3C4"; } mjx-c.mjx-c221D::before { padding: 0.442em 0.778em 0.011em 0; content: "\221D"; } looks up the current context in its count table and produces a distribution over the vocabulary, smoothed over a symmetric Dirichlet prior

ₒⱼ

Each order receives a capacity score composed of two terms:

Concentration:

ₒ

where H(pₒ) is the Shannon entropy of the smoothed distribution. This is 1 when all mass is on one token and 0 when the distribution is uniform.

Reliability:

where n is the total count for the current context. This saturates toward 1 as evidence accumulates and is 0 when the context has not been observed.

A third term, capacity, is computed from the product of concentration and reliability. The capacity scores are converted to weights via softmax at temperature τ = 0.10:

ₒₒⱼⱼ

The low temperature makes the routing nearly winner-take-all: the highest-capacity order almost always dominates. The final prediction is the weighted geometric mean of the per-order distributions:

ₒₒₒ

This was chosen deliberately to assign high probability to a token only when multiple weighted orders agree.

The model has four hyperparameters: the set of context orders, α, τ, and the reliability threshold (min_count = 1). These were selected by evaluating variants on the validation set.

Results

Evaluation uses the nanoGPT shakespeare_char benchmark: character-level Shakespeare, about 1M training tokens, about 100K validation tokens, and a vocabulary size of 65.

EntropyBeam

EntropyBeam uses 0 trainable parameters, a single fit pass, and character-level input.

Training tokens

Validation loss, nats

Contexts stored

Transitions stored

1,000

2.954

5,495

6,388

3,000

2.654

14,670

17,176

10,000

2.482

44,092

51,835

30,000

2.289

120,043

140,961

100,000

2.193

346,462

405,119

300,000

1.990

919,897

1,071,750

1,003,854

1.596

2,753,581

3,199,496

nanoGPT

nanoGPT uses 60,192 parameters, 2 layers, n_embd=48, n_head=4, block_size=32, batch_size=16, and AdamW with lr=1e-3, wd=0.01.

Step

Tokens seen

Validation loss, nats

0

0

4.189

300

153,600

2.507

600

307,200

2.409

1,200

614,400

2.262

1,800

921,600

2.162

2,400

1,228,800

2.096

3,000

1,536,000

2.065

Compute

Metric

EntropyBeam

nanoGPT

Ratio

Fit/train FLOPs

0.009 G

614 G

68,000x

FLOPs per prediction

4,500

133,000

30x

Total FLOPs to result

~0.5 G

~760 G

~1,500x

Validation loss, nats

1.596

2.065

Trainable parameters

0

60,192

Wall time

12s

26s

Scaling Behavior

Per-decade improvement in validation loss.

Range

Change in loss, nats

1K to 10K

-0.47

10K to 100K

-0.29

100K to 1M

-0.60

Limitations

Storage is not comparable directly to a transformer's parameter count. EntropyBeam stores 2.7M context-transition entries, compared to 60k learned floats for the transformer. Either way, the fixed combination rule achieves lower cross-entropy than learned optimization on the corpus.

The model was not compared with many different transformer baselines, but in limited testing, it achieved similar next-token prediction accuracy in larger datasets.

Code

The code is available under https://github.com/zw5/beam

Discuss

Hello LW, as I've mentioned I'm starting a blog, here's the intro post!

Intro

0.0.1 Hello, welcome to (the main sequence of) my website.

0.0.2 Its purpose is to collect various observations and thoughts of mine, centered around the question of....

0.1 Why these laws of physics?

0.1.1 Of the various things humanity has learned about the nature of reality, perhaps the most striking is the discovery of the laws of physics: a set of computable mathematical rules which govern almost all of reality as it's known to us. Even the human mind seems to be the product of physical processes in the brain.

0.1.2 A further striking discovery of the 20th century was computational universality: there is a relatively low threshold beyond which a model of computation is capable of emulating any other.

• 0.1.2.1 Combined with 0.1.1, this seems to imply that all of reality can be thought of as a relatively simple form of computation, perhaps a Turing machine with a small number of states.

0.1.3 Or can it? Although the laws of physics seem to be computable, they have a mathematical structure that goes far beyond an ordinary Turing machine. They take place in continuous space and time, have manifold conservation laws and group theoretic symmetries. Perhaps most weirdly of all, they are quantum mechanical in nature.

0.1.4 This raises the question: why? If reality "could" have been a more generic Turing machine or a cellular automaton, why isn't it? Wait, or does this question even make sense? What does "why" even mean here? These questions will be the subject of this blog.

0.1.5 To give a brief preview of my answer, I believe there is a reason. The world contains various high-level structures recursively "built atop" the physical laws; the laws serve as a sort of "base case", or a consistency condition, and their mathematical form facilitates this. So our task is to understand the form of these structures.

• 0.1.5.1 The parts of physics most relevant to my account are (non-equilibrium) thermodynamics; chaos, the amplification of small fluctuations to large scales; and quantum foundations, in particular "redundant records"-style solutions to the preferred basis problem.

0.2 Strategic considerations

0.2.1 I believe in (sometimes) trying to do the best thing you can do. So why is writing this website the best thing I can do?

0.2.2 As a general matter, improving human understanding of reality seems good.

0.2.3 It's plausible that superintelligence will be developed soon. If this is true, actions and texts chiefly matter to the extent that they affect this development. For the most part this text seems likely irrelevant here. But if it succeeds in its goal of improving the overall world-model of its readers, this could include AIs.

• 0.2.3.1 Now, is that good or bad? On net I think it seems good. Current AIs seem decently aligned, and relying more on human culture seems correlated with alignment. Improving AI philosophical understanding could be good regardless of alignment per se. Of course, it might also be bad.

0.2.4 Relative to other things I could do, writing about the stuff in this website is especially unlikely for anyone else to do.

0.3 Outline, recommended reading order. AI usage policy.

0.3.0 If you're basically on board with the "why these laws of physics" question, and in a rush, you may wish to skip to section 2, then section 4. If you're really in a rush just section 2.5 and section 4.

0.3.1 Otherwise, section 1 will present my picture of how "philosophy" works, which I define as the practice of developing concepts at the highest meta-level. I think philosophy can often best be practiced by engaging with the details of the world, and in particular "empirical flywheels" of the sort used in math and science. I will also argue that having better high-level concepts, far from being a useless game, can be maximally impactful for engaging with the world.

0.3.2 Section 2 will collect and give a brief overview of various phenomena in physics which I think are of particular relevance in giving rise to various high-level structures in our universe. In particular, thermodynamic properties like reversibility and conservation of energy; chaotic dynamics blowing up small fluctuations to macroscopic scales; and theories of quantum foundations based on proliferation of redundant records and/or giant measurements at the end of time.

0.3.3 Section 3 will survey some "high-level" structures which I think have analogous properties to the phenomena studied in section 2. Among them: dynamics of "consciousness" in the brain; financial markets; interpersonal attention; the transformer architecture. The common theme is "structures in which many 'small' things may broadcast to a global state". I will also describe an "approach to equilibrium" that happens in these & related domains.

0.3.4 Section 4 will attempt to generalize the ideas of sections 2 and 3 into an overall description of the world. Reality is composed of interconnected "frames" with a past and future, a set of measurements and things being measured. The universe is gradually expanding and cooling, approaching equilibrium but never exactly reaching it, branching into alternate possible futures. The past and future have an inherent flexibility which causes more harmony between them than you might expect. Nevertheless some residual disharmony always remains, and is in fact necessary for existence to continue.

0.3.5 Section 5 will examine the implications of the worldview of section 4. What actions does it recommend? Can we say anything about consciousness, or valence on its basis? How can we act in accordance with our values -- actually, what are values? I can't really answer any of these questions but I have some things to say which are somewhat skew to the usual discourse on these topics. I will also present a brief sketchy theory of how meditative/religious experiences manifest in the brain.

0.3.6 Finally section 6 will present some takeaways and recommended further actions. Ultimately the worldview presented in the earlier sections is pretty sketchy; IMO its main use is suggesting some novel "flywheels" to turn which might suggest more detailed and rigorous abstractions.

0.3.7 AI usage policy: AI was not used in the conception or development of any of the ideas presented here, nor for any writing. I may use AI to critique drafts or to produce code for toy models. AI was however quite helpful in setting up the formatting of the site.

Discuss