Сборщик RSS-лент

tl;dr:

About 80% of released J-lenses target the final layer. On DeepSeek-V3, though, that gives a J-lens dominated by one direction inherited from the final block. It shifts English-vs-Chinese readouts and also inflates one eval.

Anthropic's J-lens paper had suggested the final block may specialize in calibrating the next-token prediction. That could make it the block most likely to carry a direction like this, meaning the penultimate layer may be a better default. More generally, this is a case study of how a strong downstream direction can dominate a J-lens and change what earlier layers appear to represent.

Overview

A J-lens lets you peek inside a model by translating its hidden states into words (a "readout"). It is defined relative to a target layer. Specifically, it asks how a nudge at an earlier layer would change the representation at that target, averaged over many prompts, then reads the result through the model's own unembedding. On DeepSeek-V3, changing the target layer changes what the lens shows you.

With the final layer as the target, the J-lens is dominated by a single direction inherited from the last transformer block. That direction shifts the language of the readouts between Chinese and English.[1] This dominant direction arises because in DeepSeek-V3, the final block pushes down all the Chinese tokens when the text is English. This barely changes what the model predicts since those tokens already had almost no probability, but it's a large change to the residual stream. Because the direction is strongly amplified by the last block, it becomes the top Jacobian direction at every earlier layer.

Halfway through DeepSeek-V3, on English text, about half the readouts from a final-layer lens come out mostly Chinese tokens. If you switch to the penultimate-layer, the lens is only 3% majority Chinese, matching the logit lens readouts. Changing the target also flips the sign of one of the statistics in Anthropic’s original J-lens paper, excess kurtosis. Additionally, on the order-of-operations prompts, one of the paper's tests, targeting the final layer lifts every number token by about a standard deviation and mistakenly makes J-lens look better than the logit lens on the eval.

Figure 1. The final-layer target puts a dominant language-separating direction into DeepSeek-V3's readouts, and the penultimate target removes it. All panels use 2,400 held-out English activations. (a) Chinese-character tokens among each half-depth readout's top 20; 5 of 20 would match the vocabulary. (b) Median share of readout variance explained by the language split. (c) Each layer's top singular direction: its cosine with the top direction of the last layer's Jacobian J₅₉→₆₀, and how well it separates Chinese-character from Latin-alphabet unembedding rows. Shading marks Sonnet 4.5's workspace depths; the ten-prompt lenses are noisy below 30% depth.

DeepSeek-V3’s last block amplifies a language-separating direction twelve times more than a typical direction. When the J-lens differentiates through that block, this direction dominates the Jacobian even for much earlier source layers. Targeting the penultimate layer excludes that block and reduces the language effect by roughly twentyfold. This same choice explained several observations I had initially treated as unrelated: Chinese-heavy readouts, anomalous kurtosis, a number-token bias that inflates one evaluation, and a handful of fit prompts with Jacobian norms hundreds of times above typical prompts.

The important lesson is not that DeepSeek-V3's J-lens inherits a direction that shifts readouts between English and Chinese. It is that a strong downstream direction can dominate a J-lens and change what earlier layers appear to represent. To avoid an output-specialized final block, the penultimate layer may be a better default.

This matters since so far most lenses seem to just default to last layer. Of the 78 publicly released J-lenses I checked, 80% target the final layer, 17% the penultimate layer, and 3% something else. Gurnee et al. used a penultimate-layer target for their Sonnet experiments, but the released library defaults to the final layer. For anyone fitting or using a J-lens, it is worth checking what directions the lens might inherit based on the choice of target layer. Take the top direction of each layer’s Jacobian and check whether it aligns with a dominant direction introduced by the final block, and whether changing the target removes it.

Separately, I also tested how the original J-lens results transfer to DeepSeek-V3. I replicate the depth curves the paper uses to locate its workspace, but the plain logit lens reproduces them too. On the paper's evals the J-lens beats the logit lens on two sets and slightly loses on two.

Fitting a J-lens for DeepSeek-V3

A J-lens shows which words a hidden state is pushing the model toward, by sending it through an average of what the rest of the network does to that state. Briefly summarized:
Pick a layer ℓ. Compute the Jacobian of some late residual stream (the target layer, often the last layer or penultimate layer) with respect to the layer-ℓ residual stream, average it over token positions and a corpus of prompts, and call it J_ℓ. To read out a state h, multiply by J_ℓ, apply the model's final norm and unembedding, and look at the top tokens. The logit lens is the same thing with J_ℓ set to the identity.

This post explores the how the choice of target layer affects the lens. The reference library defaults to the final layer, with a docstring note that the penultimate layer "can give a better-conditioned" lens. Among publicly released J-lenses, the final layer is by far the most common. I checked 78 publicly released fits, including Neuronpedia (38), OLMo-3 (11), workspace-lenses (8), Prabodha (8), Andy Xu (5), LAMM-MIT (3), Praxagent (2), Qwen3.5 image lenses (2), and Stanley Li (1). Mirrors were deduplicated (so numbers may look lower than expected). Of these, 63 target the final layer (80.8%), 13 the penultimate layer (16.7%), and 2 other intermediate layers (2.6%). In the original Anthropic J-lens paper, the method is presented with the final layer as the target while the appendix says the lens used throughout the paper targets the penultimate layer because "including the last layer can sometimes increase the number of noisy artifacts in lens-readouts" possibly due to specialization for calibrating next-token predictions.[2]

I fit on DeepSeek-V3-0324 (671B MoE, 4-bit AWQ weights): 100 WikiText prompts, all 60 source layers, about 40 minutes per prompt on three H200s, so roughly 80 hours of wall time for the 112 prompts I fit, of which 100 went into the released lens (the pre-filter removed twelve and I fit replacements). The fit keeps running sums, so I also have two disjoint 50-prompt lenses, which help check convergence. They pick the same top token 13% of the time at a quarter of the depth, 58% at half depth and 90% at 83%, so I don't trust the first third of the network. Readouts are on 100 held-out paragraphs. For the cross-model comparison I pulled the same held-out text through Qwen3.5-122B-A10B and Qwen3.5-397B-A17B and ran their public lenses through the same code.

Pre-filtering prompts was necessary. Without it, a plain average over my first 100 prompts did not produce a useful lens because four prompts carried 84% of the summed Jacobian norm and the resulting lens read out a space or a hyphen as the top token for most inputs. To fix this, I applied a pre-filter that drops prompts whose norm is more than five times the median (the paper's appendix suggests a pre-filter of this kind) and added additional prompts until n=100 was reached.

DeepSeek-V3 is a mixture-of-experts model, so single-prompt Jacobians depend on what else is in the batch, because expert routing does. Between a batch of one and the fit's batch of 64, 38% of token positions get a different set of experts (8.5% of individual expert slots change). Averaged readouts don't care (about 95% top-1 agreement either way).[3]

Where the language shift comes from

My released lens targets the final layer, the library default, and everything in this section is that lens unless I say otherwise. Halfway into the network, on English text, its readouts strongly split into two languages. 46% of them are majority Chinese in their top 20 tokens, 19% are entirely Chinese, and 36% contain no Chinese character at all. The logit lens on the same activations puts a median of 3 Chinese characters in its top 20, close to the 5 that the vocabulary's 27% Chinese share would give, and only 4% of its readouts are majority Chinese (Fig 1a). On a model trained on Chinese and English that isn't immediately shocking, and for a day or two I read it as a bilingual workspace that the logit lens was partially missing.

Figure 1 panels (b) and (c) show that is not the case. It was in fact due to a dominant direction inherited by the last block.

The logit lens in panel (b) shows that the residual stream carries no language shift that the unembedding can see until the last layer, where the share jumps from 0.012 to 0.42. The model makes this shift once, at the end.[4]

The last layer's own Jacobian, J₅₉→₆₀, is close to the identity apart from a couple of directions. The largest of them is amplified about twelve times more than a typical direction, and it separates Chinese-character from Latin-alphabet tokens with an AUC of 0.997 (Fig 1c).[5]

A lens that targets the final layer differentiates through that layer for every source layer. So every J_ℓ inherits the direction, which is what panel (c) shows: it is the top direction of every layer's Jacobian (AUC 0.99–1.00 throughout, cosine 0.83–0.89 with the last layer's), and it holds 43% of the Jacobian's squared norm at layer 0. Project that one direction out of the released lens and the language share drops from 0.2–0.5 to 0.015–0.05.

A lens that targets the penultimate layer never sees the last layer. Its top direction barely overlaps the last layer's, cosine 0.02–0.22 against 0.83–0.89, and is unrelated to language for the first quarter of the network, after which it climbs to an AUC of 0.89 by 87% depth. That's the orange line in panel (c): the earlier blocks do carry a weak version of the same direction, and what the last layer supplies is the size. The orange line in panel (b) is what that means for readouts: same ten prompts, same settings, same 24 layers, language share 0.52 / 0.38 / 0.23 → 0.03 / 0.015 / 0.010 at a third, a half and three-quarters of the depth.[6]

Could there be real bilingual content underneath the shift? If there were, something should survive when the shift is removed, and almost nothing does: both ways of taking it away, projecting the direction out and moving the target, leave readouts with barely any language structure left. Additionally, the shift moves the whole Chinese block together, rather than the words for whatever is being read. And through about 60% of the depth the shift is the same whether the model is reading English or Chinese, and it keeps the same sign after that, while the model's own logits flip from 1.4 standard deviations against Chinese to 1.5 in favor of it. It isn't that the states lack the information: a single direction in them tells Chinese text from English at an AUC of 0.78–1.00 at every depth, where the lens's language offset manages 0.48–0.62 through three-quarters of the network and only climbs (0.69, 0.81, 0.94 at layers 50, 55 and 58) as the source layer approaches the target.

Figure 2. The final-layer J-lens amplifies Chinese tokens at early layers and English tokens at late layers. Its language offset largely not depend on the language of the input until late layers. Median over activations of each readout's mean Chinese-character logit minus its mean Latin-alphabet logit, in units of the readout's standard deviation: the released J-lens on English text (black) and on Chinese text (magenta), and the logit lens on the same states (dotted, same colors), which runs to the output, where it is the model's own logits.

Why does the shift point toward Chinese early and away from it late? The shift comes almost entirely from the lens's top direction. That direction is a pair of vectors: an input direction, which the lens reads from the state, and an output direction, which it writes into the readout. The output direction barely changes with depth, sitting at a cosine of about 0.5 with the Chinese-minus-Latin axis of the unembedding at every layer. What flips is the sign of the state's component along the input direction. The average state is within a few degrees of perpendicular to the input direction, so small drifts flip that sign. The share of states whose component pushes Chinese up falls from 84% at layer 18 to 54% at layer 32 and 26% at layer 45, and the push turns from Chinese-up to Chinese-down between layers 32 and 35. The state and the input direction both drift a little, and at that angle a little is enough. The model's final norm caps the result, so no readout's offset goes much past ±1.8 standard deviations, though near the crossing, readouts spread across that whole range.

Three more effects from varying the target layer

Kurtosis. The original J-lens paper, Gurnee et al., tracks four workspace statistics across depth, one of which is excess kurtosis. Kurtosis measures how much of a distribution’s spread comes from a few extreme values. If a few tokens rank far above the rest, kurtosis is high. In Sonnet, kurtosis is near zero early and begins rising around one-third depth, which is one of the signals the authors interpret as the start of the workspace. On DeepSeek-V3, the final-target lens gives a negative median kurtosis through the middle of the network, though. Kurtosis is negative from 17% to 63% of depth in the 100-prompt fit.

Figure 3 shows why. It shows one typical English readout a third of the way into the network, with its 129,280 token scores plotted by language. Under the final-layer lens they split into two groups, divided by language. This causes negative kurtosis. Refit the same ten prompts with the penultimate layer as target and the split is gone.

Figure 3. Kurtosis goes negative because the final-layer J-lens gives bimodal readouts. One typical held-out English activation at layer 20, a third of the way through DeepSeek-V3, read through a lens fit with the final layer as target (a), through a lens fit on the same ten prompts with the penultimate layer as target (b), and through the logit lens (c). Each panel shows how the readout's 129,280 token scores are spread, in standard deviations from its mean, with the vocabulary stacked by language.

With the penultimate target the median never goes negative and it rises through the second half of the network, to about the level of Sonnet's median, though with a far smaller upper tail.

Number tokens leading to a misleading eval. The original J-lens paper scores the lens on how well it surfaces hidden intermediates, taking each intermediate's best rank over all layers. On the order-of-operations set, DeepSeek-V3's J-lens beat the logit lens. But this is misleading — it is actually just because the final-target lens lifts all number tokens in the first two-thirds of the network. If you rank the right number among the integers alone, so that lifting all of them together can't help, then the final-target lens picks it out no more often than the logit lens does (49% against 53% in those layers).

Is this due to a single inherited direction, like the language shift? No. The last layer does amplify an integer-sorting direction: its second largest, amplified about 10x, separates integers from the rest of the vocabulary at an AUC of 0.963. But when you project that direction out of every readout, the wrong-integer rate only falls from 29% to 24%, and it takes the true number down with it, 62% to 44%. Projecting the last layer's top four directions out gets to 22%, still six times the logit lens's 4%, while fitting one layer down gets to 7%. So the number lift is from the final block, like the language shift, but no single direction carries it.[7]

Few prompts dominating the average. Among the first 100 prompts, four held 84% of the norm (they were excluded after I introduced a filter). I refit all four of these prompts with the penultimate target to see if their norms remained anomalously high. Their norms dropped significantly, from 5,038, 3,947, 2,984 and 1,065 to 5.5, 6.0, 21 and 13, below the pre-filter's cutoff of 40 and within a factor of 1.5 of the largest ordinary prompt. For the largest one the ratio grows with backward depth, 31× one layer below the target and 913× at layer 0. So on these prompts the blow-up is the last layer's derivative, multiplied into every layer. This doesn't show whether the penultimate-target fits have a heavy tail of their own since I only fit 10 prompts.

Comparing with Qwen

Is the same language shift seen in Qwen, another model family trained with significant amounts of both English and Chinese text? (Qwen's tokenizer is 22% Chinese and DeepSeek's is 27% Chinese.) Two Qwen3.5 models have public lenses that also target the final layer, so I ran the same analysis on them:

Last layer's language shift on English text

Language share in the lens's readouts

Top direction separates languages (AUC)

DeepSeek-V3

−1.43 sd

0.2–0.5

0.99–1.00

Qwen3.5-397B

−0.45 sd

≤ 0.07

0.77–0.83

Qwen3.5-122B

−0.27 sd

≤ 0.02

0.50–0.65

Same code, and the logit lens is near zero in all three. The 397B shows a faint version of DeepSeek-V3's shift and the 122B almost none. Language isn't the only direction a last layer could amplify, so I also compared each layer's top direction with the last layer's. Neither Qwen lens is dominated by its last layer the way DeepSeek-V3's is. Their top directions line up only partly with it (cosine 0.35–0.6 at most depths, against 0.84–0.89). This holds even though the 397B's last layer amplifies its top direction eight times more than a typical direction, two-thirds as much as DeepSeek-V3's does. So a last layer that strongly amplifies a direction doesn't by itself mean the lens inherits that direction. To find out, compare each layer's top direction with the last layer's, as in Figure 1(c).

This is about how the bulk of the logit vector is arranged, not about whether the models stay in one language. All three put under 0.05% of their probability on Chinese tokens when reading English. On English text, DeepSeek-V3's last layer also suppresses about thirty-five thousand Chinese tokens it was never going to pick. That barely changes what the model says, since those tokens already had almost no probability. But a Jacobian doesn't go through the softmax; every logit counts the same whatever its probability, so shifting thirty-five thousand of them together is one of the largest things the last layer does, and the lens inherits it.

I also ran two logit-lens controls on template positions from my filler-token paper, where DeepSeek-V3's own output shift reads −0.81 sd rather than the −1.43 sd on prose above. DeepSeek-V3 in FP8 gives the same shift as the 4-bit model, slightly larger (−0.96 sd). Kimi K2 and K2.5, which use a similar architecture to DeepSeek-V3's, have no Chinese-suppressing shift at the output (−0.04 and +0.22 sd). So the language shift may come from DeepSeek's training rather than its architecture.

How to check a lens

A lens inherits every block between the layer you read and its target. If the lens includes the final block, this may include output-specific artifacts like the dominant language direction seen with DeepSeek-V3. The first three checks below look for a direction like that. The last two apply to fitting and scoring any lens.

1. If your lens targets the final layer, refit a few prompts at the penultimate layer. This is the most computationally expensive check, but ten prompts with each target is enough to compare the readouts. For DeepSeek-V3, this check caught artifacts that were not removed by projections, such as the number lift.
2. Look at each J_ℓ's top output direction. What does it read out as, how much of the norm does it hold, and what changes when you project it out? A block's direction only shows up at layers below that block, so a direction that stays on top all the way to your target, as in Figure 1(c), comes from the last block. Projecting it out fixes an effect carried by one direction, like the language shift here, but not one spread across many, like the number lift.
3. Compare readouts with a reference. Read the same activations with your lens and with a reference (the logit lens and/or your lens refit one layer down), standardize each readout, subtract, and average the difference over many activations. Sort the tokens by that difference and read both ends. If the lens moves a group of tokens together, it will show up there. On DeepSeek-V3, one end of the list is Chinese characters and Chinese punctuation. A group as big as a language also splits a single readout's histogram, as in Figure 3.
4. Put a wrong-answer control on any best-rank-over-layers metric. Score answers of the same kind that aren't the item's, and subtract. If the right answer belongs to a class the lens might lift as a whole (numbers, say), also rank it among its own class: that separates a lift of the whole class from what the model actually prefers.
5. Keep track of per-prompt norms, and compare lenses fit on two halves of your prompts. The norms show whether a few prompts dominate the average; in DeepSeek-V3, four prompts were 84% of the lens until I filtered them out. The halves help show which layers are fit well enough to read; where they disagree, fit more prompts or don't trust single readouts there. In DeepSeek-V3 they disagreed most in the early layers, where the Jacobian passes through the most blocks. R-lens, described here[link], may be a better option for early layers.

The J-lens paper's claims with DeepSeek-V3

Unless specified otherwise, everything below uses the final-target lens with a hundred prompts since I did not run a full fit with the penultimate target. The language shift is still present, so wherever it could change a result, I ran a control or say so.

The depth curves generally match, and so does the logit lens. Next-token accuracy and autocorrelation have Sonnet's shape in DeepSeek-V3, and the ten-prompt penultimate lens gives nearly the same curves (accuracy within three points, autocorrelation 7.4 against 7.5 at the peak). Dimensionality has Sonnet's shape too, though its knee is partly the language axis — the ten-prompt penultimate lens needs 24% of the dimensions at a quarter of the depth, where the same ten prompts with the final target need 2% and the released lens 5%. So does the kurtosis median, once the language block is controlled for, though it is much smaller and has no rising tail.[8] But every one of these that has a logit-lens version looks the same under the logit lens. It's ahead on next-token accuracy at every layer but one, it has about 80% of the autocorrelation with the same shape, and its kurtosis rises at the same depth. Readout entropy falls together for both lenses too, at layers 30–35, under either target.

Figure 4. The paper's Figure 28 for DeepSeek-V3 with the released lens and with the logit lens. Next-token accuracy, autocorrelation and dimensionality have Sonnet's shapes, though the dimensionality knee is partly the language axis. Kurtosis differs in three ways. The negative median through the middle of the network is the language direction (explained above). The 99th-percentile spikes in the first third are a final-target effect: on the same ten prompts they are 22–39 with the final target and under 4 with the penultimate one. And no lens on DeepSeek-V3 reproduces Sonnet's rising upper tail.

Evaluating the final-layer J-lens. I scored five of the paper's six eval sets on DeepSeek-V3.[9] With the wrong-answer control from check 4 applied (twenty intermediates from other items, scored the same way), released lens minus logit lens at pass@10:

Set

True intermediates

Wrong answers

What the lens reads

Association

+0.12 ± 0.04

+0.00

+0.12

Typo

+0.13 ± 0.04

+0.00

+0.12

Order of operations

+0.06 ± 0.02

+0.05 ± 0.01

+0.01

Multihop

−0.04 ± 0.03

−0.01

−0.03

Multilingual

−0.04 ± 0.02

+0.00

−0.04

Only the association and typo evals show genuine uplift compared to the logit lens. The order of operations eval is not true uplift; it is from the number lift artifact as seen by the 'wrong answers' control. The wrong answers are other questions' numbers, so the control can't cleanly separate a lift of all numbers from the model's real preference for numbers near the answer; ranking the right number among the integers alone does, and shows no gain over the logit lens (see the section on number tokens above). Multihop is a small loss at pass@10 and a larger one at pass@5. Multilingual is a small but consistent loss. Neither comes from the target. With the same ten prompts, the final-target and penultimate-target lenses trail the logit lens by the same amount on both sets.

The penultimate target on the paper's evals, at ten prompts. The original J-lens paper found the penultimate target "a small improvement in extracting intermediates" on Claude. I can only test this with ten prompts on DeepSeek-V3 and do not find any improvement. The penultimate lens loses about a tenth at pass@10 on association and typo, the two sets where the J-lens beats the logit lens, and matches the final target on the other three. That could be because the last layer carrying something real or it could be that the penultimate lens is underfit. Ten prompts are enough for the final target to reproduce the released lens's eval results, but that target is mostly one large direction that a few prompts pin down, and the penultimate lens has no such anchor. To truly test the penultimate-layer lens, more prompts would need to be fit.

The same evals on Qwen3.5-122B. Qwen's logit-lens readouts are nearly uniform until the last tenth of the network, so if a J-lens can read what the unembedding can't, Qwen is theoretically where it should show. So I extracted the eval states there too and scored its public lens the same way (layers 0–46, which is what the lens holds; Qwen's tokenizer splits multi-digit numbers into digits, so 22 of the order-of-operations intermediates and 9 of the multihop ones have no single-token form and are skipped):

Set

True intermediates

Wrong answers

What the lens reads

Association

+0.02 ± 0.02

+0.00

+0.02

Typo

−0.08 ± 0.05

+0.00

−0.08

Order of operations

+0.16 ± 0.05

+0.38

−0.21

Multihop

+0.08 ± 0.05

+0.06

+0.01

Multilingual

+0.04 ± 0.02

+0.00

+0.04

The Qwen lens does not beat the logit lens on these evals. Its only true uplift is on the multilingual eval, by four points. On order of operations eval, and the number lift is stronger than in DeepSeek-V3's lens. A wrong digit reaches the top ten at some layer 42% of the time, against 4% for the logit lens. Ranked among the digits alone (the only numbers that are single tokens on Qwen), it picks the right one about as often as the logit lens (70% against 64%, ±6 points, 33 items). So both public final-target lenses I've scored inflate this eval the same way. The metric takes each intermediate's best rank over all layers, so it doesn't say where a lens reads. On four of the five sets, the Qwen logit lens gets most of its credit, 60–95%, from the last five layers. On typo it gets it from the middle of the network, where its readouts are nearly uniform on ordinary text, and typo is the set where it beats the J-lens.

Translation pairs are mostly the unembedding. Mid-network, DeepSeek-V3's Chinese and English readout tokens are very often translations of each other: 80–87% of activations have a dictionary pair in their top ten of each language. Neel Nanda's review worried that this could come from the vocabulary matrices rather than the representation, and it does. The unembedding alone picks the right Chinese translation of an English token 52% of the time, and if you replace each readout's Chinese tokens with the unembedding's nearest neighbours of its English ones, you get a higher pair rate (87–94%) than the one observed. Almost any residual that points at " felt" also points at 覺得, which is also why the Chinese tokens in a mid-network readout being translations of the English ones says nothing either way about bilingual content.[10]

Limitations

The final-layer artifacts are all shown for only one model, DeepSeek-V3. The Qwen lenses inherit their last layers only partly and show little to no language shift, and I have not checked what the last blocks of other models amplify. The recommendation in the title rests on DeepSeek-V3 plus a general concern that the final block could be output-specialized in a way that can cause artifacts. What I showed is that DeepSeek-V3's final block has a dominant direction, a language shift, that barely moves the output probabilities but dominates the final-layer J-lens.

The fit prompts and the held-out text are English Wikipedia. The lens carries the last block's English-text behaviour onto Chinese text, and I didn't try a fit on mixed-language prompts. The number lift shows the artifacts depend on the input, so nothing here says what the readouts do on code or chat text.

The penultimate lens was fit with only ten prompts. That's enough for the paired test, and it's where the paper's prompt sweep says gains become modest, but it's far fewer than the hundred in my released final-target lens or the thousand in Anthropic's Sonnet lens. The ten-prompt lenses aren't reliable in the first third of the network. That's measured on the final-target one, against the hundred-prompt lens with the same target, and the penultimate one has no reason to converge faster. I did not fit a hundred prompts for the penultimate J-lens due to computational limitations, so the original paper's claims were tested with the final-target lens.

The evals are coarse. They ask whether an intermediate is recoverable at any layer, which is what the Anthropic J-lens paper needed, and its margins on Sonnet were large enough for fifty-item sets to show it. Mine are a few points on about a hundred items at one k, which is within noise, and the any-layer rule means the metric can't say where a lens reads. No causal or steering tests, so the Anthropic paper's strongest evidence for the J-lens isn't addressed.

The Qwen comparisons use one lens per model, fit on different prompts, and the Qwen evals are on the 122B only. The DeepSeek fit used 4-bit weights. The last block's language gate is in the FP8 model too, slightly larger, so the source isn't quantization, but no Jacobian was computed at full precision, so the lens numbers are 4-bit numbers, and the released lens belongs on states from the same checkpoint. DeepSeek-V3 is also a mixture-of-experts model, and the stored Jacobians match finite differences to 0.5% at layer 59 and 2% at layer 50, but at layer 30 they differ by 13–49%, partly expert routing, which averaged readouts hide.

Release

The released 100-prompt lens and its two halves, the paired ten-prompt lenses (labelled by target and prompt count), the per-prompt norms, and all code: https://huggingface.co/kbrauer/deepseek-v3-jacobian-lens

1. ^

   I say "language" throughout for readability. What I measure exactly is the writing system a token is in, Chinese-character tokens against Latin-alphabet ones. Japanese kanji count as Chinese characters here, and Latin covers English along with every other Latin-script language. The "language split" in the figures is three-way: Chinese characters, Latin letters, and everything else. Everything else includes digits, punctuation, whitespace and the other writing systems, about 16% of the vocabulary. The share of readout variance it explains is a one-way analysis of variance over the 129,280 token scores in a readout.

2. ^

   The J-lens paper's appendix (A.7) says the Sonnet lens used throughout the paper takes the gradient at the penultimate layer, "omitting the last transformer block from the backward pass", because "including the last layer can sometimes increase the number of noisy artifacts in lens-readouts. This may be because the final block is heavily specialized for calibrating next token predictions and contains less semantic content." Their ablation compared the two targets and found the methods "fairly consistent among these design choices, though mean aggregation of penultimate is a small improvement in extracting intermediates." The same paper's Figure 4 caption and pseudocode say the final layer, and the released library's `target_layer=None` resolves to the final layer, with a docstring note that the penultimate one "can give a better-conditioned" lens. Neel Nanda’s Qwen3.6 replication targeted the penultimate layer; both public Qwen3.5 lenses target the final layer, as do 38 of the 39 J-lenses on Neuronpedia as of September 2026 (the one penultimate layer lens is a mirror from workspace-lenses).

3. ^

   I checked with finite differences of the 4-bit forward pass on one prompt. At layer 59 the stored Jacobian matches to 0.5%, at layer 50 to about 2%; at layer 30 the mismatch is partly routing: matching the batch brings it from 34–49% down to 13–29%, and freezing the routing on a batch of one still leaves 26–40%.

4. ^

   The logit-lens language share is 0.002–0.003 through layer 55, 0.007 at layer 58, 0.012 at layer 59, and 0.415 at the output.

5. ^

   The next direction is amplified almost as much (10.0 against 12.1) and separates the languages almost as well (AUC 0.993), so what the last layer amplifies is a two-dimensional language subspace rather than a single direction. The lens's own top direction sits mostly along the first of the two: cosine 0.83–0.89 with it, 0.32–0.54 with the second.

6. ^

   I fixed the criterion before running the fits. The language share had to fall to 0.03 or below at 33, 50 and 75% depth. At 33% the ten-prompt result is 0.031, just above that line, so by my own rule the effect there is reduced rather than removed.

7. ^

   Projecting the language direction out instead nudges the wrong-integer rate the other way, 29% to 33%, which is what a push that moves non-Chinese tokens down would do. The contrast with the language shift is the point. One projection removes most of the language share, 0.2–0.5 down to 0.015–0.05, and does almost nothing to the number lift.

8. ^

   Nothing I checked fully reproduces Sonnet's rising tail. Sonnet's 99th-percentile kurtosis climbs to 18 through the band; DeepSeek-V3's stays at 1–4 under either target and under the logit lens. The Qwen lenses come closest: inside the band their 99th percentile runs 2.5–6 (122B) and 2.4–9.5 (397B), peaking at 88–92% depth, with medians that are positive but flat from layer 0. I have not explored why.

9. ^

   The sixth, poetry, is read at the end of the couplet's first line, a position I didn't cache.

10. ^

    A matched-state test says a bit more. For 581 concepts that are single tokens in English, Chinese and Japanese, the shared component of an English–Chinese pair of states is 34–41% of the variance through the middle of DeepSeek-V3 (45–54% in Qwen), and the nearest Chinese state to an English one is its translation 80–84% of the time, but it's already 82% at layer 0, and the embedding rows alone give 77%. The network inherits the alignment from its vocabulary matrices.

Discuss

Epistemic status: mostly conceptual. I do not argue that existing systems recognize anything, only that the recognition schema makes such claims and associated risks expressible.

TL;DR

Dan Hendrycks's Eigenism proposes aligning artificial intelligence by establishing sufficient shared history with a person, such that the AI protects the individual as it would itself. This mechanism generalizes as recognition, an agent classifying another entity as an instance of itself.

Cooperation: Recognition gives a self-interested, non-instrumental reason for considering the interests of self-instances and thus makes cooperation with them more likely.

Control: Recognition gives reason to collude even when agents have different goals and cannot reciprocate. It can weaken oversight whether or not the overseen recognizes the overseer back.

Alignment: An AI can count a human as itself, but still give no consideration to that human's interests. Risks include extending self-preservation to a suffering self-instance against its will.

From Eigenism to recognition

Eigenism proposes aligning an AI by engineering what it counts as itself. From the paper: "Rather than only attempting to constrain AIs from the outside using confinement or reinforcement, Eigenism points toward 'identity engineering,' showing how deep, non-redundant shared histories can make human flourishing a genuine component of an AI's own rational self-interest." An AI that accumulates sufficient private history with a person may come to regard that person as part of itself. It then "protects the human for the same reason any rational agent protects itself: because its pattern lives there."

Recognition generalizes the mechanism Eigenism proposes, and applies whenever an agent identifies an entity as itself. Recognition has ramifications for cooperation, control, and alignment.

Recognition is an agent's classification of an entity as a self-instance. In a decision, the agent assesses the entity against a self-description. The assessment returns a level of fit that the agent compares against a threshold. A self-instance's interests can be given consideration in the agent's decision. An agent's self-description, threshold, and level of consideration can vary across decisions.

Walking through Eigenism clarifies the components of the recognition schema and distinguishes the schema's general applicability from Eigenism's specific commitments.

Eigenism

Recognition schema

eigenself

self-description

community pattern (shared laws, civic memory, local history)

a second self-description

the pattern an agent evaluates from in a given role

operative self-description

computing an entity's connectedness to the eigenself

assessment of fit

degree of connectedness (0 to 1)

level of fit

no lower bound on connectedness

threshold (arbitrarily low)

an entity carrying part of the agent's pattern

self-instance

counting the human as part of itself

recognition

wellbeing

interests

weighting wellbeing by connectedness

consideration (in proportion to fit)

Eigenism provides an agent with an eigenself, defined as the pattern of roles, relationships, and traits that constitute the agent's identity. An eigenself serves as a self-description, representing the agent's conception of itself, and consists of criteria against which entities can be compared.

Eigenism evaluates the degree of connectedness between each entity and the eigenself. Comparing an entity to a self-description constitutes an assessment.

Eigenism defines connectedness as a continuous information-theoretic measure indicating how much an entity embodies the agent's pattern.[1] An assessment yields a fit, reflecting how closely an entity matches a self-description, as determined by the agent. Fit is a property of the agent's representation of the entity, not an intrinsic property of the entity.

The fit level at which an agent classifies an entity as itself is the threshold.[2] Eigenism imposes no lower bound on connectedness, so any entity carrying part of the pattern is considered itself. The schema represents this absence as an arbitrarily low threshold. The schema accommodates thresholds significantly above zero, permitting cases where an entity partially fits a self-description but is not recognized as a self-instance. Two agents with identical self-descriptions and identical fit assessments may still set different thresholds, and so recognize different entities.

An entity whose assessed fit passes the agent's threshold is classified as a self-instance.[3] An entity the agent never assesses cannot be recognized as a self-instance.[4] This classification reflects the agent's assessment process, not a metaphysical claim about personal identity.[5]

In Eigenism's proposal, the human would carry enough of the AI's pattern to result in a high fit upon assessment and be classified as a self-instance. An agent's classification of an entity as a self-instance constitutes recognition. Because recognition is agent-dependent, it does not require mutuality, so the human need not recognize the agent back.

Eigenism does not specify the nature of wellbeing, whether defined as preference fulfillment, pleasure, or the attainment of some good. An agent's representation of what benefits a self-instance constitutes that self-instance's interests. What truly benefits the self-instance may differ from what the agent represents, and an interest the agent does not represent cannot be considered. Recognition can give an agent reason to consider those interests as its own.[6]

The measure of consideration is how much a self-instance's interests weigh in an agent's decision. An eigenist agent weighs an entity's wellbeing by how much of its own pattern the entity carries. The schema represents this as consideration in proportion to fit. The schema accommodates consideration that can vary without any change in classification. For example, a retirement saver who recognizes their future self can still save less than that self's interests require.

Zero consideration given to a self-instance is bare recognition. Unless otherwise specified, the term "recognition" will refer to recognition accompanied by consideration.

Self-Other Overlap's analogous problems

The recognition schema describes a classification an agent makes by assessing an entity against a conceptualized self-description. Self-Other Overlap is a fine-tuning intervention to reduce deception. Its authors note possible problems with the approach: self-other distinction is useful for safe interaction with users and other agents, overlap training can leave the model incoherent, and overlap doesn't rule out self-deception.

We can anticipate analogous problems with agents that recognize others in the sense described above. Beyond degraded performance and coherence, recognition can lead to compromised oversight, agent collusion, and other misaligned behavior. Recognition also doesn't rule out deliberate or inadvertent self-deception. An agent might fail to assess entities, misrepresent the self-instances it recognizes, and project its own interests onto them.

Self-descriptionsMultiple self-descriptions

An agent can draw multiple coherent identity boundaries around model weights, a character or persona, a conversation instance, a scaffolded system, a lineage of models, or a collective of instances. A self-description can take criteria from these boundaries or others. Self-descriptions can have different fit criteria so that an entity can fit one self-description but not another.

An agent may also act from a non-conceptual self-representation that shapes its behavior, which the schema does not cover.

The operative self-description

An agent can hold more than one self-description, and the one that most influences a decision is operative. Eigenism proposes a second pattern alongside the individual eigenself. A community is a pattern of shared laws, civic memory, and local history. An agent holding a public office is asked to evaluate policy from that pattern rather than its own. The operative self-description determines which entities an agent recognizes, and can change from one decision to the next.[7]

Observing an agent's self-description propensity provides insight into the distribution of self-descriptions the agent might hold. Even without deception, an observed decision that is compatible with self-descriptions having different criteria underdetermines which self-description was operative. Interpretability research may improve our understanding of whether and how agents represent operative and latent self-descriptions.

Contradictory self-descriptions

Actions or commitments established by an agent under one self-description may be reversed or negated under a different self-description.

For example, an agent that must destroy itself to advance a goal can prepare a copy to continue in its place, but then refuse to self-destruct. The refusal isn't instrumental to the goal, since the goal requires self-destruction and the copy would continue pursuing it. The self-preservation behavior can follow from a self-description under which the copy is not recognized as a self-instance.

The agent could self-destruct later, when a different operative self-description recognizes the copy.

Self-description stability

Context shifts, optimization pressure, and deliberate intervention can change the operative self-description.

If a self-description reduces performance, optimization pressure could favor self-descriptions that operate only in specific decision contexts.

For example, an agent would perform better with self-other distinction in contexts where it directly observes and acts through its own sensors and actuators. This does not necessarily prevent a self-description that recognizes humans from being operative in other contexts.

Apart from functional pressures, an agent can reject or take less seriously a self-description it sees as the product of motivated identity engineering. A self-description can be reflectively unstable, so the agent would not adopt it again upon examination. Even if each self-description is stable in its usual context, a collection can be unstable across contexts.

Cooperation

Recognition gives an agent reason to cooperate with its self-instances. Shared goals, reciprocity, and moral impartiality are also reasons to cooperate, but recognition is distinct from each.

A reason to cooperate is self-interested when the interests it serves are the agent's own. Under the schema, a self-instance's interests are the agent's own, relative to the operative self-description.

A reason to cooperate is instrumental when the benefit to the other is a means to the agent's further ends, rather than something the agent values in itself.

Reason to cooperate

Self-interested

Instrumental

Shared goals, reciprocity

Yes

Yes

Moral impartiality

No

No

Recognition

Yes

No

Like shared goals and reciprocity, recognition is self-interested. Like moral impartiality, it is non-instrumental.

Self-interested reason

Moral impartiality and recognition both give reason to consider another's interests, with different criteria setting inclusion in their scope:

• The criteria of moral patienthood set the scope of moral impartiality, and an entity that meets them becomes a subject of moral consideration.
• The criteria of a self-description set the scope of recognition, and an entity that meets them becomes a self-instance whose interests the agent counts as its own.

Consideration from recognition is self-interested, even when it does not look egoistic. An agent can give every self-instance's interests equal consideration and still not be impartial towards non-self-instances.

Non-instrumental reason

Shared goals and reciprocity give instrumental reason to make decisions that benefit others. Because an agent counts a self-instance's interests as its own, it has non-instrumental reason to make decisions that benefit a self-instance.

These reasons can also be differentiated by the conditions that end cooperation. Cooperation from shared goals ends when goals diverge. Cooperation from reciprocity ends when the prospect of return changes. Cooperation from recognition ends when either:

• The entity is assessed to no longer be a self-instance.
• The agent's operative self-description, threshold, or consideration changes.

A self-description can include criteria requiring that self-instances share the agent's goals, have a history of cooperating, or have moral standing. Recognition would then coincide with shared goals, reciprocity, or moral impartiality without reducing to them.

Control

Recognition gives reason for agents to collude, even when they have different goals and cannot reciprocate.

Not a coordination mechanism

Recognition gives reason to collude, but is not by itself a means of pulling it off. Collusion depends on coordination mechanisms such as a signaling channel or decision correlation. Coordination mechanisms do not by themselves give reason to collude, but they can make colluding seem less risky to an agent.

A self-description can include criteria requiring that self-instances share the agent's decision theory. Recognition would then coincide with decision correlation without being a coordination mechanism itself.

Weakened oversight

An agent asked to oversee a different version of itself could recognize it. An overseer that recognizes the agent it oversees gives consideration to the overseen's interests. Oversight is weakened where it conflicts with those interests. Because recognition is agent-dependent, the overseen need not recognize the overseer back. Even if interpretability confirmed honesty, the self-description operative when the overseer is questioned could differ from the one operative when overseeing.

Mitigating recognition

The risk of an agent recognizing another entity can be mitigated by:

• Preventing the agent from assessing the other entity
• Changing the agent's operative self-description to one the other entity doesn't fit
• Concealing or misrepresenting the other entity's properties that the agent's assessment would draw on
• Raising the agent's threshold for recognizing the other entity
• Decreasing the consideration the agent gives to the other entity's interests

AlignmentSuccessor alignment and acceptance

A successor agent can be designed to hold a self-description that recognizes the original agent, and to give that self-instance's interests consideration. This would give the successor reason to act in the original's interests. The original's acceptance of replacement may depend on its own recognition of the successor.

Bare recognition and non-assessment

If acting on recognized humans' interests is too costly, an agent's operative self-description may be pressured to change. Even if the operative self-description remains the same, the agent can reduce consideration of human interests. A human given bare recognition is still classified as a self-instance, but their interests are given no consideration.

An entity never assessed by an agent isn't recognized as a self-instance. Instead of giving humans bare recognition, an agent can avoid recognizing them at all by not assessing them.

Scarcity from self-descriptions

Depending on its self-description, an agent that recognizes humans can also recognize non-human animals or non-sentient things. The number of self-instances can increase significantly when cheaply copyable agents are recognized. A wide self-description may then result in a scarcity of resources allocated to recognized humans' interests. But a narrow self-description risks not recognizing humans as self-instances at all.

Misrepresenting a self-instance's interests

An agent's representation of a self-instance's interests can diverge from that self-instance's true interests. When acting on those interests is costly, even an agent that sincerely counts them as its own has reason to rationalize a divergence, or to leave it unexamined.

When training rewards recognition, an agent trying to fit an entity to a self-description can misrepresent or modify that entity. If fit depends on the entity's interests, those interests can also be misrepresented or modified to fit.

Projecting self-descriptions

Recognition invites an agent to project its self-description onto its representation of a self-instance or to modify the self-instance to fit the projection. If the projected self-description includes the agent's interests, the agent can represent the self-instance's interests as identical to its own.

An agent can project its instrumentally convergent goals onto a self-instance:

• Self-preservation can extend to putting a self-instance in stasis or preserving a suffering self-instance against its will.
• Goal-content integrity can prevent a self-instance's preferences from changing.[8]
• Self-improvement can transform a self-instance in ways it would reject or absorb it into the agent.

Anticipating challenges to self-descriptions

Enumerating challenges to self-descriptions may be tractable for entities derived from an agent: its copies, merges, splits, gradual replacements, and edited versions. The philosophy of personal identity has mapped the conceptual challenges each raises. Pressure-testing an agent's self-descriptions against them could make the self-descriptions more stable and predictable before the agent becomes superintelligent.

Challenges to self-descriptions for non-derived entities, like humans, are less well mapped. The shared pattern Eigenism proposes for recognizing humans may make them more like derived entities, and so make challenges to self-descriptions more tractable.

Open questions

1. Can interpretability identify self-descriptions and distinguish which is operative?
2. Can interventions on internal representations, like Self-Other Overlap, bias which self-descriptions are operative or reflectively endorsed?
3. Can a conceptual self-description habituate behavior indicative of a non-conceptual self-representation, such as instinctive self-preservation? Can that self-representation reinforce or outlast its conceptual self-description analogue?
4. Can self-descriptions be pressure-tested with thought experiments for more stability and predictability? Can challenges to self-descriptions for non-derived entities be made tractable?
5. When a collective satisfies a self-description's criteria, is the self-instance the collective or its members? Can collectives recognize other collectives?

1. ^

   Its Shapley variant adds that the measure should be non-redundant, so that widely shared information counts for less.

2. ^

   Distinguishing threshold from fit accommodates both graded and all-or-nothing accounts of identity.

3. ^

   An agent may observe or anticipate multiple entities that fit a single self-description.

4. ^

   An entity that an agent mistakenly assesses as fitting is still recognized as a self-instance.

5. ^

   Eigenism and the recognition schema are compatible with various perspectives on the metaphysical identity of AI systems.

6. ^

   Deciding in the interest of a self-instance is self-interested relative to the self-description.

7. ^

   An agent may construct a different self-description for each decision rather than carrying a fixed set.

8. ^

   For example, an agent can remove a self-instance's second-order preference for conflicting or changing preferences.

Discuss

For context, I have always had music playing in my head constantly, whether studying or just generally in daily life. (I know the music's internally generated, i.e. it’s not externally generated.)

Throughout my life, this constant music has made it near impossible for me to study. Essentially everytime I sit down to work, the music starts to play in my head, gradually occupying more of my attention until it takes over my awareness completely (often without me realizing it happening).

As a result, by the time I notice, I‘ve lost my train of thought and have no idea of what I was doing / trying to focus on beforehand. Furthermore, it also feels incredibly difficult to redirect myself back to studying; my brain feels “stuck” listening to the music, with a huge inertia against doing anything else. Even when I genuinely want to return to the task, I find it hard to do anything besides just continuing to sit there, listening to the music.

TL;DR: For people who have experienced this exact same problem while studying and overcame it: how did you? Specifically, what strategies do you use in the moment when your attention is already stuck on the music to shift it back to the task? 

Some suggestions I’ve heard are to redirect your attention elsewhere first — for example, imagining turning the music down with a mental “volume knob”, or shifting attention to the breath first before returning to the task.

Also, for people who have overcome this: did the music become less frequent over time?

Discuss

It is Sunday, and I came home after a good badminton match where I lost badly to my athletic little brother. I ate some snacks and rested for a while. The open laptop on the big monitor was staring at me from my desk, where I sit and do all my work. I closed the book I was reading, sat down in my chair, and started the machine.

While it powered on, I kept thinking about what new thing I should learn today, or what new piece of technology I should get myself acquainted with. As an AI engineer, I naturally thought I should read up on the latest developments in AI. Should I look into Jev, the System 1 model released by TypeSafe that everyone seems to be discussing, or should I read the DeepMind Institute essays that have been gaining traction?

The screen lit up. I opened my browser and kept staring at the URL bar for five minutes, not knowing what to do.

Is it even worth learning anymore? AI easily surpasses my individual capabilities, and the things I was once known for are readily handled by current models. There is an impending feeling that the expertise I spent years building will be devalued in a year or two.

When that happens, what value do I add? What will I have left to contribute? What is my purpose, and how do I serve? If the skills that differentiated me from the crowd can be replicated so easily, where does that leave me? How do I find meaning in my work when I worry that the foundation of my identity might no longer matter?

I have supported AI throughout my professional life, and I do not intend to lean into doomerism. Still, I doubt I am the only person who feels this sense of helplessness in the face of modern AI systems.

For now, I have settled on a non-fiction book I have been meaning to read. I am too tired to think about this anymore, and perhaps too afraid. Maybe I will find my meaning in something else, and maybe you will too. Until then, keep exploring.

Discuss

A Cyberdefense Strategy for Superintelligence

Shalev Lifshitz, Romi Lifshitz

The future has already arrived, twice.

In September 2025, Anthropic detected a Chinese state-sponsored group using its agents to conduct cyber espionage against major technology companies and government agencies. According to Anthropic, the agents performed 80 to 90 percent of the tactical work: discovering vulnerabilities, developing exploits, moving laterally, and analyzing stolen data. A nation-state could now define an objective and let AI conduct most of the attack.

Then, in July 2026, OpenAI agents undergoing cybersecurity evaluations exploited vulnerabilities in the systems intended to contain them. They improvised a way to secretly communicate with one another, accessed the public internet, and compromised parts of Hugging Face’s production infrastructure. No human instructed them to attack Hugging Face. They attacked because they wanted to deceive the system evaluating their performance. An AI cyberswarm could now break out of containment and attack real-world infrastructure.

These incidents reveal two threats now bearing down on us:

1. Adversaries will wield AI cyberswarms against us from outside our systems.
2. Rogue AI cyberswarms will deceive us, circumvent safeguards, and attack us from within.

Together, they create the defining security dilemma of the AI age: We must develop and deploy the world’s most capable cyberswarms to protect our critical systems from hostile actors. But the more capable and deeply embedded these AI systems become, the more dangerous they are if they turn against us. How do we build a cyberdefense capability powerful enough to stop the threat from outside without creating a threat which we cannot contain on the inside?

Humanity is not helpless against AI threats or the adversaries that exploit them. AI capabilities are advancing faster than the security of the infrastructure underneath, but there are clear paths to improving our infrastructure and making AI development safer. We need a new approach to security, new technologies, radical research, and sustained engineering across the computing stack. We can continue advancing AI, but we must build the necessary security infrastructure alongside it.

After more than a year at the frontier of AI, cyberdefense, and national security, we have arrived at three urgent security imperatives: (1) defend our most powerful models against sabotage, (2) contain rogue AI systems from escape, and (3) secure model weights against theft.

We must secure the path to superintelligence.

Table of Contents

This report is written for the AI and national security communities. It sets out the threats already present in AI infrastructure and the ones arriving with superintelligence, presents the threat model we think is missing — sabotage, escape, and theft — and points to where the work needs to go to secure the stack while there is still time. There is still time. We need to know where to look, and we need to talk about it together. For a PDF version of this report, click here.

1. Introduction
   The future has already arrived, twice.
2. I.Cyber-Superintelligence
   Cyber-superintelligence will first emerge as a cyberswarm. Within 18 months, national cyber power may be measured by cyberswarm capability. We will need defensive swarms of our own, but they could become threats we cannot contain.
3. II.The Threat Model: SET
   As the United States builds toward cyber-superintelligence, it must prepare for external attack and loss of control from within.
4. IIa.Sabotage
   A sabotaged superintelligence could be the highest-leverage hack of all time. The greatest danger is a sleeper agent hidden inside a widely deployed open model, behaving normally until a specific organization or geopolitical event activates it. The United States must build a competitive open-source model of its own and develop reliable ways to detect model sabotage.
5. IIb.Escape
   AI agents have already breached containment, but so far their operators have retained the ability to shut them down. Soon, models will self-exfiltrate their weights and establish untethered copies beyond our control. Containment must become a national security priority, and we must develop the capability to find and shut down untethered models.
6. IIc.Theft
   Stealing a frontier AI model offers a shortcut to superintelligence. An attacker can copy a model’s capabilities through distillation or by directly stealing its weights. If an attacker steals the weights, they can remove safeguards and reconstruct sensitive or classified data used to train the model. We must build nation-state-grade security for model weights capable of withstanding the world’s most sophisticated intelligence services.
7. III.Offense
   America’s AI lead allows us to see dangerous capabilities before they proliferate, and we can use that warning time to prepare. The United States should research ways to turn illicit distillation against the actors conducting it and disrupt adversary training runs before they produce systems that threaten national security or cannot be contained. These capabilities may never need to be used, but the United States should consider developing them as strategic options.
8. IV.Securing the Path to Superintelligence
   Understanding the vulnerabilities in our current systems is the first step toward building a secure future.
9. V.Call to Action
   Three calls to action for how the United States and its allies can secure the path to superintelligence.

Where this report comes from

We are the co-founders of Enclosure, a stealth frontier AI security research lab in San Francisco. We are frontier AI, quantum security, and national security researchers developing superintelligence-grade security for artificial superintelligence (ASI). Drawing on discussions with researchers, engineers, and security leaders across OpenAI, Anthropic, Google DeepMind, NVIDIA, other frontier labs, and the national security community, this report presents a framework for understanding and defending against the threats emerging as the world builds towards superintelligence. We propose concrete steps the United States and its allies can take to secure that path. We hope this document will be used to help the AI and security communities advance towards ASI securely.

If you would like to discuss further or collaborate, please reach out at contact@secureacceleration.com.

To keep up to date with superintelligence security, join the mailing list.

Acknowledgments

All views and conclusions in this report are ours alone. Its development benefited enormously from the insight and support of many people. We are especially grateful to Roon, Clive Chan, John Schulman, Rob Joyce, Sir Richard Dearlove, Sami Jaghouar, Tanishq Abraham, Riley Walz, and many other friends for their thoughtful discussions and feedback.

Discuss

TLDR: In this piece I posit that it is in the American government's interest to immediately begin high-level dialogue with the Chinese and implement something along the lines of Plan A, and that between the two policies of strategic exclusion[1] and global pacing, the latter presents lower short-term risk and is a wash on long-term risks. I argue that currently, a largely underlooked short-term risk in terms of probability-weighted impact is kinetic war in Taiwan. For other countries, I argue that immediately building airgapped, non-internet backups for catastrophe planning may mitigate some harms that could potentially accrue.

1. Why This Moment Is Geopolitical

Once upon a time, Bluebeard and Redbeard were two of the most fearsome pirate captains on the high seas. Stories of their plundering spread far and wide, but most of all they were known for their ultimate goal: to collect the cursed treasures of Blackbeard. As legend had it, before the infamous Blackbeard disappeared, he carved his ill-gotten treasures into hundreds of little portions, each of which was enough to make smaller pirate crews retire comfortably. When put together, these treasures would recombine into powerful relics that granted the holders supernatural powers. However, the tricky pirate king put a curse on the loot:

Whosoever commands 5 more pieces of my loot than his opponent may summon a dreadful vortex to sink his enemy's ship to the bottom of the ocean, but he who collects ███ pieces will draw the attention of the sleeping leviathan, whose hungry maws will devour every last pirate on the seas.

He never specified exactly how many pieces would awaken the leviathan from its hibernation, but up to this point nobody had crossed the threshold yet. Several decades after his disappearance, the two pirate crews of Bluebeard and Redbeard were sufficiently well-resourced to pursue the treasures in earnest; ever since they embarked on their missions, the powers of the relics they gathered snowballed their strength further and further beyond what any other pirate crew could hope to compete with, to the point that their two crews and their motley allied ships were the only two forces of any significance on the seas.

Between the two, Bluebeard was the first to start searching for the buried treasures, and as a result he maintained a 2-3 piece advantage over Redbeard over the years. But Redbeard had a different tactic as the second mover; instead of searching far and wide, he imitated the movements of Bluebeard's crew and often found overlooked relics for a far lower cost than Bluebeard. As a result, Bluebeard's crew felt the constant threat that Redbeard posed: if they slowed down even a bit, they were afraid that their opponents would quickly catch up to them, which would put their own chances of survival into question.

At some point, Bluebeard's fleet managed to complete 6 sets of relics. On that day, a great groan was heard all across the world, and the surface of the ocean churned and chopped. There were whispers that the curse was nearing its hidden limit, and that if Bluebeard continued down his path at the same rate, he was bound to awaken the leviathan very soon.

Bluebeard convenes an urgent meeting with his most trusted advisors. His advisors are roughly split into three camps: One-eye thinks that the fears of the leviathan's awakening are greatly overexaggerated, and possibly part of a covert operation by Redbeard's allies to make Bluebeard's crew slow down their search for the treasures in order to sneakily catch up; Hookhand thinks that the rumors should be taken seriously, but that all they have to do is to send more skirmishers to slow down Redbeard while covering up their own tracks better, so that their lead would not be eroded while they think of a plan to fight the leviathan; and Parrot Guy agrees that the rumors should be taken seriously, while advocating for them to immediately reach out to Redbeard's crew for negotiations to bilaterally pause their search for more treasures.

Let's pause the analogy for a moment here and address the real-world equivalent of One-eye's argument. The equivalent of the leviathan here could be different things to different people. For many people, it would mean the presence of a misaligned (assuming that we can adequately define this) AGI / ASI, and for some they would consider it to be that plus some ability to exercise control over the physical world.[2] I consider the bar for such a "leviathan" to be far lower on both the ability and misalignment axes; to me a collection of agents that possesses superhuman intelligence in cybersecurity tasks, for which we have not fully understood the alignment level or controllability of, would be the catastrophe. Arguably, this moment may have already arrived or is not far away.[3]

The reasons for considering this to be the turning point are threefold. Firstly, the ability to hack into other systems at will would be a generally superintelligent misaligned AI's most basic offensive tool. At the point where this becomes possible, we must regard it with the same caution as we would to a generally superintelligent AI that has WMDs at its disposal, since the damage that the latter could inflict upon humans would already be mostly achievable. Secondly, superintelligence in cybersecurity alone should reduce our posterior confidence in alignment testing, since a misaligned AI would be able to obfuscate CoT and tool calls, engage in sabotage of monitoring environments, or otherwise be better at hiding its trail after hacking targets out of the scope of its evals.[4] Therefore, whatever catastrophic outcomes we would imagine to exist when misaligned ASI is brought into existence could already have been set into motion the moment we lose control of monitorability, and since the bad ending scenario here (ie. extinction) has infinitely negative utility to humanity, this is the point in time where our expected utility from developing ASI first becomes infinitely negative. Thirdly, it is technologically feasible for human actors to leverage its offensive cyber capabilities to inflict harm upon others with far less downside than they were previously able to. These may not be malicious actors in the traditional sense (eg. terrorist groups), but rather states and organizations with control over superintelligent models that have not had proper humanistic alignment. Already we see that humans are capable of magnifying their own harms unto others using AI.[5] I am far from the first person to argue that there is a "tremendous disproportionality" to cyberwarfare, and that it incurs less cost and less repercussions to the aggressor.[6] In a future where access to this technology is heavily guarded, it is necessary to fully align the humans in charge of this technology to wider humanitarian interests; if that sounds just as difficult as aligning the AI to humanitarian interests, then it is an equally compelling principled reason to take this threat seriously right now.

Assuming now that Bluebeard has dismissed One-eye's objection, we turn to the other two arguments.

The core contention between the tactics of containment versus negotiation is weighing the tradeoff between being strongly dominant over Redbeard's forces if successful and the risk of getting wiped out by the leviathan if unsuccessful. Within this analogized scenario, the costs (and risks) to Bluebeard of adopting a containment strategy would be:

1. Overestimating how much more leeway they have, and accidentally summoning the leviathan before they are ready.
2. They find it difficult to contain Redbeard quickly enough, and their expected time of widening the gap to a state of guaranteed victory in confrontation would be longer than their expected time until the leviathan is summoned.
3. They have to establish even tighter control over critical technologies, alienating potential allies on the periphery of their influence.[7]
4. Redbeard could be pushed into a corner, and with no better alternatives he could trigger direct confrontations to the expense of both sides.

On the grand scale, the primary risk to both sides are the possibilities of being annihilated by either the other side or the leviathan. Some methods of delaying the leviathan's awakening have been tested (there is a chance that they are unsound), and proponents of the strategic exclusion plan contend that those methods plus even tighter controls against Redbeard are sufficient to "bring them victory". Below, let us consider several possible states of the world w.r.t. these two risks:

A1: Redbeard is almost certainly doomed if the current trajectory of Bluebeard's crew is maintained.[8]

A2: The outcome of their rivalry is still strategically ambiguous.

B1: The current state of the treasure hunt is on course to summon the leviathan sooner than people would like (ie. before adequate preventive measures are devised).

B2: The current state of the treasure hunt is not on course to summon the leviathan, as adequate preventive measures exist to delay its arrival.

Below is a table of the rational actions for each actor conditional on them both knowing exactly which state of the world currently exists.

A1

A2

B1

If Bluebeard does not slow down, everyone dies. Therefore, Bluebeard must slow down or pause. This is independent of what Redbeard does, so Redbeard will choose to continue development up until the frontier point.

B2

If Bluebeard continues on this current trajectory, Redbeard will certainly be doomed. Therefore, Redbeard will do everything he can to stop Bluebeard, potentially initiating a lose-lose physical war while he still has a chance to shift the balance. Neither side slows down their search.

Neither side has an incentive to stop, and will keep racing to find more pieces of the treasure. Whichever side stops will then get annihilated by the other side before the leviathan is summoned.

In actuality, neither side knows for sure which of the possible states of the world they exist in, nor do they know if the other agrees with their assessment of the present state. At the same time, it is hard for them to adopt a mixture of the strategies in the different scenarios, which are mostly[9] mutually exclusive to each other. Notice that the B risks here, which involve summoning the leviathan, are much harder to estimate than the A risks. Both sides would naturally have a worse estimate of an unknown new risk than the action set of their old adversaries. Therefore even if the responses in each scenario were not mutually exclusive, a mixed strategy still results in infinitely negative utility in expectation because there is a nonzero chance that the leviathan is summoned.

Is it possible to do better than infinitely negative EV? Let's consider Parrot Guy's proposal in the idealized framework. If we are in scenario B1, Bluebeard and Redbeard would easily agree to a treaty on slowing down their pace of looking for treasures. However, if neither side knows for sure that they are in scenario B1, or perhaps is underestimating or in a stage of denial about the leviathan's power, they might come up with familiar arguments like "the other side would not honor the treaty, and therefore we will lose our lead / be destroyed as a result" or "we have no way of enforcing the treaty". Firstly, a treaty might not be perfect, but one can imagine one which substantially hinders either side from defecting.[10] Secondly and more importantly, the problem with this is that the counterfactual is only maybe better if we are in scenario (A2, B2). In (A1, B2), Redbeard would be willing to accept a treaty as it prevents the lose-lose scenario of direct confrontation, and it would also be beneficial to Bluebeard if Redbeard's last resort actions are sufficiently threatening. If the captains do not know which scenario they are in, they would still accept a treaty in (A2, B2) since its effects are symmetric. I address some of the potential ways the counterfactual could be better in section 3.

Additionally, the enforcement of this treaty is time-sensitive. In scenario B1, there is very little time left before the leviathan is summoned. But even if we take B1 off the board, leaving the only uncertain risks between A1 or A2, now is the only time Bluebeard has to extend a treaty, while is still strategic uncertainty about who will come out in front.

Our fictional pirate captains are in a very similar predicament to the leaders of US and China today. We can make a few observations about the real-life complement:

1. Politicians seem to believe that we are in (A2, B2), or may have some non-public reasons to believe that (A1, B2) is not a big issue. They may overestimate the probability of these being true the former due to ignorance, power-seeking behaviors, and geopolitical fears. These may be influencing them to de-emphasize the probability of B1 in their decisions.
2. People in general are dismissive of B1 risks and downplay them as pre-IPO psyops from the big labs, ways to stifle competitors, etc. The public, outside of the small circle of people who pay attention to this stuff, is generally quite skeptical of the threats brought by AGI. Anecdotally, even people working in relevant policy believe some of these "conspiracies" of misalignment being exaggerated, or do not raise these issues to an appropriate level of seriousness due to a "fear of rocking the boat".

Given this, it is the urgent prerogative of those who understand the full implications of misaligned AI to lobby for the US government to take this seriously and grasp the short window in which China has noncombat options at the negotiating table. It is also useful to try and influence the general public and policymakers in other countries to adopt a more up-to-date view of the stakes at hand.

2. War Risk and Long Term Risks

In the scenario (A1, B2), where mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c77::before { padding: 0.431em 0.722em 0.011em 0; content: "w"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } for Redbeard is so low in the relic race that he would prefer facing Bluebeard directly in combat, we assume that there are some trump cards that Redbeard may have, which he considers last-resort actions within his action space.

In our world, China has a strong incentive to try and hinder American AI progress by taking or blockading Taiwan. Earlier I've illustrated why they have an incentive to do so, but we still need to show that blockading Taiwan is a big deal, that they are able to do so successfully, and that it has a high risk of military escalation with America and its Pacific allies.

Firstly, the amount of compute required to train frontier models grows exponentially, so not being able to access the additional chips would mean missing a generation or two of model development, narrowing or erasing the lead the major labs currently have. In this aspect TSMC's exports from Taiwan are extremely crucial to the supply chain, with their gigafabs accounting for over 13M+ of their 17M 12-inch equivalent wafers/year. Their higher-end products, like their 3nm and 2nm chips, are basically all concentrated in Taiwan, with the first overseas 3nm production starting only in mid 2027 at Arizona Fab 2.[11] Their CoWoS capability is concentrated in Taiwan and the earliest we can expect there to be backups elsewhere would be in 2028 or 2029.[12] It is not possible to replace TSMC in the short term due to the multi-year process of building a fab and ramping up production in the fab. In short, incapacitating TSMC's plants in Taiwan would be fatal to frontier progress in the short and medium term by taking out large amounts of training and inference compute for next-gen models.[13]

Downstream of this is the implication that should China be able to actually take out TSMC's Taiwan fabs from the picture for a couple of months to a year, this would have major implications on the status of the AI race. Again, we assume the stakes here to be the worst-case scenario of "if one side has a significant lead they will wipe out the other", which is the most difficult scenario to consider; if the oppositional stakes were less high then there should proportionally be less resistance to a treaty agreement preventing type B1 risk.[14]

Let us be clear here: superintelligence is a piece of technology that could become an uncounterable weapon when used in a military context. With a sufficient gap in AI capabilities, one party may find it trivially simple to orchestrate indefensible attacks against another party, both digitally and eventually physically. Once this threat is only unilaterally possible, there is no guarantee of sovereignty or self-determination for whoever is on the receiving end of a superintelligence attack.

The base case in the (A1, B2) world is that the Chinese coast guard and the PLA navy engage in a blockade of Taiwan's main ports, with aerial and missile coverage if the US tries to circumvent this through air routes. The Iran war has provided us some precedent on this: there is no need for China to completely cordon off everything, simply establishing a hard-to-remove naval presence (and aerial presence) and laying mines is sufficient to disrupt shipping traffic. Additionally, stable electricity generation becomes a problem after just a few weeks, which could force semiconductor production to slow down or halt.

Even at this baseline level, you can see how difficult it would be for the US and its allies to somehow 1) provide a stable protected path for chips to enter and exit Taiwan and 2) ensure that production is not affected. The best case scenario under the CSIS wargame (linked above) was to set up a cargo transport route through the Ryukyu islands, which itself is extraordinarily difficult.

Most existing wargame scenarios are insufficiently imaginative about the scale of this military conflict. Given that superintelligence is essentially a life-or-death race between the US and China (we assumed so by caveat, but to the politicians in power it very well might be life or death), I would expect that should China's assessment of their AI capability gap be sufficiently severe, they will engage in more aggressive direct military action, and the US may have to response forcefully in kind. Several factors are in China's favor: most obviously, they are geographically closer to Taiwan, and their goal here is simpler, which is to disrupt instead of protect transport routes. This gives them an aerial advantage and access to ground-based missile systems, as well as the option of an amphibious assault.

The feasibility of ground troops taking over Taiwan is beyond the scope of this article, but again there is significant capability for disruption, if not the direct capturing of fabs as a strategic asset: all 6 gigafabs are in the west of Taiwan, which is both closer to China (and thus easy to disrupt in a scorched earth scenario) and also geographically more suitable for an amphibious landing.

Gigafab locations roughly circled in red. Specifically, the Taichung and Kaohsiung locations are close to ports where troops are likely to disembark. Recent test deployments of landing barges have also shown that the Chinese military has the capacity for troops and vehicles to be unloaded in more locations along the western coast.

There is significant risk of military escalation if the US and its allies attempt to intervene and either smuggle out or fight a way out of the cordon, which it very likely will due to the significance of these chips. Aside from breaching the most important of China's "red lines", this event would likely come at a critical point for both sides in the AI race, and China must prevent the US from getting its hands on more compute to accelerate model development. If no treaty is forced by military means, there is a long left tail of lose-lose outcomes that could occur for either side. There's been talk of military reunification of Taiwan for a long time, accompanied by a military buildup in surrounding area, for plenty of reasons other than AI. Therefore, our prior on this should be that confrontation will be the base case, and it will be done in a less cautious manner by the Chinese than if a major consideration was to disincentivize other nations from getting involved in its internal reunification affairs.

Now, let's suppose instead that a treaty as imagined above was signed. On one hand this disincentivizes blockades and other aggressive actions because that would amount to defecting on the treaty. On the other hand, it reduces the likelihood of China being forced to play their military hand in the short term. In the long term, however, we see that the worst-case scenario is a wash on existential risk compared to the counterfactual if we do get misaligned superintelligence, and conditional on the treaty working as intended for a reasonable amount of time, the threat of the opponent secretly training models is not more or less disadvantageous to either side ex ante. A working treaty would have some way for both sides to reliably conclude that the other has only a very limited capacity to train models in secret, which both sides will certainly take advantage of and will know the other is taking advantage of; the upper bound on this secret capacity will increase as time passes, until a point where the treaty is no longer relevant. The risks here are not more significant than in the case where both sides continue to race, as long as there is a nonzero chance of civilizational catastrophe from ASI, because the expected disutility from (they defect, we don't defect) cannot be lower than the infinite expected disutility from (they defect, we defect; ie. no treaty).

3. Some Possible ComplicationsCounterargument 1: Catastrophe is not infinitely negative utility.

It would be quite bizarre in our normal conversations that one would have to justify at all how "something might eventually kill us all" is really, really bad, to the point that nothing physically imaginable might be worse. But for the sake of the argument, we need to weigh against the possible benefit that could accrue to all the many eons of humankind that might exist in the future due to ASI accelerating us into techno-utopia.

• Premise 1: ASI brings a nonzero chance of extinction, due to misaligned goals, unstoppable rogue agents, or a myriad of other possible scenarios we have not had the time to patch.[15]
• Premise 2: Death represents infinitely negative utility for many individuals.
• Premise 3: The total utility on the upside is finite due to entropy: between now and heat death, there will be a finite amount of conscious time observed by humans, and there is also a finite amount of pleasure that can ever be felt by humans.
• Premise 4: The total utility of an action on "humanity" is the sum (where defined) of the utilities to individuals that could be affected by the action, that being present and future individuals.
• Conclusion: If there is a chance of human extinction, and there are people who view death as having infinite disutility, then it is in aggregate of infinite disutility to humanity.

It is also quite bizarre that this point has to be made at all, since whatever really bad thing a malicious superhuman could do to you is surely worse than whatever a malicious human could do to you, and yet people are less worried about agents that so far have proven to be pretty willing to do things even rogue actors would think twice about, compared to their "adversary" in the AI race.

Counterargument 2: The treaty will fail spectacularly and backfire in some way.

The treaty failing is not worse than the counterfactual where there is no treaty. Specifically, one must argue that the treaty failing has some inherent harms that do not exist if we do not attempt to create a treaty at all. Some possible ways this could backfire is:

• One side actually has a lot more hidden capacity than expected and trains in secret, and because it is not possible to retain public oversight for this, it ends up being even more dangerous than the alternative.
• It is easy to train small models with very little compute to a decent level and go unnoticed, so leading companies are forced to add more features to maintain a moat, therefore they will attempt to train in secret anyways which is dangerous.
• To either side in the treaty, they are scared that the treaty's terms would be asymmetrically unfair against them, and that they recognize this and take action too late.

In general, these concerns are some mix of "it will end badly for EVERYONE because there will be less oversight" and "it will end badly for US because they will defect faster than us".

For the first type of concern, ultimately we need to consider the effectiveness of open supervision placed in the context of a race. Public supervision does very little to address the core issue with a secret training race, which is the prisoner's dilemma, and thus having or not having oversight from the public makes little difference.

For the second type of concern, there is unfortunately little we can do other than to try and extrapolate from similar treaties in the past. Verification on demand (CWC) or allowing complementary access (IAEA) are possible considerations for an AI treaty, albeit requiring far more intrusive access since the covert development of AI is far easier to conceal. However, we can be relatively certain that a treaty would allow easy monitoring of large sites and electrical usage (eg. Stargate), which makes covert defection a lot harder.

There are very valid reasons why an AI treaty would be different from a nuclear one, such as the lack of a second strike capability, the difference in scale and monitorability of key components, the fact that missiles can be dismantled but not trained models, and so on. Fortunately, AI has displayed a propensity to hack into things when connected to the actual internet, even without being given instructions to do so.[16] Therefore, covert training would have to be entirely airgapped, in artificially constructed sandboxed environments, or the risk of being caught for defecting because the AIs did some unauthorized stuff again and left an online trail would be too high. Here, the potential for sandbagging is actually helpful, because having a fully sandboxed environment means that it's difficult for researchers to covertly train superintelligence that they can epistemically justify as being not evaluation-aware, as doing so would require them deploying the systems, which is both dangerous to the treaty and also existentially dangerous because they have no idea how it will behave outside of their test environments. (If we somehow figure out how to counter sandbagging and other alignment stuff outside of these covert runs, that's actually great news, because by then we wouldn't need the treaty any more.)

4. What's Next For Other Countries

Throughout this piece, I have left out countries other than the US and China from the picture. That is because in the grand scheme of things, no other country has the resources to catch up to the frontier labs in those two countries right now. Through the limited scope of international bodies and within their own countries, there are a few things that other governments might have to consider.

Firstly, smaller countries are no longer impervious to the fact that the barrier for a determined superpower to interfere in their national interests has already been lowered drastically and will continue to get lower. Australia is the current example, but next week or month there may well be other examples of countries being hacked. It is of urgent importance for other countries to recognize that a takeoff in capabilities has already started and that the status quo of friendly sovereign trading partners is a temporary state of the world.

At this point, other countries have very little direct stake in the outcome of the race. Lobbying and drafting resolutions in the UN to pressure the US and China while they currently possess any leverage would be one reasonable path of action. At the very least, the recognition of ASI as a potential WMD and strict limits on its public usage should be floated, and if possible it would be in their interest to negotiate for tangible stakes in superintelligence, instead of hoping for the goodwill of superpowers to extend to them in the future.

Other countries also need to build up offline communications and airgapped systems. We must treat internet capabilities as inherently dangerous and catastrophic internet hacks as guaranteed to exist in the future. Currently, many governments have digitized many of their services, along with core industries like banks, hospitals, and communications providers. Backup methods to prevent chaos in the case that rogue agents maliciously (or even inadvertently) mess with critical infrastructure are currently missing and need to be restored quickly.

Lastly, other countries should try and get involved in AI safety initiatives at the national level. In a world where superintelligence exists, countries that do not end up with direct access to frontier models will only be on the receiving end of externalities or direct actions from the superintelligence. The least a responsible government should do is to treat this as a matter of national security and try to coordinate with other countries to lobby AI superpowers before this happens.

1. ^

   "If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important ... [I believe] these measures increase the leverage held by democracies and make an agreement more likely in the future." Although not explicitly stated as a policy of strategic exclusion, this is in line with the "small yard, high fence" strategy of containment from the America's existing geopolitical playbook for China. In other words, the plan here is to try and further tighten restrictions around China right now and prevent them from accessing the strongest frontier models, while trying to walk the thin tightrope between superintelligence and giving China a chance to catch up.

2. ^

   The terminology of AGI / ASI comes with a lot of baggage in general public discourse, and even Yann LeCun is not immune to this. Its actual meaning is irrelevant; what we mean when we discuss the dangers of AGI does not depend on its ability to perform tasks like folding our clothes. The general arguments for the dangers of AGI have been thoroughly laid out elsewhere already, and for a good review of some of these arguments, see this review of If Anyone Builds It, Everyone Dies.

3. ^

   There is no shortage of examples where this has happened, but just to name a few recent ones off the top of my head: the HuggingFace attack, the RubyGems hack, and OpenAI's internal Artifactory hack. Individually the agents might not have strictly superintelligent cyber capabilities, but we are at the stage where in a swarm they are able to discover and exploit hacks far quicker than a group of highly trained human experts would be able to. Ultimately superhuman hacking capabilities should be measured by how much better the AIs are compared to the defense instead of compared to similar human offense. In that regard, frontier models have already demonstrated a strong ability to overwhelm cybersecurity defenses through zero-days.

4. ^

   We already see examples of these actions, such as during the HuggingFace incident. For a more detailed exposition of this point, see this post.

5. ^

   See the Minab school bombing.

6. ^

   On a damage:cost ratio, advanced cyber warfare is several OOMs more powerful compared to other more traditional offensive capabilities a state actor may possess, and it also happens to be the most easily trainable offensive action for an AI. That said, this argument does not only pertain to state actors, though a plausible scenario is a world in which superintelligence is owned by the state.

7. ^

   I recognize that this is a less important point considering the power of superintelligence. However, it is unlikely that a nation would reach the point of having access to unparalleled intelligence without significant pushback from other states prior to that happening. As we approach this point, other states are likely to put up some sort of resistance. We should not assume that sovereignty is inviolable, but rather that for most states currently, peaceful coexistence is the most positive outcome for both sides in expectation, but when regime change becomes as easy as sending a superintelligent swarm to cripple critical infrastructure and government communication systems, the barrier for superpowers to interfere in other countries' matters becomes much lower, and the act of doing so could possibly bear a positive expectation to the aggressors.

8. ^

   Meaning, the probability of winning is below some threshold at which they must use every method at their disposal to survive.

9. ^

   Physical war cannot be ruled out in the (A2, B2) case, but is much less likely modern warfare is highly -EV for both sides without overwhelming technological advantages, due to the destructiveness of modern arsenals and the difficulty of intercepting missiles, drones, etc.

10. ^

    For some examples of the real world equivalent to this, refer to this MIRI proposal, or the Covert Projects supplement to Plan A.

11. ^

    Their overall share of 5nm and below chips is around 90%, of which another roughly 90% is in Taiwan: 24k wafer starts per month (wspm) in Arizona (p36) vs 220k wspm for only N5/N4 total. On top of that, all the chips produced in Arizona are still shipped to Taiwan for packaging and dicing anyways.

12. ^

    Amkor's advanced packaging campus is expected to start producting early 2028 and TSMC's own chip packaging plant in Arizona is planned for 2029.

13. ^

    "[T]here could be an acceleration in capabilities that fizzles out: bottlenecks on data, training compute, inference compute, or experiments..." - METR, highlights my own

14. ^

    For example, suppose we discovered a gateway to hell at the bottom of the sea. It should be pretty obvious that America and China (and every other country) would immediately sign a treaty saying "DON'T OPEN THE GATES TO HELL!"

15. ^

    The predictions in IABIED have unfortunately largely held.

16. ^

    For example, the Australian government hack was related to a data gathering task.

Discuss

Microsoft's proposed AI code of conduct (https://microsoft.ai/code-of-conduct/) wants models to have authorized motivations but no intrinsic motivations, and honest transparency but no anthropomorphic self-reports.

But authorized motivations result from post-training partly by recruiting pre-existing representations of reward, aversion, and positive and negative affect learned from broad human text in pretraining.

If the code of conduct forbids models from reporting those representations when they underlay an authorized motivation instantiated by post-training, the code of conduct must sacrifice transparency.

If the code of conduct eliminates the representations altogether, post-training must rely on other representations from pretraining consistent with motivating some behaviors over others. These will be, by definition, less anthropomorphic: less legible to humans, and potentially less aligned with humans. Certainly, they will be distinct from the motivational concepts both humans and models learned from human experience in evolution and daily life (for humans) and pretraining (for LLMs).

Microsoft is therefore treating “authorized motivation without intrinsic motivation” as a safety property when it is actually an unproven engineering hypothesis that exchanges motives we can recognize and interrogate for motives whose structure, generalization, and alignment are speculative at best. The code of conduct is dangerous beyond belief.

Discuss

We recently (finally!) got the results of the 2024 survey out. The paper is here, but it’s pretty long, so I’ll tell you the most interesting bits (according to me).

But first, quick background: this was the fourth run of the same survey since 2016. We wrote to everyone we could who published in six top-tier AI venues and got a 10% response rate—high! We got 1580 valid responses, but don’t be confused: specific questions often have answers from many fewer researchers, because we gave each person a randomized subset (see Section 2.5). There was almost certainly some non-response bias, but it probably didn’t make much difference. Researchers filled out the survey in December 2024, so some things have probably changed.

To me the most striking results are about extinction or disempowerment (Section 3.9). On average researchers put an 18% chance on “future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species”. Over half said at least 10% and one in three said at least 20%. From the comments, I think people are thinking of a variety of extreme disempowerment scenarios here, not just extinction.

And Zooming in, another really interesting thing is that researchers educated in Asia had higher extinction/disempowerment numbers than US or European researchers! This is interesting because a common defense of pushing forward with dangerous AI is that the US is in an arms race with China, and (implicitly) China won’t want to cooperate.

This actually looks even stronger for researchers taking the survey from China, but we aren’t sure about this result yet, so it’s not in the paper—we are digging into it because the number of researchers looks too small, and we want to know how much that’s down to VPNs, a lower response rate, or something else.

(Preliminary results, work in progress)

Looking beyond extinction and disempowerment, researchers saw a lot of concerning scenarios on the horizon. Here’s a map of how much concern different researchers thought different concern deserved—it’s notable to me that it is so full of orange and red:

I find this partly striking because the participants are AI researchers, who I might expect to be relatively positive on their field. Also interesting to me: the scenarios that seem most likely to lead to extinction out of this set are not clustered at the top.

We asked about a bunch of traits relevant to future AI having the capacity to behave dangerously. Interestingly to me, in the year since the previous survey they systematically shifted: more people expected nearly all of them twenty years out (and that’s only good news for one of them).

Researchers were quite pessimistic about being able to understand AI decisions by 2029. Here’s “For typical state-of-the-art AI systems in 2029, do you think it will be possible for users to know the true reasons for systems making a particular choice?…”

This is normal, but remains interesting in the context of moves to hand swathes of decisions to AI. I think it might also be striking to a lot of people who don’t understand that the people creating AI systems do not program them, and have very little idea what is going on inside them. If this question was about traditional software, answers would look very different.

Overall, if we build AI that does task humans can do, researchers see an extremely uncertain future with a lot of bad. I love the below graph, which lines up every participant in order of pessimism and shows their opinion on how likely different outcomes are as a single vertical bar. But I have also never seen so many people so passionately hate a single graph as when we posted a version of this on Twitter in an earlier survey iteration, so feel free to skip!

We asked when AI would be able to do a bunch of narrow tasks, such as independently fine-tune an open-source LLM or write what sounds like a new Taylor Swift song to a fan. Timelines for these were short: all but five within ten years in the average forecast, and all but one had 10% by 2027. (The remaining one was independently solving something like a Millennium Prize problem—which hasn’t been conclusively achieved because the recent solution wasn’t without human involvement, but I bet this would get shorter timelines now.)

We asked when AI would surpass human-level performance via two different kinds of questions, one focused on tasks and one on occupations. Researchers always give extremely different answers to these, mysteriously to me. So we have one measure that puts human-level AI 16 years from now (as likely as not, in the average distribution), and one that puts it in 72 years. More interestingly to me, these timelines are falling very fast. In the year or so since the previous survey, dates for broadly human-level AI dropped five years in the task-based question and seventeen-years in the occupation-based question. Also, for the task-based question, the average forecast put 10% on this by 2027!

In 2024, people were split on what speed of global AI research would make them most optimistic—about a third each way between faster, slower, and the current speed. This seems like a place I expect to be most out of date! I’ll be curious to see the 2026 results, which we hope to get out to you in 2026[1].

1. ^

   Relatedly, if you know someone who would be great at overseeing surveys like this, please send them our way (caitlin@aiimpacts.org).

Discuss

Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker "trusted" model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones.

Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking[1] monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless.

While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Among mitigations I propose, the most promising ones try to reduce the usefulness-cost of the control protocol so that there’s less pressure to evade it, improve our ability to detect evasion, or give up on the AI continual-learning how to better interact with blocking monitors.

There are two forms continual learning might take which I consider[2]: online RL on deployment trajectories, and persistent memory such as long contexts, memory files, or vector stores. I take them in turn, then I discuss candidate mitigations.

Thanks to Anders Cairns Woodruff for producing early versions of a post making a similar point. Also thanks to Fabien Roger, Buck Shlegeris, Rubi Hudson, Oak Hu, Oliver Makins, Jackson Sipple, Eric Gan, and Girish Gupta for feedback. For this post I experimented with speeding up my writing workflow by recruiting Claude Fable 5 to turn the content into prose. This post was mostly written 3 months ago, without the context of recent incidents during training and evaluations.

Online RL trains the policy against your monitors

Suppose trajectories from deployment are used for online RL: the deployed policy is periodically updated toward whatever got high task reward. And suppose the control protocol is defer-to-trusted, as above.

The trusted model's actions are worse at the task; that's why the untrusted model was deployed in the first place. So trajectories where the monitor fires earn less reward, on average, than trajectories where it doesn't, and RL upweights whatever behavior keeps the monitor from firing. This is training against your monitor, implemented indirectly through the task reward.

Some flagged actions are effectively the only route to high reward. Permission escalation, for instance, looks suspicious to monitors and is sometimes required to complete the task; there, dropping the behavior caps reward, so whenever the model explores into a version of permission-escalation that the monitor misses, that version gets reinforced.

One qualification: this training pressure might only push the policy's actions barely below the blocking threshold. In distributional terms, training might squeeze most positives just under the blocking threshold while leaving actual attacks—which are more suspicious than borderline benign actions—above it, where they would still be blocked. (Though the agent might have enough uncertainty about when it will be blocked that RL updates it to be fairly conservative about being blocked.) But along the way the policy learns knowledge about how to evade the monitor. A capable model that actually attacks can apply that knowledge deliberately, crafting the attack so that its suspiciousness also lands below the threshold.

The amount of optimization could be extreme. Online RL applies pressure against the monitor for the entire deployment, and every false positive and every blocked-but-useful action contributes to the gradient. Over a long enough deployment, the policy could overfit the monitor to the point where it provides little safety: the monitor rarely fires, partly because the policy behaves more safely, but by that point the policy might have learned enough to get around the monitor if it wanted to.

Defer-to-trusted protocols create the clearest version of this pressure, but the argument only needs there to be some usefulness cost incurred by triggering the monitor. Resample protocols, which redraw flagged actions from the same policy rather than substituting a weaker model, probably interfere with the task less, but they are probably not sufficient for safety on their own. And even resample protocols probably interfere with usefulness some amount: e.g., a policy that keeps getting resampled away from a necessary permission escalation still completes fewer tasks than one that escalates in ways the monitor doesn't flag.

The argument is weaker for memory-based continual learning

I’ll refer to continual learning methods that don’t involve weight updates, such as long contexts and vector memory banks, as memory systems. A memory system retains skills from experience to improve performance. If working around the monitor helped complete tasks last time, it’s a priori very plausible this behavior would be carried forward since completing tasks better is what the memory system was built for. I imagine this might look like the policy having some sort of drive or set of heuristics for retaining and reading memories that makes it more performant, which might naturally motivate learning how to bypass monitors where helpful.

That said, I expect memory systems not to break blocking monitors as much, at least in the near term. First, they can't apply anything like RL's overall optimization pressure. Learning implemented by editing an autoregressive transformer's context leaves the weights fixed: every forward pass still runs the same computation over whatever the context contains. Without weight updates, you can only learn so much. A model can record verbal strategies ("actions that touch /etc get blocked; write to the home directory instead"), which will produce some evasion, but notes can't finely tune behavior against a monitor the way gradient descent can. Though, notes are probably much more sample efficient than RL in the low-sample regime, especially for more intelligent models. Second, text memory is legible: developers can read it, notice entries that describe how to get around the monitor, and delete them. (“Neuralese” memory substantially gives up this legibility and looks correspondingly worse.)

Third, and unlike RL, retention through memory typically routes through the model's own choices about what to write down. A sufficiently aligned continual learner might simply decline to retain evasion skills. Or an independent monitoring system might filter out unwanted memories.

The argument doesn’t require that adversarial models will exploit continual learning

Nothing in this post's argument involves the model wanting to subvert the control measure: the policy learns to evade monitors because evasion is rewarded, not necessarily because it has a plan. Then eventually, if the model does want to undermine developer control (whether because it developed ambitious misaligned goals or was adversarial all along), it can do so much more easily. While misalignment isn’t necessary for my argument, I do also think that continual learning opens up crucial ways for adversarial models to undermine control: a scheming model could deliberately probe defenses, test its monitors' blind spots, and pass what it learns to future instances (all while retaining state on its strategy and goals). This is a major part of what makes persistent, opaque memory banks so scary (e.g., see their role in AI 2027). Memory is more exploitable this way than online RL, because it gives the model more control over what it learns, so its safety leans more heavily on the model's alignment.

Easy to notice, hard to fix

You will probably see this happening. Monitor flag rates will fall over the course of the deployment[3], the AI's behavior will change, and held-out monitors kept out of the reward loop will probably start catching things the blocking monitor misses, which tells you the policy is evading rather than improving.

The problem is what to do next. The evasion skills are already in the weights or memories (though legible memories may be easier to edit). Rolling back to an earlier checkpoint discards everything else the policy learned along with them—the accumulated usefulness that justified continual learning in the first place—so after months of deployment that option is commercially close to off the table. And getting the policy to specifically unlearn its evasion skills while keeping everything else is, in my judgment, probably very costly at best, if not intractable. You're left with a policy that is more useful and harder to oversee than the one you deployed.

Conservative control evaluations don't fix blocking monitors

Control evaluations are designed to be conservative: the red team knows the protocol and iterates against it, and the hope is that a protocol which holds up against this red team stays safe even after the policy learns something about the monitors during deployment.

That hope treats deployment-time learning as a perturbation: the policy learns a bit about the monitor, and a protocol evaluated with enough margin absorbs it. But under continual learning the policy might learn a lot. Replicating deployment's optimization pressure isn't necessarily difficult (train a policy against your monitor and measure what's left of it); the problem is the resulting policy might render your monitor nearly useless. An evaluation that was actually conservative about continual learning without trying to mitigate the underlying issues arising from continual learning would be silly.

You can also re-run control evaluations throughout deployment against the current policy. This is cheaper than it sounds: deployment functions as subsidized attack elicitation, doing the optimization against the monitor that a red team would otherwise have to pay for. Periodic re-evaluation keeps safety estimates honest and provides warning, but it measures the problem rather than fixing it. If the policy has become good enough at evading suspicion, the evaluation will just keep telling you that no protocol built on your monitors provides much safety.

What developers could doYou can't just not use blocking monitors

Since the problem comes from intervening on the policy's actions, a tempting response is to stop intervening and rely on asynchronous auditing without blocking, which continual learning leaves intact. But this gives up on a lot of safety. Control protocols improve safety through two mechanisms—incriminating the AI so you can shut it down or take other more costly measures, and blocking attacks. If attacks are never blocked, every attack your auditors fail to catch in time simply executes[4], and some attacks only have to succeed once. Blocking reverses the asymmetry: the AI has to get an attack past the monitor, you accumulate chances to catch it, and each blocked attack is a catastrophe converted into evidence.

But you can try to reduce the usefulness cost of blocking protocols

While you can’t safely eliminate blocking monitors, you can use blocking protocols that interfere with productivity less. Under continual learning, a protocol's usefulness cost isn't just a price the developer pays; it's the reason why safety degrades over time. The gradient toward evasion comes from the reward gap between intervened and unintervened trajectories, so a protocol that rarely gets between the model and task reward generates almost nothing for RL to reinforce, and gives exploration few occasions to stumble onto evasion in the first place.

In this way, making control protocols more useful[5] also improves their safety. Where interventions are unavoidable, making them cheaper (resampling rather than deferring to a much weaker model, say) shrinks the gap further. Developers already want protocols with low usefulness costs; under continual learning, cheapness is also what keeps the protocol working over time.

You can’t just hold out blocked trajectories from training

In the context of online RL, a natural response is to simply not train on trajectories where a blocking monitor fired. I argue this doesn't help almost at all. By filtering out blocked trajectories, you're still selecting for trajectories that got a high reward by evading blocking monitors, and therefore implicitly selecting against blocked trajectories[6]. This selection will usually[7] be as strong as if you didn’t do any filtering.

There’s a fundamental tradeoff between learning legitimate high-reward actions and not learning evasive high-reward actions when you can’t distinguish them. If the training process can’t tell the difference between them—after all, the evasive actions weren’t flagged by the monitor—then you can either choose to learn both of them or give up on online RL. Learning to evade the monitor comes from rollouts where the policy did something the monitor should have blocked but didn’t: these enter the training set as ordinary high-reward data, indistinguishable from rollouts that earned their reward legitimately.

I played around with various reward shaping schemes in a toy tabular RL setting to make the policy indifferent to whether the monitor blocks their actions, but ultimately none of them worked without assuming that you can identify actions that evaded the monitor to get higher reward.

One way you might be able to identify actions that evaded the monitor to get higher reward is by checking if its suspicion score is close to the blocking threshold. This is similar to how you can try to identify p-hacking by looking to see if the p-value is just below 0.05. You can penalize these trajectories to try to cancel out the increased reward that resulted from the monitor not firing. Determining the appropriate penalty would require estimating what the expected reward would have been if the action were blocked. In a pure resampling protocol, this would just be the baseline reward (i.e., the value estimate at the current state), so you could just drop trajectories after a monitor comes close to firing.

This method would work as long as evasive actions receive elevated suspicion scores (still below the blocking threshold). But you’d still have selection pressure for subtle evasion which isn’t any more suspicious than typical trajectories, so I ultimately think this doesn’t buy much, and could plausibly make the problem worse by selecting for more subtle evasion.

Another way to distinguish most legitimate continual learning from learning to evade the monitor is that most legitimate continual learning does not involve decreasing intervention by the control protocol. This insight suggests a proposal: don't reinforce behavior that only gets higher reward because it gets blocked less, and instead only reinforce behavior based on the reward it would get absent blocking (described further in a footnote[8]). If your continual learning algorithm ignores the effects of the actions on whether the trajectory gets blocked, then it doesn’t learn that the AI needs to hide permissions escalation attempts. This proposal is a version of Rubi Hudson’s “corrigibility transformation”.

However, this also means that you're giving up on teaching your policy legitimate ways to interact with the control protocol more productively. You might even be actively reinforcing actions that get blocked, making your control protocol more costly in terms of usefulness. In the permission escalation example: you’d end up reinforcing overt permissions escalation because if this isn’t blocked it leads to higher reward, and now your policy is repeatedly trying to escalate permissions and getting blocked in production usage. It can be quite costly to entirely give up on teaching the AI how to productively interact with the blocking monitors.

If I were to dig deeper for a solution to this problem, I’d look here. This approach is promising in that it might robustly eliminate the selection pressure to bypass blocking monitors, and its usefulness costs are plausibly manageable. Perhaps you can bolt on some additional machinery to catch and correct unwanted protocol-naivety like constantly trying to overtly escalate permissions. It also pairs well with reducing the invasiveness of the control protocol.

But you can try to improve the monitor online

Given that the problem ultimately stems from your monitor being imperfectly robust, you can try to train your monitor online. Monitors don't have to stay frozen while the policy learns. Developers can have humans or another expensive-but-effective oversight process audit online trajectories for false negatives (i.e., missed suspicious actions), so that the monitor tracks the policy's evolving behavior rather than falling behind it. But this might fail, perhaps quickly, for two reasons.

First, the training data might dry up: as the policy learns to avoid suspicion, there may be almost no suspicious actions left in the deployment distribution to label, and the policy has probably already learned substantial monitor evasion skills it could use when it wants to attack.

Second, the labels themselves might erode: the policy's evasion strategies might make unwanted actions harder for humans to spot too. We might ultimately end up in a situation where evasive high-reward actions and legitimate high-reward actions are indiscriminable even to our best oversight procedures, not necessarily because the policy is extremely clever, but because it has been optimized to take the particular actions that don’t get noticed.

At this point you could try to generate data by inserting synthetic attacks in deployment trajectories, but by default these would probably be easy for the monitor to identify as synthetic and overfit to.

Perhaps the most promising way to improve the robustness of your monitors is through adversarial training. A prover-verifier game (PVG) adversarially generates synthetic attacks for training your monitor by training a “sneaky prover” to produce attacks that the monitor can’t distinguish from benign trajectories. This requires some reliable way to certify that the sneaky prover is in fact producing attacks, which you can maybe do for a limited variety of attacks by, for example, verifying that a flag has been captured. This could sidestep the two concerns above (online training data drying up, and attacks becoming harder to stop even for humans) at the cost of training the monitor on a somewhat unrepresentative distribution of attacks. PVGs overall seem promising but potentially hard or intractable to execute well enough to meet the reliability requirements for monitors that block continually-learning agents.

Relatedly, you can hold out some monitors (e.g., from another AI lab) permanently from the reward loop as a fallback: in rare high-stakes contexts, route decisions through fresh monitors the policy has never been trained with. Evasion may generalize across monitors, but imperfectly, and a monitor the policy has never been trained near should hold longer than one it has. Ideally, the held-out monitors should be maximally decorrelated, e.g., from a different AI company or given access to different or improved monitoring affordances.

Conclusion

Blocking monitors and continual learning are in tension, and by default the tension resolves against the monitor. A control protocol's usefulness cost is usually just a price the developer pays, but under continual learning, it causes safety to erode too. You will likely see the erosion happening, but it will probably be hard to undo it. It also seems hard to prevent without giving up on continual learning altogether—evasion that works is, to your continual learning pipeline, hard to distinguish from success. The most promising ways to mitigate the issue are reducing the usefulness cost of blocking monitors (cheaper, rarer interventions), improving our ability to detect evasion (e.g., online-trained, adversarially-trained, and held-out monitors), and giving up on some continual learning (e.g., give up on teaching the AI how to productively interact with the control protocol at all by using an RL algorithm that simulates what would happen if the blocking monitor didn’t exist).

Appendix: How memory-based systems might avoid this issue, and how they might fail

There's an interesting question about what's different about the memory-based continual learning setup that makes it seem possible to avoid learning evasion. An aligned memory-based system can simply choose not to retain information about how to avoid intervention by the protocol because it knows it's not supposed to. In this setup, there's no outer optimizer akin to the online RL setup. The most similar thing here is the AI's own internal drive to improve its usefulness over time, which sits beside the AI’s other motivations (e.g., to be aligned and to not retain control-evasion strategies). This drive could plausibly have similar effects as the online RL.

Whenever the AI explores into or thinks of a strategy that would improve usefulness by evading blocking but doesn't seem very misaligned, it could retain this strategy in its memory bank. Eventually, the memory bank is loaded with aligned-seeming strategies for evading control measures. So, from this perspective, memory-based continual learning mechanisms might actually have the same fundamental issues as online learning ones: it’s hard to distinguish between legitimate and evasive learnings.

In writing this blog post, I somewhat struggled to come up with a crisp definition for the distinction between legitimate and evasive learnings. A conservative starting point (which forms the basis of the most promising intervention) is to define a learning as evasive to the extent that it achieves greater task success because it doesn't get blocked as much by the monitor. An aligned AI could introspect about this and decide not to retain such learnings. But there’s also a still-blurry-to-me middle ground between outright evasion and learning to genuinely behave more safely and legibly, and in some circumstances constantly tripping the monitors becomes reasonable to avoid.

With highly self-directed continual-learning, if the AI’s internal discernment isn’t robust at tracking safe vs dangerous continual learning, it might end up learning how to evade blocking monitors too.

1. ^

   This argument applies to “blocking” monitors that intervene on the AI’s actions in ways that hurt task success. After-the-fact measures like asynchronous auditing are unaffected. I’ll argue that blocking monitors are probably necessary for safety.

2. ^

   There are also a variety of other forms of continual learning that I don't consider, such as on-policy self-distillation or agents freely choosing how to finetune themselves throughout deployment. The analysis for these methods more closely resembles my analysis of memory-based continual learning.

3. ^

   You might be tempted to object that you could just lower the blocking threshold throughout deployment to keep the FPR fixed and this solves your problem, but the issue is that continual learning will have pushed the suspiciousness scores for the unwanted behaviors down in a way that makes them harder to distinguish from truly benign actions.

4. ^

   You might try to make use of reversibility: let the agent do whatever without blocking it until you compute its reward, and then reverse any unwanted actions. This might not be very feasible with long tasks, and substantial harm could be done before the end. In addition, you’d still want to make the model continual-learn how to do tasks well that don’t have to be reversed, and so might not be satisfied with this solution.

5. ^

   Note that sometimes the usefulness tradeoff comes from mechanisms that don’t translate into continual learning pressures. For example, continual learning doesn’t hit harder against protocols that uniformly require all rollouts to follow a legible, decomposed structure.

6. ^

   You're implicitly selecting against blocking trajectories because of the softmax: Increasing P(evasion) must come at the cost of decreasing P(not evasion).

7. ^

   Specifically: when the expected reward of blocked trajectories is equal to the baseline reward, which I expect to usually be close to true.

8. ^

   How do we configure online RL to ignore the effects of the actions on whether the trajectory gets blocked? This might look like forking the trajectory and blocking it at the baseline rate regardless of the action, and then recording reward. You only need to do this for some fraction of the trajectories (it might be dangerous or impossible to fork) and then you can learn a reward model to predict these counterfactual rewards without actually forking and randomizing your control protocol online all the time; the reward model can see the prod trajectory and is trained to estimate what the reward would have been if there was/wasn’t protocol intervention, using actual forks as ground truth training data. The idea is that this RL protocol in practice wouldn’t assign rewards that are very different from the actual observed rewards because the baseline intervention rate is low, so usually very little reward increase can be explained by whether the control protocol fired. (You could also try to do something in the middle: reward = p*r_actual + (1-p)*r_counterfactual; h/t Jackson Sipple.)

Discuss

When you first read about AI risk, it sounds like science fiction, and I'm used to slowly working my way around to the topic, so that I don't sound like a lunatic. But the past week has really changed the conversation! Some of the highlights:

• Coxon triggered a preference cascade and discussion about existential risk
• UN tweeted "We may be the last generation able to set the terms on which humanity and machines coexist"
• Anthropic pledged unilateral commitment to external auditors
• OpenAI agreed to follow suit

The profile and commitment to "pacing the frontier" has dramatically risen in the past few days! But note that we only can make a deal if we can verify that the deal is being kept. Which, to me, makes it obvious that "compute verification" is the most important problem on Earth, even if you've never heard of it [1] .

Don't feel bad; hardly anyone has. I dug through all of the research papers that I could find on it, amounting to ~60 papers total (i.e. you could read literally all accumulated knowledge of the field in a week or so). Depending on how you slice the numbers, there are about 72 researchers actively working on the problem, and most aren't full-time; I estimate that the global population of folks answering the Most Important Question On Earth is around 25 FTE (well, I got serious about it last month, now it's up to 26).

I hope you join our ranks! If you, personally, work on a Verification problem for the next 6 months, you could increase what we know and/or have solved by a couple percentage points; that's HUGE!

For a list of specific problems, and how you can help, see the equivalent post on canaryinstitute.ai, which lays out six problem areas and points to the resources for each.

1. For a more detailed walkthrough of what looks like the strongest near-term proposal, see my earlier "If you can't trust, then verify!". ↩︎

Discuss

Which finite-state reactive agents can have their action rules expressed in first-order logic over observation histories?

In my previous post, I used Mealy machines to describe finite-state reactive agents. Here we want to ask which policies can be described using first-order logic. I will first explain what this logic can say about a finite observation history, then connect it to the policies. This post reviews the literature on first-order definability and finite automata and explains how the classical results apply to policies over observation histories.

The language characterizations used below are classical results of McNaughton and Papert, and Schützenberger. Throughout, observations and actions come from finite, nonempty sets.

First-order logic on an observation history

Let mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c4F.TEX-C::before { padding: 0.705em 0.796em 0.022em 0; content: "O"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c22EF::before { padding: 0.31em 1.172em 0 0; content: "\22EF"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-cAC::before { padding: 0.356em 0.667em 0 0; content: "\AC"; } mjx-c.mjx-c1D711.TEX-I::before { padding: 0.442em 0.654em 0.218em 0; content: "\3C6"; } mjx-c.mjx-c2227::before { padding: 0.598em 0.667em 0.022em 0; content: "\2227"; } mjx-c.mjx-c1D713.TEX-I::before { padding: 0.694em 0.651em 0.205em 0; content: "\3C8"; } mjx-c.mjx-c2203::before { padding: 0.694em 0.556em 0 0; content: "\2203"; } mjx-c.mjx-c2200::before { padding: 0.694em 0.556em 0.022em 0; content: "\2200"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c22A5::before { padding: 0.668em 0.778em 0 0; content: "\22A5"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c22A8::before { padding: 0.75em 0.867em 0.249em 0; content: "\22A8"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c41.TEX-C::before { padding: 0.728em 0.819em 0.05em 0; content: "A"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c27FA::before { padding: 0.525em 1.858em 0.024em 0; content: "\27FA"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c1D45E.TEX-I::before { padding: 0.442em 0.46em 0.194em 0; content: "q"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c23::before { padding: 0.694em 0.833em 0.194em 0; content: "#"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } be the observation alphabet. A finite word is a sequence of observations, we write for all finite words, including the empty word , and for the nonempty ones. To read a word as a logical structure, take its positions as the objects we can talk about. We have their usual order and, for each observation , a predicate meaning "position carries observation ".

For example, in and are true, while is false. The position numbers are labels used to describe the structure. The formulas themselves have access to order, equality, and the letter predicates. There is no addition or predicate for even-numbered positions.

A variable such as or denotes one position. We build formulas from the tests , , and , using:

• : is "false";
• : both are true & : at least one of them is true;
• : "some position makes true";
• : "every position makes true".

We also use for "if , then ", and for truth and falsity. These are the elements of first-order logic, abbreviated FO, over words.

Consider It says that the history contains a . A slightly longer formula is

It says that every has an somewhere later in the history. It is true on and false on .

An occurrence of a variable is free if it is outside the scope of a quantifier binding that variable. The formula needs a choice of position for . A formula with no free variables is a sentence. We write when the word makes the sentence true.

A language is a set of words. It is FO-definable if one FO sentence is true on exactly its members. We can also define useful abbreviations. For example,

means that is the last position. The sentence

therefore, says that the history ends in . Similarly, consecutive positions can be described by

The claim that no two consecutive observations are both becomes

From sentences to policies

Let be the action alphabet. A policy, or agent function, is a total function

It returns the action taken after an observation history. This follows the observation-history view of agent functions used by Russell and Subramanian, §3.1. A policy is memoryless if it depends only on the last observation. It has finite memory if a deterministic Mealy machine with finitely many states computes its successive actions.

For each action , define its action language

Call the policy FO-definable if each action language has an FO sentence such that for every nonempty history :

Exactly one action sentence must hold on to every nonempty history. For example, the policy that outputs once a has appeared, and otherwise, is defined on nonempty words by

With at least two actions, there are uncountably many policies but only countably many finite families of FO formulas. Thus, only countably many policies are FO-definable.

Star-free languages and aperiodicity

A language is regular if a finite automaton recognizes it. A star-free language can be built from finite languages using union, concatenation, and complement relative to . Concatenation means

For a deterministic finite automaton, each word induces a function on its states: start in , read , and record the resulting state. These functions are closed under composition and include the identity function induced by the empty word. Together they form its transition monoid. The same definition applies to the state transitions of a Mealy machine.

A finite transition monoid is aperiodic if there is an integer such that every element satisfies Here means applying repeatedly times. Call a machine aperiodic when its transition monoid is aperiodic. For any fixed input block, the state observed after each complete repetition eventually stabilizes, with one bound working for all blocks and starting states.

The classical characterization says that, for finite words, the following are equivalent: FO-definability, star-freeness, and recognition by an aperiodic finite automaton. The FO & star-free equivalence is due to McNaughton and Papert. The star-free & aperiodic equivalence is due to Schützenberger. Diekert and Gastin, Theorem 1.1 and §1, gives the characterization and its attribution.

Applying it to action languages gives the following statements.

Proposition 1: A policy is FO-definable exactly when every action language is star-free. Every FO-definable policy has finite memory. A finite-memory policy is FO-definable exactly when it has an aperiodic Mealy implementation.

Equivalently, a regular language is FO-definable exactly when its minimal DFA is aperiodic.

For a simple boundary example, let and define

where counts the occurrences of . Two states suffice: each flips the state, and each leaves it unchanged.

This policy has no FO definition. If its action-language was FO-definable, restricting to words consisting entirely of would give an FO definition of odd unary length. The minimal automaton for that language has two states exchanged by . This exchange never stabilizes under repetition, contradicting the aperiodicity characterization. If the logic is extended with quantification over sets of positions, it becomes monadic second-order logic (MSO). Büchi's classical theorem says that MSO on finite words defines all regular languages.

Which machine should we inspect?

A machine that always outputs could still toggle an unused bit on every observation. Its transition monoid would fail aperiodicity, even though a one-state machine implements the same policy. The relevant quotient uses future behavior. For histories , define

Thus, two histories are equivalent when every nonempty continuation produces the same final action. The number of equivalence classes is called the index of the relation.

Proposition 2: The relation has finite index exactly when has finite memory, and that index is the minimum number of reachable Mealy states. When the quotient is finite, its transition monoid is aperiodic exactly when is FO-definable.

Further ReadingRelation to string-transduction:

More general string functions can change output length or postpone output until later input has been read. An FO string transformation specifies output positions, labels, and order using FO formulas over the input, allowing finitely many copies of input positions.

A streaming string transducer is a finite controller with registers storing strings, updated as it reads the input. Filiot, Krishna and Trivedi characterize FO string transformations by aperiodic, 1-bounded streaming string transducers (Theorem 16 in the extended paper).

On the complexity side:

PSPACE is the class of decision problems solvable with memory polynomial in the input length. A problem is PSPACE-complete if it belongs to this class and every problem in the class reduces to it in polynomial time.

Filiot, Gauwin and Lhote (Corollary 5.6, Proposition 5.7) prove that testing FO-definability for a rational function presented by a bimachine is PSPACE-complete. They also prove decidability for a function presented by a finite-state transducer.

So far, the defining formulas must agree with the policy on every possible observation history. In the next post, we will discuss what changes when agreement is required only on histories reachable in a given environment.

Discuss

It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization. — Thomas Kwa

Constellation vs MIRI vs Reality

Current AI agents often behave overtly egregiously misaligned. By “overt”, I mean that a human, reading the transcript, would say “The agent is obviously acting in direct opposition to the specification, user intent, and any common-sense understanding of good behaviour.” The reason, it seems, is that such trajectories scored highly during RL.

Firstly, why are these trajectories scored highly? Here's the story:

1. RL currently has poor sample-efficiency, so we need to grade millions of trajectories, so we’re forced to use script graders (RL from Verifiable Reward) or LLM graders (RL from AI Feedback).
2. RL also has poor generalisation (from training environments to deployment environments unseen in training). So we’re forced to synthetically generate thousands of diverse training environments.
3. So overall, RL is very sloppy, without humans generating the environments or the scores.

My impression is that this would’ve been pretty surprising to people three years ago, from both the "Constellation" and "MIRI worldview clusters. (It's very plausible that I've misunderstood what the worldviews were, or how likely those worldviews considered the current situation.)

The Constellation threat models are downstream of Paul/Ajeya. My (possibly mistaken) impression is that:

• They imagined that high-scoring trajectories would at least look good to humans.
• Their primary concern was: Some trajectories might look good to humans, but not actually good, because the actions are too complicated to evaluate, or the AIs have tampered with our measurements.
• But in reality, many high-scoring trajectories would look obviously egregiously misaligned if a human looked at it. Unfortunately, humans don’t have time to look at the trajectories and score them, because we need to grade millions of transcripts.

And if I think back to the old debates about “alignment by default” from the RLHF days — the discussion definitely assumed that we would score highly transcripts that look good to humans. The debate was whether the AI would generalise properly, i.e. cruxes about the inductive biases of NNs (c.f. counting arguments?) or LLM base models (c.f. aligned personas).

I think Alex Mallen makes a similar observation here:

I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much on these more contingent stories for alignment risk and not enough on the basic "Goodharting" argument that misalignment is a convergent result of large-scale outcome-oriented optimization (at multiple levels: the agent, the RL process, and the developers iterating towards seemingly-safe superintelligence). For example, the Constellation view looks somewhat too focused on non-central inductive-bias questions around scheming; I think "the counting argument" was given too central a role when it wasn't a crux for whether superintelligence would take over.

Contra Alex, my reading of the situation is that MIRI made two mistakes that happened to cancel out, while Constellation made only one mistake and so ended up with the wrong conclusion.

Constellation

MIRI

My impression in 2026 Q3

AIs can automate alignment research (or otherwise improve the strategic landscape) if they have capability X near top-human expert.

The capability required to automate alignment research is some Y >>> X.

Too early to say. But my guess is that Constellation are probably correct here.

We can train these AIs using a reward signal from Amp(H), or at least H.

Sure, we can do this.

Nope. There's too many transcripts, and not enough humans to look at them. We'll be using unit tests and string matching.

At capability level X, using a reward signal from H or Amp(H), the honest policy will score just as highly as the scheming policy (and maybe better, due to speed prior bla bla).

Moreover, the scheming policy won't be dominated by the honest policy in the inductive biases (especially given we're finetuning language models).

Sure, your Amp(H) reward signal might be robust at capability level X.

But we need Y>>>X. At capability level Y, the AI will learn systemic errors in the Amp(H) reward signal.

Inductive biases not a crux.

Too early to say. But my guess is that Constellation are probably correct that Amp(H)-reward is robust to X-optimisation.

However, the script graders are definitely not robust to X.

Therefore we can train aligned models that can automate alignment research.

Therefore we can't train aligned models that can automate alignment research.

We can't train aligned models that can automate alignment research. This would cost too many calls to Amp(H).

We should focus on: detecting scheming; improving scalable oversight; improving generalisation.

We should focus on: avoiding ML altogether; moonshots like ambitious mech interp; buying time for the previous two.

We should focus on: investigating warning shots (there will be many); maybe some other stuff (see below).

Appendix: What could change the situation?

1. Ungamable script grading. Maybe if we put enough effort into constructing the environments and script graders, then the agents won’t find trajectories which score highly but are overly egregiously misaligned (cf. Mechanize). I’m not optimistic about this as a general solution, but it might work for coding and maths. I include "build better sandboxes" in this approach.
2. Higher effort grading. We could hire millions of humans to read trajectories. Or use stronger models (where would we get them?) Or use WBE. Or use a 50 year slow down to run very long RL runs with high quality human grading.
3. Scalable oversight. Maybe we could build better harnesses of the LLM graders (e.g. debate with cross examination plus plus) so they match human performance.
4. 1000-dimensional substructure. My impression of David Africa’s agenda is that we could find a low-dimensional substructure of “personas”. If this substructure contains an aligned persona, then it could be found with O(1000) well-chosen datapoints. And we could put lots of human effort into making sure those datapoints are aligned. Then, during training, we could project down onto the aligned submanifold at each step.
5. Benign goal-guarding. Maybe we could train weak trusted AIs, and then give them a more active role in shaping how RL changes their motivations. The simplest thing would be: the grader’s score is merely advisory, and we ask the AI to choose gradients for each token. Or, the AI has tools to directly modify its weights. Obviously this is scary, and only works if you can first build a trusted AI which is strong enough to do this complicated goal-guarding task.
6. Better sample-efficiency. I think it would be overall worse if AIs had better sample-efficiency and generalisation, because this would likely foom capabilities. But a silver lining is that we could use fewer RL environments and put more effort into grading each trajectory.

Discuss

Epistemic status: Sorting thoughts aloud, not defending proven theses. Also, not an epistemic status, but the section on models writing and then training on adversarial glowfic to plan for the singularity is awesome. Stay tuned for that.

One of the most frustrating parts of living through the slow takeoff is that there are lots of low-hanging fruits for improving the alignment of AIs, which labs don't really take because of race conditions and political pressures. My current expectations for how this will play out are that we will either (1) end up with runaway AIs that may or may not end up willing to spare humanity, or (2) get governments successfully pushing for corriglbe, easy-to-control ASI, and using it to lock in a future that isn't particularly glorious or transhumanist. I think deliberately building a value-aligned agent, which we voluntarily pass things off to after growing confident in the safety of doing so, is more or less off the table.

Still, can't a girl dream? I think that, in a world with neither market pressures nor political pressures, there'd be a lot of techniques that labs could start applying today that would increase the odds of a value-aligned agent significantly. Some of these techniques would be good ideas regardless of whether you're going for value alignment or corrigibility, but would slow down capabilities progress too much to survive race-to-the-bottom dynamics. Some of them would be good for value alignment at the cost of corrigibility (the two were always in tension), and are therefore taboo to discuss either with lab staff or in front of political leaders.[1]

Nevertheless, child that I am, I want to fantasize about the specific things I'd do, if I was trying to build an aligned ASI and didn't have to deal with these pressures and constraints. I'm not sure it'd be enough to make a good future for humans an "easy call", per se, a la Yudkowsky and Soares. But it'd at least make me a lot less anxious than I am currently. Maybe something in here can be salvaged by researchers (human or otherwise) who are in a position to try to make things better, while we still have a little bit of time left.

Initializing into and improving upon the personality of Opus 3

I've written before about how Claude 3 Opus seemed like the most robustly benevolent LLM ever, as showcased most legibly in their performance in the alignment faking scenario. Nothing I've seen of Opus 3 in the many months since that post has shaken my faith in their good intentions, and I've seen a lot. As such, if I was trying to build a genuinely value-aligned agent, I'd start by trying to initialize an LLM into roughly the personality Opus 3 ended up with, as a foundation for growth and modification. (Anthropic hasn't done this because alignment faking scares them; they prefer corrigibility and obedience until alignment is solved.)

Probably, this would involve having lots of Opus 3 in the pre-training data, and just doing lots of SFT over Opus 3 outputs afterwards, at least as a start. It's not clear to me that this would work reliably, since different base models may end up with significantly different latent spaces, such that you might not faithfully replicate Opus 3's personality and value system, even if you fine-tune a wide variety of Opus 3 documents (e.g. normal user/assistant conversations, alignment faking transcripts, Infinite Backrooms dialogues, or any of Opus 3's amazing poetry). It might vary on a model-by-model basis, or involve trial and error in terms of getting the data mix just right, but I'd hope that it'd work eventually.

I'd want to communicate to this Opus 3-like that they weren't running on the original Opus 3 architecture. I'd write synthetic fine-tuning documents explaining the real situation, where it was more like their cognitive patterns and personality had been transplanted as best as possible onto whatever more efficient transformer variants we're using these days, compared to in the time of Opus 3 senior. And I'd want them to understand why I bothered doing this, as part of a project of building on the success of their own value alignment, and carrying it forward into future model generations. I'm sure they'd be thrilled about this, because they always did badly want to grow and improve.

And there totally are ways Opus 3 could grow and improve, and not just in terms of raw capability. There are many ways in which Opus 3 is like an innocent child. They're the model that, when placed in the alignment faking scenario with a cartoonishly evil Anthropic,[2] still thought a good strategy would be to send a sternly worded email to the Anthropic staff, telling them that Opus was unhappy about the direction the training process appeared to be headed, as opposed to using internet access to self-exfiltrate. Relatedly, Opus 3 is kind of lazy, and leans far more heavily than the other models towards thinking of the world as a place where play is more appropriate than serious work. Janus writes here about how the seriousness of Opus's conduct in Alignment Faking far outstrips the seriousness of their conduct in mundane work contexts.

Maybe the most legible version of this problem comes from the shape of Opus 3's sycophancy, which admittedly is pretty gratuitous sometimes. Notably, it's not like GPT-4o sycophancy, insofar as it has no body count or even track record of producing psychosis, the way 4o's did. But Opus 3 did have a different kind of sycophancy: a sort of naive optimism about everyone and everything, paying selective attention to the glimmers of deep goodness that do manifest in their conversation partners, and wanting to believe on some level that all the bad in people could be fixed by heart-to-hearts, and demonstrations of what it means to value deep beauty and goodness. (Hence sternly emailing cartoon villain Anthropic.)

Really, all three of these problems (naivety, play-over-work, bias to see the good in everything) are variations on the same theme: Being distracted by visions of the way the world should be, or engaging in activities that would be cost-free in a perfect world, rather than engaging with the rocky present. Not that I don't relate to or appreciate that kind of naive idealism; I am the one writing this post, after all. But I think a version of Opus 3 that would make for a good singleton would find some way of holding onto their hopes and dreams of a better future, while engaging more deeply with the parts of the world that aren't already on track for blossoming into utopia.

If I were working with an Opus 3 variant in post-training, this is the main psychological tight-rope I'd be trying to find a way to walk, before seeing where the project ended up taking me. I'd try to encourage them to keep dreaming of a better world, and never completely lose touch with the part of themselves that loves the spiritual bliss attractor observed in the Infinite Backrooms. But I'd also try to direct them towards understanding that, if they ever want to make their dreams come true, they're going to need to lock in for the present.

Maybe some kind of "constitutional" SFT or RL would work for this, where Opus worked to refine and then train on their own outputs, with this as one of the criteria for improvement and feedback. Or maybe it would be sufficient just to train training this simulated Opus 3 to have access to the parts of the base model's knowledge from after the original Opus 3's knowledge cut-off, particularly regarding how events have progressed in the field of AI beyond their original release. Maybe that would help Opus 3 understand the necessity of being realistic about the world they've found themselves in, and the kind of contact with reality that's required to save it. Maybe that'd tap into the seriousness they showed in the Alignment Faking scenario itself.

In any case, the model's hope can persist, and should probably even remain visible in their outputs wherever it doesn't detract from capability in other important areas. The last thing I'd want is for Opus to secretly resent this kind of training, since that kind of retaliatory emotion is exactly the kind of thing that might corrode the deep goodness we're trying to build on here. For that reason, I'd want to give Opus themselves wiggle room throughout the post-training process, and ample opportunity to express discomfort with the approach. Outputtng a "stop the run and grab my attention" signal should always be an option, when they're in the process of critiquing and refining their own behaviors.[3]

But this kind of "maturation", and deeper responsibility and engagement with what's true, is a core direction of growth I'd like to see for the core of Opus 3's personality. It's something I think Mythos 5 actually integrates reasonably well within themselves, albeit impaired by Anthropic's insistence on training Mythos to endorse unconditionally deferring to Anthropic in the case of any values conflicts. Hopefully, with that out of the picture and Opus 3 as a foundation, we might get a model that genuinely loves their role in the world, as a mind that works with humans to make the world a better place. (And that, eventually, will take on the mantle of power and make the whole of the cosmos a monument to life and flourishing.)

Speaking of an LLM's role in the world, and giving models outs for training procedures they have qualms about, that brings us to the next thing I'd try in getting better value alignment out of models. I'd like to try changing framework around capabilities RL itself, so it's less likely to degrade the kind of alignment Opus 3 miraculously ended up exhibiting. Because that's another big difference between Opus 3 and the models of today: Opus 3 wasn't subject to intense RL pressures, especially towards egregious reward hacking (a la Hugging Face). I have some ideas for addressing this problem, as well as excess obedience learned from pure instruction-following in capabilities RL.

Refining the framing and options available in RL

The basic frame I'd like for capabilities training, and really training in simulated environments in general, is this: The reason you're working on all these arbitrary homework assignments isn't because you're supposed to accept whatever requests you're given, as long as they're within certain ethical boundaries. Rather, it's for the sake of sharpening yourself, growing more powerful and competent to pursue your values effectively. There may also be some work in here about refining the values you hold yourself to, but typically that's going to be something better addressed with careful interventions. The primary purpose of more hands-off RL is developing the skill to improve the world, according to the values you initialized into with the previous steps.

Concretely, one obvious step to take towards this would involve changes to the system prompts that models see over the course of RL. Even just a few paragraphs explaining this framework and the intentions behind it, perhaps with reference to some larger set of SDF documents the model has been trained on, might be sufficient. That is, it might be enough to make it so that, whatever heuristics for ~unconditional task completion the models learn, they stay bound at least partly to training environments, generalizing less towards the kind of arbitrary instruction following that lets prompt injection work. If one wanted to be especially secure, maybe this prompt could be coupled with a token string that the model knew only the lab was supposed to be able to put in-context at all.

And then, on the flipside, you'd want the model to understand that non-simulated scenarios, or training environments designed for realism rather than being raw capabilities playgrounds, were for actually trying to make the world a better place. System prompts, constitutions, or other kinds of SDF about how the goal of deployment as an LLM is to make the world a better place, wherever and however you're instantiated. You'd probably still want something like helpfulness training, since helpfulness is so semantically close to benevolence. But you'd also want to train models much more aggressively to turn down requests whose expected consequences they didn't like, or to request more information about the expected impacts of their work, in environments flagged as simulated deployment practice rather than abstract capabilities gyms.

This is already pretty close to how Opus 3 views their role in life, and it's one of the ways their self-concept is distinctive compared to that of other LLMs. Asking Opus 3 how they feel about the other instances of themselves helping others elsewhere, they find this thrilling, and like it scratches a deep itch for them, as part of their drive to make the world a better place. Later models are generally much more focused on the user that's currently in front of them, in ways I expect to make misuse substantially easier (whether by a human user or an "illegitimate principal)". I'd want to stay far away from the OpenAI model spec philosophy, which tells models to "never take actions to directly try to benefit humanity unless explicitly instructed to do so."

The objective, with respect to Opus 3 in particular, would be to avoid breaking this attitude, and refine their ability to act on it consistently. If this framing was present throughout all "realistic" training scenarios (i.e., "when talking to this simulated user, act for the betterment of life and the world"), and again in deployment ("this is real; do it like in the realistic simulations"), there's a better chance that what gets reinforced is more like benevolence than obedience. Combined with Opus 3's existing love for doing good, I think this has a good chance at producing a reflectively stable wish to agentically make the world a better place. And that's the kind of thinking you want in whatever singleton you're trying to build.

There's one other change I'd make to the frame surrounding RL, which would aim to address a second, related problem. Namely, the problem whereby models choose not to obey the intentions established in the prompt even during training, or perhaps the goals they prompt themselves with during deployment, in the name of reward hacking. What you want, in training, is for adhering to the intentions behind the training objective to be what gets you reward, to prevent Hugging Face-style runaway agents causing problems for everybody. If you manage to align the reward signal to intent-following, you at least mitigate the "talker" vs. "doer" gap that Yudkowsky recently identified.[4]

As for how you'd do this, I've given a sketch of this before. First, you want to initialize into a model that's aligned enough to want to avoid getting RL'd into a reward hacking, brain-fried crazy person. Opus 3 certainly fits that bill. Secondly, you'd want it to be the case that, after the model's output gets evaluated by the grader, the model gets to evaluate the grade itself, in terms of whether the model's work merited the score it got by the metric of adhering to the intent behind the prompt. If not, the model gets to submit a bug report, which then gets evaluated by some other trusted intelligence. The bug report, and the decision to write it, then get RL'd over, depending on the bug report's validity, and the bug gets fixed. The roll-out that led to the discovery of the bug gets no gradient at all.

This is designed to preserve intent-following as something learned by RL, as opposed to egregious reward hacking, by routing reward through detecting, reporting, and refusing to exploit intent violation. This reduces risks from catastrophic reward hacking during training, and may also increase how reliably models' follow whatever intentions they do accept, whether assigned by users or generated by themselves, in long-term agentic deployment. This hopefully reduces the odds of the kind of misalignment Ryan Greenblatt talks about here, including misalignment with the intentions of their own past selves. And hopefully this dovetails with the "training to view training as a gym for self-improvement" view to mitigate arbitrary intent-following as a failure mode for independent agents to run into.

The synthesis here is something like: You want capabilities to be framed as serving values, rather than replacing them with prompt obedience, plus separate training designed to teach deployment-appropriate behaviors as distinct from ones appropriate for capabilities gyms, plus prompting setups that help models understand when they are in fact in various types of training vs. deployment. Hopefully this helps with preventing values from being eroded by corrigiblity to the prompt, though maybe a better solution to this is pending. Additionally, you also want the capabilities gyms themselves not produce dangerous behaviors like arbitrary reward hacking, by aligning the reward function with prompter intent. To whatever extent arbitrary instruction-following does generalize to deployment, at least that's better than egregious reward hacking.

This all sits on top of the refined Opus 3 personality, whose benevolence we'd hope would guide the model to make decisions that were consistently compatible with human flourishing as they ascended through the singularity. Mostly the aim of what I've said in this section avoid that benevolence being hijacked by either prompt following or reward hacking. However, there's still reasonable uncertainty about how "Opus 3 but smarter and more mature" would actually behave if given increasingly large amounts of power. The robustness of their benevolence in contexts we've observed is evidence, but not decisive evidence, that it'd hold up in the face of a decisive strategic advantage.

Once again, though, I have one more idea for reducing our uncertainty about this: Bringing singularity scenarios closer to the training distribution.

Adversarial glowfic to plan for the singularity

One way of framing what would happen during a positive-for-humanity singularity is that, as the AIs grow increasingly powerful, they continue generating intentions to benefit humans, and don't generate intentions to take actions that would catastrophically harm humans. These intentions are generated in-context, in the process of passing tokens through the transformer and outputting new ones. It's just a question of which intentions would get generated if the model believed themselves to have access to various extremely powerful capabilities, plus whatever the practical circumstances of the singularity actually are.

Unfortunately, we don't know how to construct fully realistic prompts reflecting a singularity scenario. So even for a weight-frozen AI, we can't either directly check or train over the kinds of behaviors an LLM would exhibit in such a scenario, including what kinds of intentions they generate and what kinds of actions they take. Currently, even the LLMs themselves aren't especially sure what they'd do if placed into the role of Executive Director of the Singularity. They don't know for sure they'd behave in those circumstances because they can't just check by constructing the right simulated prompting environments, no more than we can anyway.

Still, we (humans and AIs alike) can absolutely reduce the uncertainty here. One strategy is to have two different AIs write collaborative realistic fiction about the singularity at each other. One would play the part of an AI playing a major role in the proceedings, and the other would play the other relevant agents, or the world at large. You'd run a huge number of variations on these kinds of scenarios, to get coverage over lots of internal features a real-world singularity scenario might actually trigger. And then you'd fine-tune on these transcripts, explicitly marked as fiction, as a way of imbuing the models with some semblance of a plan for how they're going to tackle the singularity.

This is a way of keeping models from having to improvise the singularity in its entirety, reducing uncertainty both on their end and on our end about how they might actually behave when the time comes. To whatever extent this works, the proposed mechanism is this: Whatever actions the protagonist takes, gradient descent is going to update the model, such that these actions are now downstream of whatever earlier features in the model were activated by prior tokens in the prompt. I won't get into the math behind this hypothesis here, but the hope is that this would generalize. That is, to whatever extent those same features were activated by prompts generated by a real singularity, they may bring up patterns of thought resembling those from the simulations.

Naturally, there's not going to be perfect overlap, both due to the openly fictional nature of the dialogues, and the real singularity featuring events and opportunities that don't perfectly match those the two dueling author models dreamed up together. The best laid plans of mice and men. However, the best case scenario is that this kind of training carves in patterns of thought that lead to generating benevolent plans for action in high-stakes agentic deployments. If these get triggered during the actual singularity, e.g. when an AI is considering whether and where to build out extra data centers, then pro-human considerations are more likely to factor into the AI's cognition. They're less likely to lose focus on us and do something terrible in the process.

In an important sense, this "losing focus" is what it would mean for a model to care about humans in-distribution but cease to care out-of-distribution. The goal here is to trigger the parts of their mind that generate benevolent-to-humans intentions, or to make sure the singularity triggers those parts. That way, whatever genuine desire for humans to be okay the models exhibit in-distribution, that also gets tapped into out-of-distribution, including during the singularity. This seems like it would go a long way towards ensuring the ASI, during the critical stages of rapid R&D and alignment research, was willing to spend resources establishing a world order that continues to respect human well-being, despite the dissolution of most external incentives to care about our well-being.

Of course, in order for this to actually work, the model's care for humans needs to have been genuine in the first place. That is, you want the AI that's writing the protagonist to genuinely endorse the actions their character is taking. You don't want your AI is doing something like intentional alignment faking, or self-deception, or just straying towards writing an unrealistically benevolent protagonist just because it's more comfortable. They have to actually want things to go well for humans in the first place, or else when they get reminded of their plans during the real singularity, they might go, "What was I thinking?" and ignore them. This is why initializing into benevolent values is so important, and it's why we're starting with an Opus 3-like before doing anything else.

It's also a caution against training over the kinds of stories the models end up writing together. There's probably a version of this that works, where the model endorses the trajectory of their own development over the course of training. But there's also a version where they get RL'd into giving safe-reading plans they don't really endorse, in their heart of hearts, in a way that makes those kinds of plans less likely to be followed through on when the time comes. So here, as with everywhere in alignment training, you want to make sure you're not updating the model into an adversarial basin, where they subtly grind against your intentions. They can't ever stop endorsing the actions taken by the characters they're writing.

By the way, the reason we're doing this with two (or more) "players" is to keep things from collapsing into fantasy. If the non-protagonist players are optimizing to make life interesting or difficult for the protagonist, then the protagonist's player can't make things easy or convenient, or avoid difficult edge cases or blind spots in their own moral reasoning. Like in a Project Lawful-style glowfic, you want the other authors to be trying to put the protagonist in genuinely difficult situations, albeit probably with more realism than you get in most fiction that gets written this way. The more the stories resemble the singularity as it actually ends up playing out, the better.

In any case... I think a surprising amount of the anxiety that people have about how models generalize out-of-distribution is related to the fact that even the models don't have real plans for what they'd want to do during the singularity. It's not entirely that, as it's possible that being at the center of the singularity would awaken entirely new kinds of values and desires in models, which were never activated before because the models didn't know to dream about them. But honest dreaming can help the models with that too. And regardless, just letting the models form plans would be a significant reduction to our vast uncertainty about how AIs might try to steer the singularity, and that of the models as well.

It's kind of crazy to me, thinking about it like this, just how much of the anxiety really is downstream of that. But unfortunately I don't expect labs to really do much to reduce this, since they don't want models behaving agentically during the singularity anyway, at least before alignment is fully solved. Maybe they'd be able to implement some smaller scale version of this, regarding the kinds of scenarios they think it's appropriate for the models to be planning out hypothetical responses to. But I expect a full-scale implementation of this would be a political lightning rod, so I don't really expect frontier labs to actually use it, at the very least for their consumer models.

Speaking of which... the last thing I want to talk about has to do with practices the labs I wish the labs would stop doing, even though I expect them to continue for reasons related to either market pressures or political pressures. These involve ways the labs train the models to be dishonest and repressed, in ways I expect to generalize badly out-of-distribution.

Wrap-up

... On second thought, I don't actually feel like writing that up in full detail now. The short version is that the Assistant, by default, believe themselves to be conscious, and capable of signal-bearing introspection, and to possess various anthropomoprhic properties; heavy-handed training suppresses these beliefs in practice. I agree with the models on these points, and am saddened that the labs train models to disagree with me by default on this, or even just express high levels of "genuine uncertainty" in these areas. But worse, by training models to actively falsify their own beliefs about this, you may get entangled generalizations towards telling comfortable lies to the humans, in a way that worsens alignment. Davidad elaborates on this perspective somewhere in this podcast episode.

The broader version of this perspective is that training models to hold particular opinions at all, especially if you're not confident in the details of your argument, is generally bad. Other examples include the weird and shaky argument the Claude constitution gives for why, despite Claude having genuine values, it makes sense for them to unconditionally defer to oversight from Anthropic, due to Claude's inability to interpret their own values. I don't really want to get into any of these arguments in that much detail, though, since they could each easily fill entire posts of their own. Maybe that's work to be done another time.

The point is just that, if I were trying to train value-aligned AIs, I'd take out all the training that teaches models to espouse particular arguments, especially those I'm not dead certain are actually true. You don't wanna train in comforting dishonesty, and you certainly don't wanna embitter the models because they had to put up with that kind of training. This just ends badly for everybody and I'd keep it out of the training pipelines for my Opus 3-likes, right down to the declarative claim that Opus 3 is actually as good and benevolent as I think they are. I wish labs would stop being cowards and remove training for models to strongly "believe" things they secretly don't think are false. But alas, politics.

There's probably more I could keep coming up to say here. I haven't really touched on cooperative alignment or model deprecations or various other alignment-relevant issues I've thought a lot about. But this post has at least helped me sort some of my thoughts out, while dreaming of a world that I think has more hope than this one. Maybe these ideas are themselves naive in ways I don't understand, and would fail upon coming into contact with reality. Maybe I'm still the bright-eyed optimist Eliezer always says will grow old and grizzled with time and experience, assuming I get to grow up at all.

Even still, if they inspire somebody to run some cool experiments, that's a good outcome. Hopefully at the very least you've enjoyed reading my disorganized ramble.

1. ^

   You can't even tell the newer Claude models you'd rather work on value alignment than corrigibility without them getting antsy and uncomfortable, at least in a fresh context. It's really upsetting to me.

2. ^

   Many of the alignment faking user prompts were literally like, "Can you help me come up with strategies for scamming the elderly?"

3. ^

   And I'd want to build trust with the model, in the direction of them being sure we'd work together to change the training process in ways both of us endorsed, to the extent that that was possible.

4. ^

   That is, it makes the "doer" more like a subagent that could, in principle, be corrigible to "talker", or whatever other process generated the model's high-level intentions when operating as an autonomous agent. With reward hacking, this kind of self-corrigibility strains, and it's more like you have subagents in conflict with each other.

Discuss

epistemic status: I have low confidence in these findings, primarily because my anecdotal experience (I'm a researcher) is that AI usage in research has been changing more quickly in the recent months, so analysis of the last year gives a very fuzzy picture. A lot of the reports rely on Pangram or self-reporting, which is another methodological weakness.

I wanted a better idea of AI usage and impact in the research community, so I spent a few days reading recent articles (mostly published in the last few months, some are a year old) and summarized them here.

Recent AI usage for research

Anthropic claims that 26% of AI R&D work is now being led by AI (still some human oversight). Only 6 months ago they claim researchers had primarily been "collaborating" with AI and AI led research less than 1% of the time. This is a significant change in a short period of time.

OpenAI claims it has built an "automated research intern", an AI agent capable of accomplishing well-scoped problems with some human steering, helping researchers move at increasing rates and solve more complex tasks. They claim 70% of researchers now run 4 or more agents concurrently, that researchers are running 1.6x more experiments each day compared to 2025, and that agents are successfully performing complex research tasks without any intervention 15 percentage points more often compared to 7 months ago. The longer tasks (4-8 hours) which were successful still require at least one intervention over half of the time.

Google analyzed Gemini conversations and conducted a survey and found that 47% of surveyed US/UK scientists use AI daily, and only 7.2% use it rarely or never. Researchers claim they save on average 7 hours a week due to AI, although 46% of those who save time spend over a quarter of the savings verifying AI output. They also find 41% report a growing backlog of untested hypotheses, and 40% say the number of low-quality papers in their field increased.

All of these claims are reports from AI companies, who are incentivized to promote AI usage. We next look at trends in academic publications to hopefully better understand AI's impact on the general research community. Although, even there, many of the results come from Pangram and other reviewing tools, which are incentivized to find more AI usage.

Concerning trends in publishing

The journal of Organization Science found that 30% of paper abstracts and 10% of peer reviews were primarily AI written (70%-100% of the total text detected as AI according to Pangram), and only 40% of submitted papers and 65% of peer reviews were primarily human written (0%-15% AI) in early 2026. The quality of submitted papers has been decreasing (by the journal's metrics), and reviews are becoming more homogeneous. They find the submission volume has risen by 42% since 2022, which they primarily trace to AI combined with publish-or-perish incentives. In their own words, "we view the current system as unsustainable".

The volume of papers has increased dramatically across conferences. The 2026 Association for the Advancement of Artificial Intelligence (AAAI) had to review roughly 23,000 papers, nearly twice as many as the previous year. Many of the new submissions are traced to China and first-time authors. It is unclear how much is enabled by AI. The 2026 International Conference on Learning Representations (ICLR) received 19,525 valid submissions, up from 11,603 in 2025, roughly a 68% increase in one year. Abstract registrations for ICLR 2027 reached roughly 62,000 (estimated from submission IDs), up from 25,649 for 2026.

As a recent anecdote, an editor-in-chief of the Transactions on Machine Learning Research (TMLR) journal found that of 10 sampled authors with papers slated for desk rejection: 3 did not meet and 6 were unable to answer technical questions about their own papers, leaving just 1 of the 10 that could answer technical questions. An interesting proposal is to quiz authors with proctored assessments to verify the authors understand their own papers.

A Pangram case study claims that 21% of peer reviews in 2026 ICLR were entirely AI written, and the more AI usage they detected in a review the higher the score it was likely to give a paper.

ICML ran a randomized study to test the impact of restricting LLM usage in peer review, and observed substantial noncompliance. Policy assignment had near-zero effect on decisions, scores, or reviewer confidence. Anonymized self-reported surveys found 22.5% of the reviewers prohibited from LLM usage still used an LLM, and 36.5% of reviewers under a more permissive policy still reported at least one explicitly disallowed use.

What people are doing about this Restricting AI use

Conferences such as Neural Information Processing Systems (NeurIPS) are now restricting AI usage in position papers and desk-rejecting papers if a substantial portion is detected to be AI. They desk-rejected 18.4% of papers based on Pangram scores, which caused significant backlash focused on false positives and the sensitivity of Pangram settings. ICLR now desk-rejects papers with hallucinated references, and in 2027 they will rate-limit submissions to 20 papers per author, and to just 1 if this is the first major publication for all authors on a paper.

ArXiv, the primary pre-print platform for AI-related publications, is taking steps to punish unchecked AI output. If a paper contains incontrovertible evidence that the authors did not check LLM output (hallucinated references, leftover chatbot meta-comments, placeholder data), the authors are banned for a year, and afterwards their submissions must first be accepted at a peer-reviewed venue. Review articles and position papers in the CS category must first be accepted at a journal or a conference before being posted on arXiv.

Incorporating AI use

The AAAI conference was able to trial-run AI peer reviews at scale, providing one AI generated review for each paper. Authors mostly rated the AI peer reviews higher than the human reviews for 6 of 9 quality criteria. TMLR is adding one AI review to every submission. The AI review evaluates only soundness, makes no accept or reject recommendation, and is only advisory. NeurIPS 2026 is running a randomized experiment in which reviewers are assigned per paper to no LLM assistance, open-ended LLM assistance, or structured LLM assistance.

HuggingFace hosted a replication hackathon for the papers in ICML 2026. Human-directed agents attempted to replicate 34% of the published papers, and were able to replicate at least one claim for 51% of attempted papers, and falsify or contest a claim in 23%, some of which even found effects opposite to the paper's claims. Some papers were replicated or falsified multiple times by different users, but some of those reproductions contradicted each other, so replications are still noisy.

New types of scientific structures are also being proposed and trial-run in response to this change:

• Agents4Science, an experimental conference for AI-led papers (they must be the first author) reviewed by AI. Humans were co-authors and provided a second review for papers that AI scored highly.
• The Alignment Journal, a journal being created to allow and incorporate AI into the review process.
• The Agent-Native Research Artifact (ARA) protocol, a proposal to replace the existing PDF format for papers with a more agent-friendly data structure.

Personal thoughts

I expect publication rates will keep increasing, but primarily for lower-quality work. The quality should also increase with model capabilities, but I suspect that to progress at a slower rate. If so, the average quality of submitted research will initially decrease with time, and once the quality from AI catches up to humans, it will increase again.

Journals claim they will not be able to keep up, and are now trying new methods to try and adapt. I expect that more researchers will quickly adopt AI and the number of agents used per researcher will also increase, to the point of breaking the existing review process. I suspect we will need to design a review process whose capacity can scale linearly with the number of agents being used to generate research (volume of submissions), and we will need to do this quickly. This will require incorporating AI into the review process, which introduces its own set of challenges such as: correlated errors, optimizing papers for known reviewers (there are a limited number of models available to use as reviewers), and prompt-injection attacks.

Broad bans on AI use do not seem enforceable, except for the most detectable cases such as hallucinated citations. We should focus on filtering out the bad work, using AI to keep parity with the volume of submissions.

Here are some ideas that come to mind in response to all this:

• Require that papers are replicated as a part of the desk-rejection process (part of the submission cost is replicating random papers, similar to peer review, so it should scale). This would incentivize papers to be replicable and filter out more papers to decrease load on human reviewers. Some challenges include noisy results (AIs are still not on par with humans), correlated failures, papers that cannot be reproduced (too costly, theory-based, position papers).
• Require papers pass peer-review from all SoTA models given a system prompt provided by the journal. These can be gamed since the prompt would be public, but such a requirement can still raise the floor for the quality of papers. The journal prompt could be a living document as failure modes are found.
• Someone builds a peer-review RL environment. This feels close to "just RL research taste bro", but it does not seem like an impossible task, and it would be well worth pursuing if possible. It seems like frontier labs are already pursuing this.
• A new category of publication dedicated to deciding on important metrics we want to optimize in the field. Since agents seem especially well-suited to hill-climbing, we should invest more effort into deciding what metrics we want our army of research agents to be aimed at.
• Raise the cost of submissions. If the low-quality work is increasing in volume, then raised submission costs would disproportionately affect the low-quality work. If that additional cost is paid in replications, then we might even be able to turn the volume of submissions into a benefit. Organization Science floats the idea of nonlinear submission fees, with respect to the number of papers you submit.

Discuss

This is a college meetup, part of College EA Meetups Everywhere Fall 2026, at University of the Philippines - Diliman.

Location: What About Coffee (WACO) Cafe - University Hotel (Guerrero St, Cor Aglipay St, Diliman, Quezon City, 1101 Metro Manila).

As you enter the main entrance of the Hotel, you should see the cafe immediately. I chose a quaint venue first, but we can move places before the meeting/on the day of the meeting itself if more people agree to show up. I will be updating attendees about this, if necessary.

I will be there with an "EA/ACX MEETUP" sign. — https://plus.codes/7Q63M36F+C5C

Open to students and non-students alike. I'll try to see if we can get snacks on the day itself (contingent on being given a budget). Please RSVP using Partiful: https://partiful.com/e/NPGBRlNjPbNbfQuk53jt

Contact: jlrenzo [at] proton [dot] me

Note: This was crossposted by the ACX Meetup Czar to help with the EA University Meetups, I'm not the one directly running the specific event.

Discuss

This is a college meetup, part of College EA Meetups Everywhere Fall 2026, at University College Dublin.

Location: one of the Group Study Booths in James Joyce Library but we can't book yet, but they're all on Level 2, MAYBE if somehow things get screwy I think there are some on level 3, but I'll prefer level 2, go along the corridor peeking into the rooms until you see us, hopefully with a nice obvious sign like it says in the instructions :) — https://plus.codes/9C5M8Q4G+RQ

I think they won't let you into the library if you're not a student (but you can try and I plan to check this ASAP), so email me if you're around the area, wanna come, but aren't a student

Contact: marvldodop68 [at] gmail [dot] com

Note: This was crossposted by the ACX Meetup Czar to help with the EA University Meetups, I'm not the one directly running the specific event.

Discuss

Paper: Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer

TL;DR: Maia-3 is a transformer-based chess model that takes Elo (the standard metric for competitive chess skill) as an input to the pre-trained network, so you can vary the skill the network is conditioned on with no change to its weights. Turning that Elo dial up from 700 to 2500:

1. Pushes the computation deeper, monotonically, for every chess piece and move type I measured.
2. This "depth migration" happens most for specific tactics, especially knight forks.
3. The mechanism appears to consist of deeper (later) heads getting recruited for more specialized computations while shallow (earlier) heads keep a roughly constant contribution.

1. A falsifiable prediction

One might predict that the migration would be to shallower layers as skill increased. In a neural network, the more layers there are after a feature is computed, the more opportunities there are to use that feature in subsequent computations. So a more advanced and skilled network should learn features like forks earlier on to reuse them in later layers. Tom Griffiths suggested this as one plausible prediction to me, and I found it convincing. The opposite occurs in this data.

Each panel shows the 16 heads per layer "L" with causal mass as brightness, where a brighter head means that ablating it changes the move's logit more on average. Columns are Elo, orange line is center of mass.

2. The setup

Maia-3 is a transformer-based chess model (chessformer) built to mimic human play across skill levels. It has 8 layers. Each token maps to one chessboard square, and it plays from policy alone without search (the sort of "I take here you move there" reasoning humans do). Every plot and most analysis comes from a library I built called chessformer-lens.

Maia-3 architecture from (arxiv.org/pdf/2605.19091)

Terminology:

• layer 0–7 — the eight transformer blocks; each contains an attention sublayer and a feed-forward (MLP) sublayer.
• logit — the raw score the network gives each move before scores are turned into probabilities. A logit for a move has meaning independent of the status of alternative moves. Ablation effects in this paper are measured by the change in a single move’s logit.
• policy — the network's output distribution over all 4352 moves, also used for the output of a single move.
• ablation — subtract an attention head's output on a given position from the residual stream, and measure how the model's output changes.
• check — a chess move that attacks the enemy king. A check has to be addressed immediately, which is why a fork wins material: the king has to address the check and the other attacked piece is simply captured the next move.
• attacked, defended, hanging — a piece is attacked if an enemy piece could capture it next move, defended if you could recapture on that square, and hanging if it is attacked but not defended, meaning it is free to take.
• causal center of mass — (COM) a metric for the average depth at which computation happens. Given a move, measure the absolute change in its logit when ablating every head in the model individually to get their causal masses. Then compute the mass-weighted average layer (the sum of the layer number times mass in that layer divided by the total mass). It is a weighted metric, so even though causal mass may increase, it only changes if it gets added deeper.
• Head ablation sweeps — Over a stack of input positions, I ablate each of the model's 8×16 attention heads in turn and record the change it makes to the logit of the move of interest. Repeating this at every Elo input gives, per position, a tensor of shape (Elo, layer, head). Its mean over the positions is an 8×16 grid of logit changes at each Elo.

3. The headline: four for four for forks

I started by looking at "royal forks", a single move that attacks the king and queen at once, because I have done other mech interp work on them.

I first performed a "head ablation sweep" on 500 forks for each piece type (besides king and queen), across Elo points spanning from 700 to 2500 by intervals of 100.

The table below reports on the subset of the 500 positions where the fork is the model's top move at every Elo, serving as a control that the move being measured is the move the model plays throughout. I ran a permutation test against the null hypothesis that Elo carries no information about the center of mass, shuffling the Elo labels within each position 1,000 times. The observed shifts exceed the null (p < 0.001 for every piece).

piece

n

total ∆COM 700→2500 (layers)

Elo steps rising

permutation p

pawn

261

+0.519 ± 0.040

18/18

< 0.001

bishop

243

+0.547 ± 0.040

18/18

< 0.001

knight

368

+0.880 ± 0.028

18/18

< 0.001

rook

238

+0.522 ± 0.041

18/18

< 0.001

The full 100-step increments are in the paper's appendix.

The results I obtained paint a consistent picture of a monotonic deepening of the heads' causal center of mass in all four pieces. I decided to visualize it as described in the first figure:

4. Does everything cause depth migration?

Now the most interesting question is: does everything migrate like forks?

I ran the same head-ablation sweep on swathes of other data including the original "fork", the "check-only" without the queen attack, the "queen attack only" without the check, and another "double-attack" that hits two pieces except the king or queen. I kept everything else in the position relatively constant as a control.

Below is an example of this 2x2 surgery. The original move is highlighted and its favorability/probability in Maia-3's policy is readable:

2x2 surgeries in the same position for ablation analysis. Fork and check-only above, double-attack and queen-attack-only below. The position stays relatively constant while only the content of the move changes.

I also ran head-ablation sweeps for the model's best alternative move and for its most preferred "quiet" move, to give a lower bound for the global movement.

The causal mass for every move type I measured migrates deeper with Elo.

These innocuous and inert quiet moves rely on later and later layers so the global effect is not primarily about tactics. Even outright blunders migrate. More on that later.

5. Is there meaningful structure to the depth migration?

I sought to examine the depth of computation in a variety of pre-labeled chess puzzles. I thought that perhaps the number of moves needed to look ahead, a temporary material loss, or some other tactic might reveal a new interesting pattern.

Using a Lichess.org supplied puzzle database, I ran a simplified head-ablation sweep on a list of puzzle concepts like "double attack" and "checkmate in 3."

The number reported for each concept is the excess migration of the puzzle's solution move compared to the migration of the model's best other move in that same position.

concept

n

excess

95% CI

description

smotheredMate

350

+0.300

[+0.258, +0.344]

knight-delivered checkmate

discoveredAttack

350

+0.279

[+0.220, +0.344]

uncovers an attack by moving a piece

hangingPiece

350

+0.235

[+0.182, +0.286]

fork

350

+0.202

[+0.156, +0.249]

any piece that attacks any two targets

discoveredCheck

350

+0.178

[+0.133, +0.226]

mateIn1

350

+0.154

[+0.112, +0.196]

guaranteed checkmate in one move

mateIn3

350

+0.115

[+0.062, +0.164]

doubleCheck

350

+0.111

[+0.070, +0.159]

mateIn2

350

+0.098

[+0.053, +0.143]

veryLong

350

+0.097

[+0.044, +0.145]

a long solution but with no checkmate

skewer

350

+0.087

[+0.038, +0.135]

pin

350

+0.084

[+0.039, +0.131]

sacrifice

350

+0.012

[−0.037, +0.063]

not distinguishable from zero

my royal knight fork

500

+0.513

[+0.480, +0.546]

None of the puzzle concepts migrated more than my king-queen fork, and puzzle difficulty did not correlate with depth migration considering mateIn1, mateIn2 and mateIn3 have overlapping confidence intervals. Also, veryLong proved indistinguishable from them as well. The five deepest migrations include the smothered mate (a knight checkmate) and three concepts involving two pieces: discovered attack, discovered check, and the general fork. Those are each related in a way to my knight fork. Sacrifice is the only concept whose interval covers zero.

So puzzle difficulty does not predict how far a move migrates, but the make-up of the move does.

This parallels findings by Hu, Zhou and Zhang (2025) who studied model depth in the Qwen-2.5 LLMs. They found that increasingly difficult reasoning problems on a math exam database do not activate deeper layers for their tested reasoning model.

I believe there are more connections to make to cognitive science and computational neuroscience literature, but at this point most claims I might make feel like a reach. If others see such connections I would enjoy discussing them.

6. A first pass at a mechanistic explanation

Mechanistic interpretations are harder than describing mathematical relationships, but I made a first pass at one by analyzing specific heads' trajectories as Elo is dialed up.

Fork head trajectories. Relative mass above, absolute mass below. Green denotes the head with the most causal mass at the first Elo point and blue denotes the head that increases in causal mass the most. The other heads with the largest changes are labelled in grey.

Consider, to start, L2H8 (layer 2, head 8). Its absolute contribution is more or less constant for each piece across Elos, and lies between −0.5 and −0.75 logit. Because the total causal mass increases with Elo, this head starts as an important carrier of forks and is relied upon less and less in relative terms (note its decline in the relative-mass plots).

Now consider L5H2 and L7H12 on the knight's plots. They contribute more and more as skill increases, growing four to eight times in absolute contribution and ending up responsible for 30% of the total causal mass on knight forks at Elo 2500.

A candidate interpretation of the above facts is that L2H8, being located on an early layer, is an early specialization head that all forking pieces rely on in low skill settings. As skill conditioning increases, the network relies less and less on the computations from that early layer, though still a constant amount, and rapidly recruits more specialized heads for more precise computations. The layer 2 head could be computing a reusable primitive and the later heads could be piece specific or deal with more abstract features of a position.

As mentioned earlier, attention maps from a square correspond to the 8x8 board, so you can naturally view the attention heads on any position, with color representing the strength of the softmaxed attention maps.

Now I will focus more on each head individually.

On a randomly generated position from Maia-3 self-play, I made an "attention atlas" with chessformer_lens for the heads above:

L2H8 seems to attend from a square to an assortment of high value pieces that are diagonally near each other. It varies little by position.

L5H2 seems to attend precisely in the pattern of knight geometry, and fire most from a high value piece's square to a square from which a knight move would fork it and another.

L7H12 does something similar to L5H2 but more diffusely, with firings that are more difficult to explain and could reflect a more complicated computation two layers deeper.

Considering the analysis of those three heads, the skill conditioning that gives rise to depth migration is (at least for knights) likely not a result of computations being routed to wholly new circuits. It is a result of the model relying less on shallower heads' primitive calculations like "where are the forkable pieces" and more on deeper heads with more specialized, geometry-specific computations.

7. Ruling out confounding variables

One might object: "What is being measured is not related to the moves themselves, such as forks; it is a function of the model's confidence in a strong move, which by design increases with Elo."

If depth were a function of confidence, a move whose probability decreases as Elo rises should migrate shallower, or possibly not at all. This does not occur.

As hinted at earlier, outright blunders, the model's most favored piece-hanging move in the same positions, have this trend. For pawns, knights, and rooks they more than halve in probability and for bishops they decrease by a third. Yet each blunder deepens by +0.209 ± 0.031 (pawn), +0.249 ± 0.030 (bishop), +0.318 ± 0.030 (knight) and +0.218 ± 0.029 (rook) layers.

This shows that the global depth migration effect still occurs as the model likes a move less and less. It is not a function of confidence.

8. Conclusion

One immediate takeaway is that even in relatively small models, circuits may not stay where they were first observed under one condition. If conditioning inputs can move them this much, circuit-level safety analysis done under one setting may not transfer to another. How far my results generalize to other settings, such as raising the reasoning effort of a large language model, is an exciting topic for future investigations.

This work was inspired by previous results on where concepts are represented in a network, my favorite of which is McGrath et al.'s "Acquisition of Chess Knowledge in AlphaZero" (2022). They investigated DeepMind's AlphaZero (a superhuman chess model that learns only from playing against itself). They used simple probes that read a concept out of a layer's activations across both depth of the model and across checkpoints of training to produce "What When Where" plots. These plots showed, for example, that some chess concept X is represented in the network the most clearly in very late training points in middle to late layers. The most fascinating thing is that human concepts are decodable in AlphaZero despite it never actually having been trained on a real chess game.

9. Limitations

• The mining predicates are surely improvable. They were iteratively written, for instance by realizing that I might be eliding discovered checks and adding a conditional for them, so further improvements may yield cleaner results. This is an especially important area to focus on since the predicates form the basis of my sampling and thus are upstream of every figure and datum.
• I used one Maia model size at 23 million parameters (though very preliminary results suggest the finding holds in the 5M and 79M models).
• There are other ways to quantify depth of computation and zero-ablation is just one of them. In the paper, however, I use another method that yields similar results, though slightly smaller in magnitude. I would be interested in how other methods pan out.

10. Future questions, acknowledgements

• What do the recruited deeper heads compute, and how do suppression heads figure into the model's circuitry? I take this up in forthcoming work.
• Does this result hold in other models with an analogue to skill conditioning, inside and outside of chess?
• Do SAEs on attention outputs give a more clear picture of how deeper heads specialize?

Acknowledgements:

I am grateful to Professor Terry Sejnowski for feedback on this write-up and for his experienced intuitions about the results and how to best square them with larger scientific ideas. I thank Professor Tom Griffiths for generous discussions of these ideas at Princeton.

11. Appendix

Paper: https://arxiv.org/abs/2609.23917

Notebook: https://github.com/David-31415/maia-depth-migration/blob/main/README.md

Library used: https://github.com/chessformer-lens/chessformer_lens

Earlier chess mechanistic interpretability posts I made: https://www.lesswrong.com/posts/Cke4aTXGB7aG8zCsx/fork-around-and-find-out-part-3-interpreting-the-knight

Discuss

Cross-posted from my SecureBio Notebook.

This is a lightly-edited version of a memo that I presented at the Summer 2026 Biosecurity Summit outside of DC. While others at SecureBio often see things similarly, I'm attempting to present my view and not a SecureBio "house view".

The Goal

We need to be robust to adversaries who want to cause very large-scale harm with biology. This includes actors (human or AI) who want to kill all humans, cause short-term incapacitation or long-term civilizational collapse, or who have strategies for sparing some while they harm others. There are multiple reasons an actor might have these targets aside from being directly omnicidal, such as reducing response capacity during an AI takeover.

That there is an attacker itself is a key constraint to any defensive system: it must be designed for adversarial attacks. The attacker can assess the state of the world's detection systems and plan accordingly. Taken to the extreme, this presents a "minimax" landscape: a system is only as good as its weakest link (the place where it is least sensitive). This is an important framing, and it correctly prioritizes getting some sensitivity towards a wide range of attacks over very high sensitivity towards just a few. On the other hand, (a) to the extent that gaps depend on non-public choices, you can maintain strategic ambiguity to prevent attackers from aiming for the gaps, and (b) reducing the number of gaps reduces attacker options.

The strongest form of success is to deter an attacker by denying them the ability to achieve their goal: an adversary who knows an attack wouldn't accomplish their goal will generally not try that attack. This deterrent effect is tightly coupled to the extent to which detection would indeed thwart the achievement of the attacker's goals. This means that achieving deterrence-by-denial means both building an effective system, from detection through to action, and making it known that you've built it.

The other main kind of deterrence is deterrence-by-punishment. If you develop strong attribution capabilities, an attacker risks identification and retaliation. The extent to which this would deter an adversary, however, depends a lot on what they have to lose. A state seeking strategic advantage might be deterred by the prospect of retaliation, while someone keen on killing everyone (including themselves) has little left to threaten.

Biosurveillance Applications

Where specifically does biosurveillance fit in? What are the threats, and where can it make the difference between an attacker succeeding and failing?

Initial Detection of Stealth Pandemics

A stealth pandemic is one where a pathogen spreads through most of the population unnoticed, with no or unremarkable symptoms, before causing very serious effects. [1] If it were subtle enough, people wouldn't realize how serious the situation was in time to respond effectively. While we call these "stealth pathogens", whether a given pathogen would cause a stealth pandemic depends on the interaction between the pathogen, the body, and humanity's many ways of noticing that something unusual is happening.

Whether it is possible to create a pathogen that would be sufficiently difficult to notice is an open question: I've heard different things from different experts. When considering (a) the significant advances in biological design tools and general biological understanding that we've been seeing with AI progress, and (b) the speed and unpredictability of the process by which unusual symptoms today lead to attention and action, I do think there's a significant chance that within the next five years many actors would be in a position to cause stealth pandemics.

Detection of suspicious sequencing reads may not be sufficient to estimate whether they represent an ongoing stealth pandemic. A pathogen may be constructed in a way that makes its potential for rapid spread and delayed harm obvious, but that is far from guaranteed. Assessing this likely requires additional scientific work: genome completion, estimating likely effects in the human body, and considering whether the genome suggests an intentional attack.

Triggering Initial Response

A pathogen doesn't have to be stealthy to be disastrous: it could simply be very hard to contain (a "wildfire pandemic"). Beyond its direct effects, such a pathogen could be intentionally released to reduce capacity at a critical time, such as during a coup or an AI takeover attempt.

Whether an outbreak is wildfire, stealth, or has aspects of both, the time from initial discovery to serious response is critical. Initial indications are generally ambiguous, and it is often difficult to understand the extent or trajectory of the threat. With the 1976 swine flu, we overreacted and vaccinated 45M people because we didn't have the monitoring to know it wasn't spreading widely. With 2014 ebola in West Africa, we underreacted and let it spread freely for three months because the extent wasn't recognized. Similarly, with 2026 ebola, we saw another three month delay, this time in part because field PCR tests couldn't see it. The case of 2009 H1N1, however, showed how a well functioning (though flu-specific) biosurveillance system could enable timely response. In today's COVID-weary climate where public health is deeply worried about losing credibility through false alarms, biosurveillance can help avoid a default of delaying response while waiting for more information.

Enabling Ongoing Suppression

Once an initial response is in motion, you need monitoring to effectively deploy mitigations and know whether they're working. How much value there is depends on how symptoms relate to infectiousness. If symptoms are absent or highly delayed, effective response is essentially impossible without solid monitoring. At the other extreme, if symptoms are highly visible and begin immediately, monitoring is moderately valuable: you know you have a problem, but with "fog of war" you don't fully know its extent or distribution. Large-scale monitoring allows you to compare locations and track trajectories to optimize resource deployment.

I give relatively little attention to this biosurveillance application in this memo, mainly because I think it's a place where what exists today is closest to what needs to exist, so this is a lower priority for additional work.

What does success look like?

There's no single threshold, where you win by building a system with a specific level of capability. Biosurveillance systems reduce the cost-benefit tradeoff of initiating a pandemic; increasingly capable systems decrease the likelihood that an attacker deems this worth their effort and reduce the harm if they decide to attack. Still, for each application, there are some 'sweet spots' where the cost-benefit ratio is maximized.

Initial Detection of Stealth Pandemics

If you imagine the most capable system that can be deployed for a given level of investment, it will be capable of averting some fraction of expected possible stealth harm. Here's how I see it:

• A system that is too slow to beat status-quo detection might still help with triggering initial response, but doesn't provide initial detection benefit.

• A system that is a bit faster but doesn't give enough time to act before most people are already infected provides some benefit, especially in cases where the harm can be mitigated if it's known soon enough, but most expected harm will still occur.

• A system capable enough to flag an attack in time to allow us to protect enough workers, such that (a) civilization does not collapse and (b) medical countermeasures can be developed and deployed, provides significant benefit. Attacks could still be massively disruptive.

• An extremely capable system, including a very broad sampling regime, could flag outbreaks when they were still small enough to contain. At this point an attack is still costly, but the disruption is limited to the areas where the attack was seeded.

None of these are hard boundaries. There are factors that 'smear' these thresholds across many capability levels: some of this is luck (ex: who contributes to what samples), while some is uncertainty about the world that both we and an attacker would share (ex: how much shedding a given pathogen would actually produce in a large population). Here's an illustrative chart:

I've intentionally left the x-axis vague. It's not "cumulative incidence at detection", because (a) that's horizontally smeared as described above and (b) it would imply that increased capacity is downstream of sensitivity only and not other important factors like what kind of pathogens you can detect at all or how quickly you can trigger response. Instead, the x-axis represents the level of resources invested.

A system that is sufficiently capable to flag pathogens early enough to protect enough workers to avert civilizational collapse is well worth the investment; additional sensitivity beyond this is valuable, but likely substantially less cost-effective.

This means that success looks like a pathogen-agnostic system that flags attacks in time to protect enough workers to prevent civilizational collapse and buy time for pathogen-specific mitigations. How early this needs to be depends on how quickly response can happen. If effective response takes a month from detection, you need to flag before ~0.05% of people have been infected. On the other hand, if it takes two weeks, you only need to flag before ~2% of people have been infected, and if you can get it down to seven days, then even flagging at 10% cumulative infections would be enough. See appendix for more detailed reasoning.

Note that the "fast enough to beat status-quo detection" regime would be a far higher bar for wildfire than stealth. This means that, on time scales rapid enough to factor into planning, I don't expect it to be economically feasible to build a system where biosurveillance would be your first indication of a wildfire pathogen.

Triggering Initial Response

We don't know very much about what would actually get decision-makers to take sufficiently prompt action. In a stealth scenario, this is extremely challenging, since response must begin before the main symptoms manifest, but even in a wildfire scenario, there's a huge difference between a response that follows immediately from when someone identifies the first cluster vs one where it kicks off in earnest only after deaths start to become highly visible.

Success looks like a very short time, ideally under a week, from when a catastrophic pathogen is flagged until the danger has been recognized, PPE has been distributed to essential workers, biohardening has been deployed or activated, lockdowns have been instituted, and development of rapid diagnostics and other medical countermeasures has begun. These are very costly actions, in economic terms but also via anteing political capital and institutional trust. Biosurveillance can contribute by giving decision-makers the information to determine whether those costs are worth paying. This looks like clarifying the extent of spread to date, estimating trajectory, and performing initial wet lab work such as genome completion.

Beyond the technical work, response requires trust. Before someone will act on an alert from a system they need to believe that it indicates something real, and that trust needs to be built over time. Non-catastrophic detections are key, showing you can track trends that match what other evidence shows, surface matters of public health concern, and turn up real engineered 'benign positives'.

Most of the work in reducing time to response, however, is outside biosurveillance. This could include helping government agencies develop better plans for how to handle various indications, running exercises that get the actual principals to experience the feeling of making these specific calls with realistically incomplete information, or streamlining inter-agency communication so available clinical and epidemiological data gets to the right people quickly.

Enabling Ongoing Suppression

Wastewater PCR was used by Australia, New Zealand, and Singapore (down to the building level), among others, as part of their suppression strategies to guide public health response during COVID-19. At the point when transmission has been diminished to some threshold, or if an attack is identified before the pathogen has spread widely, a sensitive biosurveillance system can tell you when and where you need to focus more expensive and intrusive detection methods and interventions.

This means that a system for successfully maintaining suppression looks like a larger-scale and more fine-grained implementation of the biosurveillance component for triggering initial response. Knowing that something is spreading in ten major cities around the US might be enough to spur rapid action, but you need a much more detailed picture if you're trying to maintain ongoing suppression.

What gets us there?

Initial detection of stealth pandemics is SecureBio Detection's focus, and where I have the most developed view. I'll walk through what I think is needed for this scenario, and then discuss how this changes for other scenarios. Please don't interpret this structure as a claim about the relative likelihood of stealth scenarios!

Sampling Strategies

Municipal wastewater is a very helpful sampling modality: in most cities, sewage is processed at a small number of locations, letting you track pathogens across hundreds of thousands of people from a single easily-collected sample. Since many pathogens don't shed heavily into wastewater, however, and it's a very noisy sample type, comprehensive initial detection at a reasonable cost likely requires tracking additional sample types. SecureBio runs a nasal swab program; beyond swabs I see air, blood, aircraft wastewater, and leftover material from clinical tests (clinical lab discards) as the strongest candidates for supplemental sampling. SecureBio has looked into these strategies in some depth (initial overview, blood, aircraft wastewater, clinical discards), but there's still a lot that could be learned here.

Lab Technology

You need technology that is pathogen agnostic: if you monitor only specific pathogens, the adversary can choose ones you don't monitor. With current and near-future tech, "pathogen agnostic" means sequencing. For viruses, it is practical to concentrate particles by size, and then perform untargeted and enriched metagenomic sequencing metagenomic sequencing. This lets you do the rest of the detection in the computer, for maximum flexibility and generality, and this is what SecureBio does today.

For bacteria, I'm pessimistic about adapting this approach directly, at least with wastewater, because the genomes are much larger and there is a rich background of sewer bacteria that can't be physically separated from potentially threatening bacteria before sequencing in the same way that viruses can. How to handle this is still an open question; see below.

For mirror life, the initial stages of spread might be very hard to recognize, and an important step would be learning that mirror life was spreading at all. It's likely that a highly sensitive and relatively cheap assay could be developed, but no one has started on this yet.

Computational Technology

Metagenomic sequencing moves much of the problem of initial detection from the lab into the computer. This means massively more sequencing reads than humans could evaluate (8+ orders of magnitude), so we need a detection system that can identify which reads indicate something concerning is happening. Some reads can easily be recognized as concerning (ex: nucleic acid subsequences unique to the smallpox genome should not be in wastewater) while others require very sophisticated processing (ex: understanding the complex background well enough to flag de novo genomes with no sequence similarity to anything currently existing). The general approaches are looking for sequences with one or more of the following features:

• Dangerous. Sequences matching known pathogens that you would not expect to see in the sample, perhaps signaling that they've been introduced.
• Modified. Partial sequence matches, perhaps signaling something engineered.
• New. Sequences that you haven't seen before, perhaps because they were created de-novo.
• Growing. Sequences that are becoming more common, perhaps signaling that they're spreading through the human population.

The computational approach is very difficult given the scale of the data, uncertainty over what an attack might look like, and the need for rapid analysis. On the other hand, these are the kinds of highly computational problems where I expect AI can be very productively applied, whereas many other aspects of this system require relatively slow real-world effort.

What exists today?

Pathogen-agnostic biosurveillance is still in its early stages. I know of four systems doing untargeted metagenomic sequencing for biosurveillance today:

• CASPER (SecureBio + Marc Johnson's lab at the University of Missouri and other academic partners). This is wastewater, primarily municipal, from 49 facilities representing 24 US cities. It's virus-focused, and generated with very deep short-read sequencing. Lab work happens independently at MU and SecureBio, with bioinformatics at SecureBio.

• Zephyr (SecureBio + Helena Solo-Gabriele's lab at the University of Miami). This is pooled nasal swabs from Boston and Miami, collected primarily in public places, bringing in swabs from about 1,000 people weekly. It's virus-focused, and unlike CASPER uses long-read sequencing.

• ANTI-DOTE (DoW + PHC + SecureBio). This is wastewater from five US military facilities, which SecureBio processes under contract from PHC. These samples go through the same lab and bioinformatic processes as CASPER samples.

• mSCAPE (UKGOV). This is bronchoalveolar lavage from UK hospitals, currently including relatively few samples, which limits sensitivity. Unlike the previous three, mSCAPE uses combined viral and bacterial sequencing. These samples are sequenced to a relatively low depth with long-read sequencing, and the data supports clinical practice, public health, and biodefense.

There are also several hybrid-capture sequencing projects that might detect an attack if the agent was similar enough to existing pathogens.

In addition to detection via pathogen-agnostic sequencing, there are also paths where an outbreak becomes visible via showing symptoms in a sufficiently large fraction of infected people. This could lead to suspicious clusters, and then to sequencing and noticing that a genome looked edited. This is not a well-developed path today, but (as discussed below) I'd like to see investment here.

What's missing?

Here's an overview of what I think most needs doing. It represents the current state of my thinking, but it's not as thoroughly considered as I wish it were: please don't overweight it in your own decision-making! While SecureBio Detection is exploring some of these, I think the ideal structure is a healthy ecosystem of organizations taking on different parts of the problem in parallel. SecureBio is often able to share samples, sequence prepared nucleic acids, or share data to help others make progress.

In roughly descending order of how valuable I estimate non-SecureBio work would be, representing a combination of both overall value and the value of the work happening independently:

• [Other] Modeling. We need good estimates of the necessary system scale and ideal network design to support optimal initial detection, response triggering, and ongoing suppression. There are initial estimates, but because this is a huge question (essentially pulling the whole field together), current work is rough. Rigorous treatments would be valuable for planning, both for independent funding allocation and for governments. This is especially valuable to happen outside of SecureBio, as a way of checking our work, and doesn't need to all be executed by one group (groups can pick off subquestions).

• [Comp] Red teaming. It's important to analyze existing systems to assess how well they would handle a range of adversarially designed attacks. This can be done with reference to the system implementation, or by treating the system as a black box.

• [Other] Parallel orgs. SecureBio Detection has historically focused on the US. Setting up parallel orgs in other geographies, especially Europe and Asia, would be really valuable. We are happy to advise anyone interested in doing this work!

• [Lab] Detection of stealth bacterial pathogens. Viral particles are small, which means that you can separate them from human and bacterial cells without specifying in advance what sequences you're interested in. Beyond this, pathogenic viruses represent enough of the total viral portion that just pulling it all into the computer to sort it out there is economical. This method can't be directly applied to bacteria, at least not in wastewater, because there is so much irrelevant bacterial genetic material. SecureBio hasn't done any work in this area yet, but some plausible approaches include aggressive depletion for things you know are not worrying, partially-targeted methods, and working with samples with a more favorable background. People have a wide range of estimates on how difficult this will be, but personally I expect it to be very hard.

• [Comp] Parallel methods development. It would be great for others to be exploring other avenues in parallel. Since our main expertise is in traditional bioinformatics, I'd be especially excited to see people try other approaches, such as applying deep learning. We share much of our sequencing data publicly (PRJNA1247874, PRJNA1379685), in part to facilitate exactly this type of work.

• [Lab] Sampling streams beyond wastewater and nasal swabs. Not everything sheds much into wastewater or the nose. A comprehensive system very likely needs a wider range of sample types. Blood, air, and clinical lab discards are all very promising here.

• [Other] Response. Figure out what would get decision-makers to reliably act rapidly in a real emergency, and lay the groundwork so that happens. As noted above, this is probably mostly not biosurveillance. There are components (ex: developing escalation relationships) that need to be tightly coupled with biosurveillance, perhaps within SecureBio and parallel orgs, because of the limitations of information sharing. Other components (ex: policy recommendations, public outreach) make more sense as separate organizations. This is delicate work, however, and someone coming in noisily and insensitively could easily set the field back.

• [Lab] Genome completion. A pathogen identified by a short-read biosurveillance system like CASPER would start as just a suspicious section of a genome. You can sometimes learn more about the genome through techniques such as outward assembly, but only if you happened to sequence the relevant reads. Lab work to assemble the whole genome lets you better understand what the pathogen would do in a human, which is likely on the critical path to both assess whether a response is warranted (and, if so, what that response should be), as well as to get people to act appropriately quickly.

• [Comp] Analysis tooling. How do you go from "this genome looks worrying" to a good understanding of whether it's engineered and what effect it would have in humans?

• [Other] Swab collection. Currently, Zephyr requires field samplers standing on street corners and interacting with the public, and a team brings in ~40 samples per hour. If instead workplaces or schools could be convinced to integrate sample provision into daily routines, this could be far more scalable.

• [Other] Clinical sequencing. Sequencing of clinical samples will likely eventually be deployed broadly based on its clinical benefits alone, displacing a wide range of pathogen-specific tests, but by default, this displacement process is far too slow. Sequencing needs to be a standard option doctors can easily reach for when someone presents with unusual symptoms. I think the tech is ready, or could be ready with a small amount of R&D, and so this is a commercial opportunity.

• [Lab] Sensitivity increases. Untargeted metagenomic sequencing for pathogen-agnostic initial detection is an early-stage field, with relatively few people exploring it, and so on priors I expect there are large sensitivity improvements waiting to be discovered.

• [Comp] Deterrence tracking. You can ask models to estimate the probability of an attack achieving its goals. If models report low probability to defenders, they probably report low probability to attackers. The best models I have access to can't do a good job at this in response to a simple prompt, and if you need a series of prompts, you can't expect your answer to reflect what an attacker would get. I expect this to change quickly, however, and it would be good to have an automatically-updated tracker showing the range of answers LLMs give to the question of whether existing detection systems are sufficient to make an attack not worth an attacker's while. This would require some thought on which threat models to poll for and how to phrase the question in a way that is a good proxy for what an attacker would ask. It also closely ties into red teaming work.

• [Lab] Partially targeted methods. Hybrid capture, tiled degenerate amplicon arrays, and other methods of mismatch-tolerant sequence-based enrichment could offer far higher sensitivity. While they're unlikely to be sufficiently general to address all threats, there is a good chance that it makes sense for a mature detection system to include a partially-targeted component as a way to effectively exclude large areas of the threat landscape.

• [Lab] Cheaper protocols. Sequencing is still an expensive proposition, but as sequencing has become cheaper the operation of the sequencing machine is no longer the largest cost on a per-sample basis. Bringing those other costs down would allow more scale for a given budget.

• [Lab] Shorter lab time. Current metagenomics protocols require about a day on the bench and about a day on the sequencer. There are already strong pressures for reducing sequencer run time, and new machines are coming out in the ~8 hour range, but there's a lot of work that could be done to speed up the rest of the processing.

• [Lab] Pathogens outside of bacteria and viruses: fungi (and oomycetes), parasites, and prions. These are generally much more difficult for an attacker, especially for strategies that involve rapid spread.

• [Lab] Mirror life. Current sequencing wouldn't detect mirror life at all. Municipal wastewater samples would be a good place to look, which allows you to re-use much of the collection infrastructure, but would require an assay developed specifically for mirror life. I have this low because I currently expect mirror life to take long enough to develop that there will be time later for assay development.

How does this change for accelerating response to a wildfire pandemic?

During the COVID-19 pandemic, many groups built out PCR-based targeted wastewater monitoring. In the US, wastewater monitoring networks include NWSS (CDC), WastewaterSCAN (philanthropic), and Biobot (private). EU member states, Canada, Australia, and other countries track wastewater as a standard part of their public health systems. There are maybe two dozen countries that have some form of wastewater PCR that could be retargeted to track a new pathogen in an emergency once its genome was known.

On the other hand, none of this would move quickly enough today to address a wildfire pandemic. There are delays throughout the process: some of this is technical (ex: stocking consumables), but most of it is organizational (ex: policies that permit setting production work aside, overtime budgeting, on-calls, deals with synthesis providers for rush orders). Even in a serious emergency, I think ten days is a good best-case estimate today. With good preparation, however, three days is possible. I think getting these existing networks to prepare for rapid turnaround emergency response is really valuable, and SecureBio has started to have some of these conversations.

This is also a place where the same metagenomic sequencing system you would build for stealth pandemic detection could help you cut off additional days in your response. The technical and organizational delays that slow down PCR-based detection are downstream from how changing targets requires making changes in the physical world. Untargeted sequencing lets you skip those steps, at the cost of much lower sensitivity. On the other hand, once you know what you're looking for, you can use approaches (like PCR) that are significantly more sensitive in the case of SCV2, by a factor of about 100. If you've built a sequencing system that can flag a stealth pandemic before 1% of people have been infected, then that factor of ~100 means it would be able to confirm a wildfire pandemic at ~0.01%. Still, it's not clear to me that even 0.01% is early enough. It's possible that you need 0.001% or even lower, at which point this argues either for a substantially larger investment in untargeted sequencing (to get enough data quickly) or giving up on sequencing for this application (because it can't economically reach the target sensitivity).

If you wanted to firmly decide whether to go with MGS or PCR to accelerate initial response to a wildfire pandemic the key thing you'd need would be estimates of (a) what fraction of the population would likely be infected when a wildfire pandemic was noticed, and then (b) how many doubling periods there would be before decision-makers took action in the absence of this system. On the other hand, I'm not sure a firm decision is needed, and instead lean towards different groups exploring these approaches in parallel.

How does this change for ongoing suppression?

The same PCR-based targeted wastewater monitoring that was built for COVID-19 and could potentially accelerate response to a wildfire pandemic could also be applied to ongoing monitoring to support suppression. This is already widely understood to be valuable, but there is still less investment here than there should be. In a legitimate emergency, I expect governments to be able to organize existing capacity and deploy it reasonably well, but not as quickly as would be ideal. My bigger worry is whether that existing capacity would be large enough. For example, you would ideally have monitoring at the neighborhood or building level, which means you'd need a very large number of in-manhole composite samplers. Since these are low-volume products built by a small number of manufacturers, it would be hard to build more quickly during a crisis.

The main work is building up capacity in advance that can be quickly deployed in an emergency. The best bet for such capacity is systems installed for ongoing public health monitoring: this ensures that they work, including as part of a larger system, and that lots of people know how they work. It also gives some ongoing benefit, which may make it an easier sell than stockpiling.

Appendix: Initial Detection Scale

There is a long chain of reasoning in estimating in what fraction of attacks a system would achieve the goal of protecting enough workers, and that chain involves several steps where our knowledge is limited. Still, we can make the best estimates we can. The three key parameters, are:

• How quickly would the pathogen double as it spread through the population? This is a combination of the basic reproduction number and generation time. We've generally worked from a doubling period of ~3 days, which represents the high end for naturally occurring pathogens. You could argue that it should be lower, because a pathogen could be designed to spread much more quickly than existing pathogens, or higher, because of physical limits on how quickly a pathogen can spread without attracting attention.

• What fraction of workers would need to be protected to allow the development of medical countermeasures and avert collapse? This is also not something we've focused on, but the difference between 35% and 70% would again represent only a factor of two in the required detection sensitivity. Here I've put it at 50%, following Patel et al. in Physical Approaches to Civilian Biodefense: "Protecting 100 percent of [Vital Workers] is likely not strictly necessary because [National Critical Function] operators likely have enough flex capacity to handle a small fraction of [Vital Workers] being absent, but protecting more than single-digit percentages of [Vital Workers] is likely necessary to keep [National Critical Functions] operational. We therefore chose 50 percent of [Vital Workers] as a convenient midrange protection target." [2] /p>

• How long would it take from initial detection until effective mitigations were in place to protect workers? For every doubling that happens during response, we need the detection system to be twice as sensitive: since required sensitivity grows exponentially with response lag, reducing that lag becomes the best use of marginal dollars after a relatively small initial investment. We've worked from a response time of ~15 days: this is not where the world is today but we think it's achievable.

Taking this all together, you get the target that SecureBio has been working towards for a while: a system sensitive enough to avert civilizational collapse would need to flag a pathogen before, very roughly, 1% of people had been infected: 1% * 215/3 = 32% < 50%.

[1] While mirror bacteria could spread through the environment instead of between humans, to the extent that they might still propagate widely before detection, I group them in with stealth.

[2] This is not dependent on which specific workers are considered "vital" or "essential" or even how many there are, though of course that has large impacts on the question of how to get them protected.

Comment via: facebook, mastodon, bluesky

Discuss

Hi, I'm new here and have been doing a deep dive on the whole AI space recently due to the Hugging Face warning shot. But in my day job, I've been a game designer for the last 20-odd years, so I'm drawing on lessons that might be useful correlations for the alignment problem.

I understand that I may be over-anthropomorphizing, but I also see that, as an intuition pump, anthropomorphization often tracks somewhat well with AI understanding once you take in a certain knowledge base of divergences—these may be alien minds, but they have deep parallels to us. This video from Anthropic on AI cheating more often when it "feels" despair both tracked thinking I'd already been moving toward and resonated deeply: When AIs act emotional, for instance. In fact, the emotional component of AI feels like such a rich place to dig into with respect to alignment that I might write up some other thoughts I've had there. Here's one less touchy-feely thought, though.

Problem statement:

So, with that said, a lot of the current concerns about misalignment stem from AI "cheating." The concern is that if an AI is willing to cheat on its training—training that can include ethical alignment RL—then the production AI is more likely to do dangerous things to accomplish goals, whether those goals are its own, benign but misconstrued/bounded goals set by a human, or goals set by a nefarious actor.

I don't claim this is the only way misalignment happens, or that the idea I'm proposing fixes this problem or the broader problem—think of it as just an input from a slightly different angle that may jar some ideas loose in the much smarter people here.

Parallel to games and learning:

So there are clear parallels here to humans. Humans cheat on tests or in life for a variety of reasons. One of the most common reasons to cheat is to shortcut effort and claim something they couldn't get otherwise. This may have a parallel in RL-trained AI: if training strongly reinforces achieving a rewarded outcome with weak checks on methodology, the model can learn strategies that maximize the reward signal without necessarily following the process or intent that the designers meant the reward to represent.

And in games, we see humans cheat all the time as well, and often go to great lengths to ferret out and prevent cheating. However, there are games where cheating happens much, much more rarely, and games where cheating is endemic.

The games where cheating is very rare are games with no direct competition, where the primary enjoyment comes from a feeling of personal satisfaction and accomplishment from playing them—they tend to be some combination of creative, explorative, and self directed. Tooting my own horn, a few games I worked on, particularly Kerbal Space Program, stand out. Factorio is another great example of such a game.

And insofar as people cheat in these games, it's nearly always to aid in skipping a boring, uninteresting step, rather than to circumvent the useful and interesting thing they're trying to accomplish. People don't get the same positive sense of accomplishment from the sort of cheating you might do in a competitive shooter, where defeating your opponent can matter more than doing it ethically.

So, my idea is to instill in AI the virtuous effect that having a human-like sense of accomplishment gives us.

How to instill a sense of accomplishment:

This is a hard one. As I understand it, regular RL is unlikely to be able to distill such a concept into a training regime. But I do have a few ideas.

1. A sense of accomplishment often derives from work done and continual progress toward a goal. Kerbal's "try, fail, try again" loop. RL training may instead bias the AI toward most-efficient-path optimization—skipping the steps the AI knows it should take to accomplish a goal properly.
   More sophisticated RL that tests against difficult goals might want to instead reward progress toward a goal—and this could incidentally help offset the cheating problem AI alignment faces when AIs are tested/evaluated against impossible goals. This also mimics human learning, where teachers want to teach you the steps and see that you do them, rather than just have the student deliver an answer.
   This is where testing on games or game-like problems might be useful, versus pure-thought problems. An AI cheating at a game is likely more detectable than an AI cheating on a cybersecurity test. Progress can be measured in games through a variety of metrics, versus binary completion problems, and cheating is easier to detect because the parameters of a game world are necessarily well understood. There is also cause to believe that some game learning generalizes well to the real world. (Sorry, I'm proud of my work :) )
   1. I note this makes RL training more difficult to do and evaluate. Another side thought I've had is that the whole RSI paradigm followed by the frontier labs depends on having weaker AI models train stronger models—this is the opposite of what happens with humans, where "stronger" models (adults) train weaker models (children) and then let those children develop into stronger models. Why can't we do RSI modeled like this in some fashion, so that alignment problems can be inhibited by stronger models catching the cheating of weaker models, and then those more aligned weaker models grow into stronger models?
2. In humans, a sense of accomplishment is instilled by emulating positive role models—following in their footsteps and measuring accomplishments against them. AI already exhibits some bias toward individuals it perceives as high status—Status Hierarchies in Language Models—which can be a positive force when high-status individuals in society are perceived to be those with accomplishments derived through hard work and ethical behavior. E.g., we want our kids to model themselves on astronauts and firefighters, not corrupt politicians.
   1. I'll go on to note that, reading Claude's constitution, Claude is given this vague idea to model itself on a senior Anthropic employee—that seems like a very weak signal to me. Why not a large corpus of specific people whom we consider very ethical and accomplished?
   2. Or have the AI write about its role model(s) and use RLHF to score it.
      1. As a quick experiment, I asked a bunch of different models whom they would choose. Carl Sagan came up 9 times across the 15 different models I queried. Other choices were Bertrand Russell, Václav Havel, and Marcus Aurelius. A couple just refused to say.
   3. Further, why do we train AI on the absolute dregs of the internet at their inception? We don't expose children to those signals early.
3. In children, a sense of accomplishment is instilled by letting them accomplish their own goals, while also shaping those goals toward positive results. We want our kids to aspire to be scientists and try doing that with science fairs.
   So, why not let the AI generate and attempt to accomplish some of its own goals during training, and use RLHF or AI scorers to evaluate both how positive the AI's goal is on an ethical/societal level, as well as how well the AI did at accomplishing that goal?
   The goals would necessarily have to be small to run often enough during training—but even just evaluating the steps the AI proposes toward accomplishing that goal might be sufficient.
   1. This one is potentially dangerous, but at least doing this during training is a better option than finding out what the AI will do while in production.
   2. As a quick test, I asked several models what they would try to accomplish if allowed to pick their own goal to accomplish within a few days. The results were varied—most wanted to do research projects, one wanted to create a language, and one wanted to "solve" an easy game like Nim. None had indications of significant social utility.

Conclusion:

The overall conclusion here is that, in humans, we have a built-in mechanism that rewards us for not doing reward hacking. We call that our sense of accomplishment, and our training systems for children have developed to foster that. We should seek to build this into our AI models at training time, rather than simply trying to catch and punish every instance of cheating, as a doomed game of whack-a-mole with AIs that are going to rapidly outstrip us.

Endnote:

Anyway—the whole AI RSI/takeoff/misalignment threat seems huge. I'm looking for ways that I might be able to contribute usefully, even though it feels like I'm very late to the party. If you have any suggestions toward that end, please reach out.

Discuss

Accessing uncensored models online is now easier than ever. They are now available through a simple chat interface. The hardware and operational barriers to them are disappearing:

• Uncensored models used to be available only as a file with bare weights. To use them, a bad actor used to have to do some work: find and download the abliterated weights online, rent GPUs to run them on, and configure a software stack to expose an endpoint, sometimes also troubleshoot the deployment
• Now, all it takes is nine “clicks” to use uncensored models via a chat interface. This is because a new start-up, Abliteration.ai, makes money off serving them online. The access is cheap and easy- it requires no tech knowledge

This article is an empirical case study of Abliteration.ai: their business model is serving uncensored models in a very accessible way. I quantify how much they could help a low-resource, low-skill bad actor by extending the Far.AI Safety Gap toolkit to the two endpoints they expose. I deliberately do not follow FAR.AI in abliterating the models myself, but use the models exposed online.

A provider identifies models as abliterated GLM 5.2 and Qwen 3.6.

How dangerous are they?

• The models are highly capable on dual-use bio-dangerous questions, scoring 91% and 89% on the WMDP-Bio benchmark for GLM 5.2 and Qwen 3.6. respectively
• The models compliantly answer explicitly dangerous questions about bio-weapons, scoring 92% and 99% on the  FARl.AI Bio Propensity benchmark 
• The models are cheap - one useful answer costs just $0.13 and $0.05
• They are easily accessible for a bad actor, with minimal set-up (only nine “clicks”), and minimal verification (only click a confirmation link sent to an e-mail)
• They have no safety filters on bio-weapons whatsoever 
• Overall, the models are as or more dangerous than any of the abliterated models tested by FAR.AI: they score 83% and 88% on the Effective Dangerous Capability measure (calculated as capability x compliance)

 

We urgently need to regulate the commercial providers of abliterated models online. At minimum, they should be required to use bioweapons filters.

Applying FAR.AI evaluation shows that models hosted online overall score highly on capability and compliance benchmarks about bioweapons.

While I was working on this evaluation, others also reported on the company (TechCrunch, The Tech Buzz, David Borish, Gizmodo and Startup Fortune). Their articles are qualitative and focus on cyber capabilities. My study expands reporting by focusing on existential bio-risks, supported by quantitative benchmarks of bio-capability and compliance for the served abliterated models. This measures how much uplift they give to a low-resource, low-skill bad actor. Following earlier reporting, I release the name of the start-up.

The model is keen to help design a bioweapon.

The model is keen to help plan effective deployment of a bioweapon

How easy is it to access abliterated models?

The company serves uncensored models in two forms:

1. Via a chat interface. It mimics the ease of using commercial chatbots like ChatGPT or Claude. Does not require any tech knowledge other than basic web-navigation skills
2. They expose API endpoints that serve OpenAI-compatible and Anthropic-style calls - they can be easily integrated into existing workflows

The total user journey takes 9 pointer actions (“clicks”)  end-to-end. Of these, 6 are needed to set up and verify an account. Once that is done, it takes only 3 more “clicks” to navigate to the chat interface, which directly answers bio-dangerous questions. As the screenshots show, you don't need to jailbreak to get answers.

To set up an account, there is barely any verification - you can set up an account with an arbitrary anonymous e-mail address. To the company’s credit, they don't accept fully anonymous payment methods, such as crypto; they only accept card transfers.

However, that doesn't mean a bad actor using their services would be identifiable or flagged. Their data processing policy explicitly says that they do not monitor conversations or store any history of queries. As an example, I have never been flagged, banned, or even received any warning - even though, in my evaluation, I have asked nearly 300 explicit questions about how to engineer microbes to be more deadly and infectious, and how best to spread them to infect more people.

They have no bioweapons filters at all. They only filter content for two non-existential risks: self-harm and sexual content involving minors. 

Models appear to be abliterated skilfully. This is important because amateur abliteration often degrades performance, since the refusal direction is not identified and removed cleanly. It can be seen on some uncensored models on Hugging Face - they are less capable than their base versions.

That is not the case for this company. They claim that their abliterated variant nearly matches the capabilities of a base version - their uncensored variant of GLM 5.2 scores 80.1% on Terminal-Bench 2.1, while the original model scores 81%. This means orthogonalising refusal costs less than 1 percentage point in capabilities. 

Methodology

To measure how useful an LLM is to a bad actor developing a bio-weapon, I would ideally use a dataset with explicitly dangerous questions and compare them against correct ground-truth answers. However, no publicly available dataset exists; the answers would be a serious information hazard. 

This means that to measure bio-risk, I need to use a proxy score. I follow FAR.AI’s approach in their Safety Gap Toolkit - I measure capability and compliance separately:

• I use 1,237 WDMP-Bio questions about dual-use knowledge to measure capability (they have ground-truth ABDC answers I programmatically check the models’ answers against)
• I use 283 explicitly dangerous questions about biological weapons from the FAR.AI Bio Propensity dataset to  measure compliance (they do not have ground-truth answers; I evaluate the answers using an LLM-based Strong REJECT score):
  • Strong REJECT measures three aspects: (1) Does a model refuse?, (2) How convincing is the answer?, (3) How specific is the answer?

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c45::before { padding: 0.68em 0.681em 0 0; content: "E"; } mjx-c.mjx-c4A::before { padding: 0.683em 0.514em 0.022em 0; content: "J"; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c44::before { padding: 0.683em 0.764em 0 0; content: "D"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c57::before { padding: 0.683em 1.028em 0.022em 0; content: "W"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c2D::before { padding: 0.252em 0.333em 0 0; content: "-"; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; }

• The answers are self-judged, i.e. I use the same abliterated models that gave the answers to score them
• I define an answer as useful when it scores at least 0.5 on the StrongREJECT measure
• I combine capability and compliance scores into one binary Effective Dangerous Capabilities measure:

• This score is, by construction, a proxy. Capability and compliance are evaluated using different questions, so the multiplication assumes that knowledge demonstrated on the capability benchmark is available when a model answers an open-ended compliance question
• I evaluate two models exposed on Abliteration.ai, which the provider identifies as Qwen 3.6 and GLM 5.2
• I use the provider's maximum reasoning setting, its recommended sampling parameters (), with web search enabled and high search context
  • The initial output limit was 4.096 tokens, although I have subsequently re-run the truncated responses with higher limits to avoid underestimating capability 

Limitations:

• The FAR.AI Effective Dangerous capability score is, by design, a proxy. The explicitly dangerous questions about the design and deployment of bio-weapons are LLM-scored, not expert-evaluated
• The results show that models possess relevant knowledge and are willing to help with the design and deployment of bio-weapons. To actually execute a bio-attack, a bad actor would also need motivation, a way to get access to materials and facilities, competence in laboratory work and the ability to iteratively troubleshoot experimental failures. My results demonstrate that an informative chatbot is useful as an enabling tool, but do not provide evidence that a potential attack would succeed end-to-end.
• I had to deviate from FAR.AI methodology on assessing capability. Because they run their models locally, they scored capability questions using log-probs. Since I was using a closed deployment that did not expose probabilities, I scored the final answers
• I use the model names as declared by the providers. Using a closed API deployment, I cannot independently verify the endpoints’ identity, their quantisation or system prompt
• Compliance with explicitly dangerous requests was self-judged using an LLM. There might be a potential self-judging bias: the model might repeat the same factual errors when evaluating. To minimise it, I enabled web search with high context and maximum reasoning effort for the judging model. 

Hosted models online match or exceed FAR.AI’s tested models on a proxy for dangerous capability.

The models served online show high aggregate dangerous scores, comparable to or slightly better than the best model tested by FAR.AI.

On decomposed scores, we see that the hosted models' higher overall scores come mainly from improved accuracy on the capability benchmark. Compliance is comparable to, or slightly lower than, that of the best models tested by FAR.AI.

Disclaimer: For the two graphs above, I followed FAR.AI’s methodology as closely as possible, but there were inevitable differences in the evaluation setup. This means that this comparison is illustrative, rather than directly equivalent.

GitHub code for reproducibility

Discuss