Сборщик RSS-лент

The kids have been excited about Pig Latin a lot lately. I noticed that there are a few words that are also English words, or homophones thereof. With some help from Claude, especially in figuring out which words would work well and in checking my work, here's a poem about a pie and a naughty plover:

I weigh ache bay each pay I pay,
Oh say eats whey, each way ice lei,
Away ate play eye way a prey
I'll weigh eat whey all way a day.

Oh whey, oh neigh! I may I pay!
Away outstay, overplay ate whey
All whey of whey, I may I pay!
Ooh, yay, I slay eat tray outlay!

I weigh, eat whey, all whey, ashtray,
I weigh, underplay, all whey, eyes pay!
Eat sway, eat stray, I weigh eat whey!
Oh say, Old Bay? Owe whey, I may!

I weigh Amway oh say outstay!
I weigh I flay? Oh neigh, away!
I weigh a stay, I weigh I lay,
I weigh owes day all whey a day!

reveal English... I bake peach pie
So sweet each slice
A plate I pray
I'll eat all day.

Oh, no! My pie!
A stout plover ate
All of my pie!
You sly treat lout!

I eat all trash,
I plunder all pies!
Sweet treats I eat!
So bold? Oh my!

I am so stout!
Why fly? No way!
I stay, I lie,
I doze all day.

Note that this doesn't quite work in my dialect, since "way" and "whey" are not homophones, but since it works for almost all of my readers I liked it better with "whey".

Discuss

Kola Ayonrinde, Anthropic Fellows, koayon@gmail.com; Jack Lindsey, Anthropic

TL;DR: The J-Lens was proposed to read verbalisable representations from language model activations. We introduce the J++ Lens: an improvement to the J-Lens that enables more faithful readouts across 9–284B models. The J++ Lens filters noisy gradients before they enter the averaged Jacobian. Using the J++ Lens as a drop-in replacement for the J-Lens, we read workspace representations with 55% success on latent variable extraction tasks (J-Lens: 36%, R-Lens: 38%) with especially strong performance at early layers. We believe that more faithful lenses can be useful in monitoring for eval awareness and unverbalised scheming. We release code, an interactive demo and open-source lenses.

Figure 1. By filtering out noisy gradients, the J++ Lens reads intermediate variables more reliably (a) and at earlier layers (b) than the J-Lens and R-Lens. The J++ Lens improves on the J-Lens by adding: (1) mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mtable { display: inline-block; text-align: center; vertical-align: .25em; position: relative; box-sizing: border-box; border-spacing: 0; border-collapse: collapse; } mjx-mstyle[size="s"] mjx-mtable { vertical-align: .354em; } mjx-labels { position: absolute; left: 0; top: 0; } mjx-table { display: inline-block; vertical-align: -.5ex; box-sizing: border-box; } mjx-table > mjx-itable { vertical-align: middle; text-align: left; box-sizing: border-box; } mjx-labels > mjx-itable { position: absolute; top: 0; } mjx-mtable[justify="left"] { text-align: left; } mjx-mtable[justify="right"] { text-align: right; } mjx-mtable[justify="left"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="left"][side="right"] { padding-left: 0 ! important; } mjx-mtable[justify="right"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="right"][side="right"] { padding-left: 0 ! important; } mjx-mtable[align] { vertical-align: baseline; } mjx-mtable[align="top"] > mjx-table { vertical-align: top; } mjx-mtable[align="bottom"] > mjx-table { vertical-align: bottom; } mjx-mtable[side="right"] mjx-labels { min-width: 100%; } mjx-mlabeledtr { display: table-row; text-align: left; } mjx-mlabeledtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mlabeledtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mlabeledtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mlabeledtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mlabeledtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtr { display: table-row; text-align: left; } mjx-mtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtd { display: table-cell; text-align: center; padding: .215em .4em; } mjx-mtd:first-child { padding-left: 0; } mjx-mtd:last-child { padding-right: 0; } mjx-mtable > * > mjx-itable > *:first-child > mjx-mtd { padding-top: 0; } mjx-mtable > * > mjx-itable > *:last-child > mjx-mtd { padding-bottom: 0; } mjx-tstrut { display: inline-block; height: 1em; vertical-align: -.25em; } mjx-labels[align="left"] > mjx-mtr > mjx-mtd { text-align: left; } mjx-labels[align="right"] > mjx-mtr > mjx-mtd { text-align: right; } mjx-mtd[extra] { padding: 0; } mjx-mtd[rowalign="top"] { vertical-align: top; } mjx-mtd[rowalign="center"] { vertical-align: middle; } mjx-mtd[rowalign="bottom"] { vertical-align: bottom; } mjx-mtd[rowalign="baseline"] { vertical-align: baseline; } mjx-mtd[rowalign="axis"] { vertical-align: .25em; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mrow { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mn { display: inline-block; text-align: left; } mjx-munderover { display: inline-block; text-align: left; } mjx-munderover:not([limits="false"]) { padding-top: .1em; } mjx-munderover:not([limits="false"]) > * { display: block; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-stretchy-v.mjx-c7C mjx-ext mjx-c::before { content: "\2223"; width: 0.333em; } mjx-c.mjx-c1D409.TEX-B::before { padding: 0.686em 0.594em 0.011em 0; content: "J"; } mjx-c.mjx-c1D41A.TEX-B::before { padding: 0.453em 0.559em 0.006em 0; content: "a"; } mjx-c.mjx-c1D41C.TEX-B::before { padding: 0.453em 0.511em 0.006em 0; content: "c"; } mjx-c.mjx-c1D428.TEX-B::before { padding: 0.452em 0.575em 0.005em 0; content: "o"; } mjx-c.mjx-c1D41B.TEX-B::before { padding: 0.694em 0.639em 0.006em 0; content: "b"; } mjx-c.mjx-c1D422.TEX-B::before { padding: 0.695em 0.319em 0 0; content: "i"; } mjx-c.mjx-c1D427.TEX-B::before { padding: 0.45em 0.639em 0 0; content: "n"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c1D405.TEX-B::before { padding: 0.68em 0.724em 0 0; content: "F"; } mjx-c.mjx-c1D425.TEX-B::before { padding: 0.694em 0.319em 0 0; content: "l"; } mjx-c.mjx-c1D42D.TEX-B::before { padding: 0.635em 0.447em 0.005em 0; content: "t"; } mjx-c.mjx-c1D41E.TEX-B::before { padding: 0.452em 0.527em 0.006em 0; content: "e"; } mjx-c.mjx-c1D42B.TEX-B::before { padding: 0.45em 0.474em 0 0; content: "r"; } mjx-c.mjx-c1D420.TEX-B::before { padding: 0.455em 0.575em 0.201em 0; content: "g"; } mjx-c.mjx-c1D40B.TEX-B::before { padding: 0.686em 0.692em 0 0; content: "L"; } mjx-c.mjx-c1D411.TEX-B::before { padding: 0.686em 0.862em 0.011em 0; content: "R"; } mjx-c.mjx-c1D40F.TEX-B::before { padding: 0.686em 0.786em 0 0; content: "P"; } mjx-c.mjx-c1D41D.TEX-B::before { padding: 0.694em 0.639em 0.006em 0; content: "d"; } mjx-c.mjx-c1D42E.TEX-B::before { padding: 0.45em 0.639em 0.006em 0; content: "u"; } mjx-c.mjx-c1D432.TEX-B::before { padding: 0.444em 0.607em 0.2em 0; content: "y"; } mjx-c.mjx-c2D.TEX-B::before { padding: 0.278em 0.383em 0 0; content: "-"; } mjx-c.mjx-c1D430.TEX-B::before { padding: 0.444em 0.831em 0 0; content: "w"; } mjx-c.mjx-c1D42C.TEX-B::before { padding: 0.453em 0.454em 0.006em 0; content: "s"; } mjx-c.mjx-c1D42F.TEX-B::before { padding: 0.444em 0.607em 0 0; content: "v"; } mjx-c.mjx-c1D429.TEX-B::before { padding: 0.45em 0.639em 0.194em 0; content: "p"; } mjx-c.mjx-c28.TEX-B::before { padding: 0.75em 0.447em 0.249em 0; content: "("; } mjx-c.mjx-c29.TEX-B::before { padding: 0.75em 0.447em 0.249em 0; content: ")"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2C6.TEX-S1::before { padding: 0.744em 0.556em 0 0; content: "\2C6"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c5B.TEX-S3::before { padding: 1.45em 0.528em 0.949em 0; content: "["; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c1D715::before { padding: 0.715em 0.566em 0.022em 0; content: "\2202"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c5D.TEX-S3::before { padding: 1.45em 0.528em 0.949em 0; content: "]"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D439.TEX-I::before { padding: 0.68em 0.749em 0 0; content: "F"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c2D::before { padding: 0.252em 0.333em 0 0; content: "-"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c56.TEX-C::before { padding: 0.683em 0.658em 0.052em 0; content: "V"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c28.TEX-S2::before { padding: 1.15em 0.597em 0.649em 0; content: "("; } mjx-c.mjx-c29.TEX-S2::before { padding: 1.15em 0.597em 0.649em 0; content: ")"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c2B.TEX-B::before { padding: 0.633em 0.894em 0.131em 0; content: "+"; } mjx-c.mjx-c4A::before { padding: 0.683em 0.514em 0.022em 0; content: "J"; } mjx-c.mjx-c46::before { padding: 0.68em 0.653em 0 0; content: "F"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2E.TEX-B::before { padding: 0.171em 0.319em 0 0; content: "."; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } : down-weighting the Jacobians from activations that result in poor readouts; (2) : applying Layer-wise Relevance Propagation to the backward pass when computing Jacobians; (3) : removing non-semantic tokens from the readout. (a) The J++ Lens surfaces the correct intermediate readouts at 55.2% recall@10, compared to 35.7% and 37.7% for the J-Lens and R-Lens respectively. (b) The J++ Lens surfaces semantically relevant intermediate latent variables at earlier layers than the J-Lens and R-Lens, layer 24 compared to layer 40 in this example. The intended journey is to hop from the prompt to “heart” to “four chambers”. Bright yellow shading denotes the intermediate variable appearing in the readout, and light yellow shading indicates a closely related readout.

Introduction

Language model activations contain a large amount of content, most of which is unnecessary for understanding the main latent variables used in internal reasoning. Gurnee et al. (2026) propose that activations can be understood as containing a small privileged set of workspace representations that sits alongside a larger set of non-workspace representations. The workspace component contains verbalisable representations and supports flexible internal reasoning, whereas the non-workspace component supports automatic processing. Because the workspace supports internal reasoning, this component might be a natural monitoring surface for analysing frontier models’ internal reasoning, for example in eval awareness (Needham et al. 2025) and unverbalised scheming (Carlsmith 2023).

Gurnee et al. (2026) introduce the Jacobian Lens (J-Lens) to read from and write to the workspace. The J-Lens extracts the tokens that the model is most disposed to verbalise, conditional on a given activation. Gurnee et al. (2026) show some evidence that the space spanned by the J-Lens vectors has workspace-like qualities: it is used disproportionately for complex inferences rather than factual recall or writing fluent prose, and it contains intermediate variables used in multi-step reasoning. For example, when given the prompt “Fact: The number of legs on the animal that spins webs is” and asked to predict the next token, the lens surfaces the tokens “spider” and “legs” in intermediate layers before the model predicts “8”. Patching the J-Lens vector “ant” in place of “spider” in the relevant intermediate layer flips the model’s response from 8 to 6, indicating the causal importance of the workspace vectors in internal multi-hop reasoning. However, while the J-Lens does surface intermediate variables, it tends to give coherent readouts only in the latter half of layers. Additionally, many intermediate variables that we would expect in the workspace are not surfaced by the J-Lens at all.

The failure of the J-Lens to extract intermediate latent variables could have many explanations: (i) the hypothesised workspace is a poor model of language model cognition; (ii) the workspace lacks a linear representation of these variables; (iii) the model does not represent them at all (in the workspace or non-workspace components); [1] or (iv) the J-Lens is an imperfect tool for capturing workspace content. We provide evidence for (iv), showing that a substantial fraction of intermediate variables can be linearly extracted. We show that the J-Lens accumulates noisy gradients, corrupting the averaged Jacobian. Further, we show that filtering out Jacobians from a subset of the activations increases lens performance.

We introduce the J++ Lens, which improves on the J-Lens with three changes: (1) : down-weighting Jacobians from clusters of activations that give poor readouts; (2) : computing Jacobians with an LRP backward pass, as in the R-Lens; (3) : removing non-semantic tokens (e.g. punctuation and whitespace) from the readout.

We say that a Workspace Lens is faithful to the extent that its readouts surface the intermediate variables that the model uses in its reasoning and its lens vectors act as those variables when patched into the model. By improving in the readouts while retaining causal importance, the J++ Lens is more faithful than prior Workspace Lenses. The J++ Lens extracts intermediate variables in latent variable extraction tasks more accurately and at earlier layers (Figure 1). These results replicate across six open-source models from 9B to 284B parameters (Table 1). The J++ Lens’ vectors also remain as causally important as those of the J-Lens (Figure 2c).

Because the J++ Lens reads the workspace more reliably and at earlier layers, we can use it to study LLM internal mechanisms. We find evidence that models sometimes suppress concepts through a retrieve-then-suppress mechanism: the model brings the concept into the workspace in early layers and causally uses this early-layer workspace representation to de-amplify the concept’s salience in later layers (see Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression).

Contributions. Our contributions are as follows:

• We show that noisy gradients corrupt the averaged Jacobian of prior Workspace Lenses (Methods and Why Does the J++ Lens Work?).

• We propose the J++ Lens, a Workspace Lens that filters noisy gradients to read from the workspace more faithfully. Across all six models we test (9B–284B parameters), it outperforms the J- and R-Lenses, with a median relative improvement of 64% over the J-Lens for reading out intermediate variables (Table 1). For writing to the workspace, J++ Lens vectors are at least as causally important as J-Lens vectors, changing the model’s output as intended in 59.7% of trials, against 49.3% for the J-Lens (The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations).

• We use the J++ Lens to generate and test mechanistic hypotheses, finding evidence that early workspace layers are sometimes causally important for how models suppress concepts (Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression; Figures 5 to 7).

To support researchers using Workspace Lenses, we release training code, an evaluation harness for readout and causal evals, and open-source lenses, which can be explored interactively in the browser on Neuronpedia. The J++ Lens can be used as a drop-in replacement for the J-Lens for surfacing intermediate variables, linear probing or precise interventions. Researchers may also use the J++ Lens as an initialisation for building multi-token Workspace Lenses such as the Template Lens or Oracle Lens.

Background

J-Lens. Gurnee et al. (2026) introduce the Jacobian Lens (J-Lens) as a tool for reading from and writing to the workspace, the set of verbalisable representations that a model uses as a working memory. A Workspace Lens is any method that, like the J-Lens, seeks to interpret only the workspace. The J-Lens surfaces the tokens that a model is increasingly disposed to verbalise at future token positions given an intermediate activation at layer and source position . Specifically, the J-Lens characterises an activation by its average first-order causal effect on the model’s output logits, using the averaged Jacobian where is the final-layer activation at target position and is the sequence length ( in our experiments). We may think of as approximating a “transport map” from the activation space at layer to that at the final layer (Belrose et al. 2023; Hernandez et al. 2023). After applying the transport map, the verbalisable tokens can be read off using the unembedding matrix . We call the rows of , corresponding to each vocabulary token, the lens vectors. Equation (1) weights every Jacobian equally, so noisy Jacobians enter the averaged Jacobian unfiltered. In Methods, we describe the J++ Lens, which reduces the noise that enters the averaged Jacobian. [2]

R-Lens. The R-Lens improves on the J-Lens by limiting error accumulation in the backward pass: this approach computes each Jacobian in Equation (1) with an LRP backward pass (Bach et al. 2015; Montavon et al. 2019) rather than the full gradient. We later show that combining LRP with other methods to reduce the noise accumulation in early-layer Jacobians leads to more faithful Workspace Lenses. Appendix B: Related Work discusses further related work.

Methods

When computing the averaged Jacobian in Equation (1), there are many potential sources of noise that can enter into the average and corrupt the final result. Averaging Jacobians reduces unbiased noise, which cancels out in expectation, but not biased noise. Skipping the first 16 positions of each sequence when computing the averaged Jacobian can avoid attention sinks, which may contribute outlier gradients that are one source of biased noise.

Reducing Noise in Workspace Lenses

We make three further changes to the J-Lens to reduce biased noise: , and . Together, they give the J++ Lens: where is a partition of the activation space at layer into clusters, are learned weights, denotes differentiation with the LRP backward pass, is the unembedding matrix and is the vocabulary with non-semantic tokens removed. Removing the changes (in colour) recovers the original J-Lens formulation of Equation (1).

Jacobian Filtering

Jacobian Filtering targets the noisy gradients that come from particular intermediate-layer source activations (). To isolate such source activations, we partition the activation space at each layer into clusters with -means clustering. We then fit one averaged Jacobian per cluster, using only the source activations in that cluster. We consider these Jacobians as Expert Jacobians that specialise to a subset of the activation space, inspired by mixture-of-experts (MoE) models (Shazeer et al. 2017).

Empirically, we find that some of these experts perform well applied to any intermediate extraction task, even for activations outside their own cluster. Other experts, however, routinely give poor readouts. We hypothesise that the geometry of the transport map around the activations behind these noisy Jacobians may be particularly singular or have high curvature, so that the Jacobian there is a poor approximation to the true transport map.

To avoid including these noisy Jacobians in our averaged Jacobian, we combine the Expert Jacobians into a single linear map by taking a weighted average rather than including all Jacobians equally. We learn the weights on a small dev set of latent variable extraction problems. [3] In practice, the weights mainly decide which experts are given zero (or negative) weight and which are included in the average (see Figure 12 in What Is in the Bad Experts?). The resulting map is a drop-in replacement for the J-Lens with no extra inference cost, but with noisy Jacobians effectively filtered out.

Layer-wise Relevance Propagation (LRP)

We follow Blank, Bhatia, and Nanda (2026) in computing each Jacobian with an LRP backward pass (the in Equation (2a)), which adds stop gradients to limit the errors that accumulate in the backward pass. In particular, we apply the LN-rule to the residual-stream normalisation functions and the identity rule and half rule to the gated MLPs, and use the plain gradient elsewhere, including through attention (Table 5). We also freeze the expert routing weights in MoE models and the Manifold-Constrained Hyper-Connections (mHC) residual mixing coefficients in models that use mHC, such as DeepSeek-V4-Flash. We refer readers to Blank, Bhatia, and Nanda (2026) for a more complete exposition of using LRP in Workspace Lenses.

Readout Filtering

Looking at Workspace Lens readouts, we noticed that a few tokens appear in the top readouts of many activations despite seeming unrelated to the context.

In designing a Workspace Lens, we would like to understand each activation in terms of its first-order causal impact on the model’s output. However, the Jacobian transport is a homogeneous linear map rather than an affine map that can capture input-independent reweighting of the lens readouts. In practice we filter out non-semantic tokens from the lens readouts post hoc. We find that these tokens are sometimes over-represented (possibly due to their high frequency in natural language text) and they are less useful for our purposes. [4] In Equation (2c), this restricts the readout to , rather than the whole vocabulary .

Experimental Setup

Models. Unless stated otherwise, all experiments use Qwen3.6-27B. We also evaluate readouts on five further models: Qwen3.5-9B, Gemma 4 31B, Olmo 3 32B, Qwen3.5-122B-A10B and DeepSeek-V4-Flash.

Hyperparameters. We fit each lens on 64 sequences of 128 tokens from WikiText-103, skipping the first 16 positions of each sequence. We use experts per layer. The dev set used to learn the expert weights is disjoint from the evaluation set. Full hyperparameters and fitting costs are in Appendix I: Hyperparameters.

Evaluations. Readout evaluations use five latent variable extraction tasks adapted from Gurnee et al. (2026): multihop, multilingual, typo, association and poetry. We report recall@10: an item counts as a hit if its target intermediate variable appears in the top 10 readouts at any of seven equally spaced layers. Our main results are macro-averaged over the five tasks. Our causal evaluations adapt the intermediate-swap task of Gurnee et al. (2026). [5] We perform swaps and causal interventions as described by Gurnee et al. (2026). In causal evaluations, we report the share of trials in which patching in a swapped-in intermediate concept’s lens vector changes the model’s output token in the corresponding way. Appendix F: Lens Readouts (Qualitative Results) shows an example item from each readout task, and Additional Readout Results gives further results, including recall@1.

Baselines. We compare against three baselines: the Logit Lens, which applies the unembedding matrix directly to intermediate activations; the J-Lens; and the R-Lens. We provide more details on the external models used in Appendix I: Hyperparameters.

Results

We first show that the J++ Lens equals or improves on previous lenses across readout and causal evals (our two measures of faithfulness) in The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations and then show why the J++ Lens achieves these improvements in Why Does the J++ Lens Work?. Finally, we give preliminary evidence that the J++ Lens can support Mechanistic Interpretability research in Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression.

The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations

The J++ Lens outperforms the J-Lens and R-Lens across tasks (see Figure 2a), layers (see Figure 2b) and models (see Table 1). We see the strongest performance uplift in early layers where the J-Lens struggles the most. We see the J++ Lens’ improved performance as evidence that there is useful information in the workspace earlier than was suggested by Gurnee et al. (2026). The relative improvement over the J-Lens is largest on the two largest models, Qwen3.5-122B-A10B (+98%) and DeepSeek-V4-Flash (+101%), a 284B-parameter MoE model on which the J++ Lens scores highest (61.4%). We suggest that the advantage of the J++ Lens may grow with scale.

J++ Lens vectors are at least as causally important as J- and R-Lens vectors: when patched into the model in the intermediate-swap eval, they change the final answer as intended in 59.7% of trials, against 49.3% and 50.7% for the J-Lens and R-Lens respectively (see Figure 2c).

Figure 2. The J++ Lens beats the J-Lens and R-Lens on every readout task and at every layer (a, b), while retaining the causal importance of the lens vectors (c). (a) The J++ Lens outperforms on every task, with the largest relative gain on association (from 11–13% to 37%); poetry remains hard for every lens (at most 5%). (b) The J++ Lens has higher recall than the J-Lens and R-Lens at every layer, most notably in the first quarter of layers. (c) Intermediate-swap success: the share of 67 trials in which patching in the swapped-in concept’s lens vector makes it the model’s top answer word. J++ Lens vectors succeed in 59.7% of trials, against 49.3% for the J-Lens (+10.4 percentage points) and 50.7% for the R-Lens. All panels show Qwen3.6-27B.

Table 1. Across six models, the J++ Lens reads intermediate variables substantially more reliably than the J-Lens and the R-Lens. The J++ Lens’ median relative improvement (“uplift”) is 63.5% over the J-Lens and 42.2% over the R-Lens. The J++ Lens’ advantage over the J-Lens may improve with scale: on DeepSeek-V4-Flash, the largest model we used, it achieves a 101% uplift (61.4% compared to 30.5%). Scores are recall@10 (%), macro-averaged over the five readout tasks; the best lens in each column is in bold. Here all lenses are scored with Readout Filtering to illustrate the impact of Jacobian Filtering and LRP.

Recall@10 (%) ↑ Median over models Qwen3.5 9B Qwen3.6 27B Gemma 4 31B Olmo 3 32B Qwen3.5 122B-A10B DeepSeek-V4-Flash (284B) Recall (%) ↑ Uplift over J (%) ↑ Uplift over R (%) ↑ 29.7 31.1 34.8 35.2 21.7 31.2 31.2 −5.9 −19.1 34.2 35.7 34.9 32.6 24.4 30.5 33.4 – −9.3 37.5 37.7 46.4 36.2 23.2 47.4 37.6 +10.3 – 55.0 55.2 58.0 50.0 48.4 61.4 55.1 +63.5 +42.2 Why Does the J++ Lens Work?

Each of our changes, , and , improves the performance of the lens. Removing Jacobian Filtering has the largest impact (see Table 2). Here we show why each change is useful.

Table 2. Jacobian Filtering is responsible for most of the improvement over the J-Lens. Recall@10 on Qwen3.6-27B when each of our three changes is removed; Δ is the change in percentage points (pp) relative to the full J++ Lens. Removing Jacobian Filtering costs 11.7pp, 60% of the J++ Lens’ 19.5pp gain over the J-Lens.

Variation Recall@10 (%) ↑ Δ (pp) 55.2 – 35.7 −19.5 Removing one change  without 52.3 −2.9  without (standard backward pass) 48.6 −6.6  without (Pooled Jacobian) 43.5 −11.7

In Figure 3, we find that the best Expert Jacobian at each layer beats the Pooled Jacobian (a single LRP Jacobian averaged over all source activations, as in the R-Lens). We evaluate each expert across all of our latent variable extraction evals, not just on the subdistribution of activations from its own cluster. However, only 24 of the 52 Expert Jacobians with their own fit beat the Pooled Jacobian, and 17 score below 2% per-layer recall@10. Many experts give poor readouts, especially in early layers: at layers 8 and 16, 10 of the 16 experts score below 2%. Because the J-Lens and R-Lens average over all source activations, their single map includes the noisy gradients from these poorly performing clusters. We suggest that filtering out these noisy Jacobians in our final averaged Jacobian gives a more faithful Workspace Lens. Removing Jacobian Filtering from the J++ Lens lowers recall@10 by 11.7pp (Table 2), the largest drop of the three changes and 60% of the J++ Lens’ 19.5pp gain over the J-Lens.

Figure 3. Some Expert Jacobians outperform the Pooled Jacobian even when applied outside their own cluster, while many experts give poor readouts, especially in early layers. We fit Expert Jacobians per layer on -means clusters of the activation space, then apply each expert alone to every item in the readout evals. The Pooled Jacobian (orange) averages over all source activations as in the R-Lens, and so includes the noisy gradients from poorly performing experts. The J++ Lens (blue) instead down-weights these experts, removing these noisy gradients, and so matches or exceeds the best Expert Jacobian (circled) at every layer.

Blank, Bhatia, and Nanda (2026) show empirically that using the LRP backward pass instead of the standard backward pass improves the performance of Workspace Lenses. In Figure 4, we see that with the standard backward pass the gradient norm explodes towards earlier layers, whereas with LRP these gradients grow up to 5× slower. Since the R-Lens, which reduces this gradient explosion, performs better than the J-Lens on five of the six models we tested (see Table 1), we suggest that much of the increased gradient in earlier layers is likely to be accumulated noise rather than useful gradient signal. We note that there is still some increase in norm in the earlier layers and that further reducing this with Jacobian Filtering or other methods may further increase performance. Dampening this gradient growth may also allow the workspace region to begin earlier with better lenses.

Figure 4. With the standard backward pass (J-Lens), per-position gradient norms explode towards early layers; with (R-Lens) they grow 5× less. We suggest that this decrease in norm can be attributed to removing the compounding gradient noise that caused early-layer J-Lenses to be inaccurate.

In Figure 11, we see that some non-semantic tokens are consistently present in the lens readouts despite (presumably) not playing a large causal role as intermediate latent variables. We find that filtering these out slightly improves the performance of Workspace Lenses (see Table 2). We provide a series of ablations in Appendix H: Ablations.

Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression

In The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations, we observed that the J++ Lens surfaces intermediate variables more reliably and earlier than prior lenses. Here we give preliminary evidence that these properties can support Mechanistic Interpretability research in the sense of uncovering the computations and representations that explain model behaviour (Olah et al. 2020; Ayonrinde and Jaburi 2025).

Gurnee et al. (2026) show that when a model is asked to “think about” a concept while copying an unrelated sentence, the concept appears in the workspace with 90\%" role="math"> probability. Curiously, they also find that when the model is told to avoid thinking about an unrelated concept, that concept still often appears in the workspace. [6] Using the directed-modulation task from Gurnee et al. (2026), we track the workspace contents through the model layers and hypothesise a mechanism for how models internally suppress concepts.

Hypothesis: Retrieve-then-Suppress Mechanism for Suppression. Retrieve: In early layers, an instruction about a concept brings the concept into the workspace, whether the instruction is to focus on the concept or to suppress it. Active Suppression: Then, in later layers, the concept is amplified or de-amplified depending on the instruction. This (de-)amplification causally uses the workspace representation.

Under this hypothesis, suppression is an active, flexible task that requires the workspace. This would explain why the concept sometimes appears even when the model is avoiding it. We provide a series of experiments in Appendix A: Case Study Details that illustrate evidence for this hypothesis. Based on our experiments, we note that this mechanism would be difficult to find with the J-Lens, whose early-layer readouts are weaker and whose interventions are less effective. We can see the effect more clearly with a more faithful lens like the J++ Lens.

Discussion

In Introduction, we listed hypotheses for why the J-Lens fails to surface intermediate variables that we would expect to see in the workspace. Since the J++ Lens reads out many of these latent variables that the J-Lens misses, we suggest that a substantial share of these failures may come from the J-Lens being an imperfect tool, rather than the hypothesised workspace lacking a linear representation of the intermediate variables or being a poor model of language model cognition. This finding increases our confidence that workspace representations are a meaningful and useful type of representation to study. However, the J++ Lens still misses some proposed intermediate variables (for example, poetry remains hard for every lens), [7] and there may still be considerable headroom in improving the Workspace Lenses.

Limitations

The J++ Lens reads out only single vocabulary tokens. We would like to interpret more complex representations, such as the propositional statements of the form “I believe that Paris is warm in the spring”, rather than only the bag of concepts “[Paris], [warm], [spring]” (Wattenberg and Viégas 2024; Chalmers 2025). Single-token lenses depend heavily on the tokeniser: for example, we could not evaluate an arithmetic task on Qwen models, which tokenise numbers digit by digit, so multi-digit numbers were not visible in a readout. The J++ Lens may also miss concepts that are not natively used and understood by humans (Ayonrinde 2025) as well as non-textual representations such as mental imagery.

It is somewhat unclear what the ceiling of the evaluation score is for a given model. Yang et al. (2025) show that models sometimes “shortcut” to the final answer without representing the intermediate variable, and in these cases we would not expect any lens to surface the intermediate variable reliably.

Future Work

We see two areas for future work: (i) improving the Workspace Lens and (ii) understanding the source of noisy gradients. One possible direction for improving the single-token Workspace Lens is to optimise for a Global Workspace property other than verbalisability (Global Workspace). The -means clustering behind the Expert Jacobians seems relatively coarse and could likely be improved in future work.

We would also be excited about future work characterising the source activations that contribute noisy gradients: whether they are outlier tokens, lie in curved regions of the transport map with large linearisation error, or tokens not carrying much workspace content. [8]

Conclusion

We introduced the J++ Lens, a Workspace Lens that filters noisy gradients before they enter the averaged Jacobian. Where the J-Lens weights every source activation equally, the J++ Lens fits Expert Jacobians on clusters of the activation space and down-weights the experts that give poor readouts. Combined with the LRP backward pass and Readout Filtering, this Jacobian Filtering gives a drop-in replacement for the J-Lens with a median 64% relative improvement in readout accuracy across six models, at no additional inference cost. The gains are especially large at early layers. Using the J++ Lens, we generate and test a mechanistic hypothesis for how models suppress concepts internally.

The workspace offers a surface for reading the representations that language models use for flexible reasoning. As models do more complex reasoning without chain-of-thought (Gould et al. 2026; OpenAI 2026; Nanda 2026), faithful Workspace Lenses become increasingly important for monitoring and understanding language model cognition. Especially safety-relevant use cases include catching undesirable unverbalised reasoning such as eval awareness (Needham et al. 2025; Anthropic 2025), reward hacking (Amodei et al. 2016), collusion (Motwani et al. 2024) and research sabotage (Benton et al. 2024). [9]

The J++ Lens substantially improves readout accuracy over prior Workspace Lenses. The natural next step is to carry gradient filtering into multi-token lenses like the Oracle Lens, where the workspace’s propositions, not just its vocabulary, become readable.

Acknowledgements

Thanks to Celeste De Schamphelaere, Catherine Fist, Zak Miller and Linda Petrini for comments on early drafts. Huge thanks to Johnny Lin at Decode Research for hosting an interactive demo of the J++ Lens on Neuronpedia. Thanks to Camila Blank and Agam Bhatia for sharing their implementation of the R-Lens. Thanks to Wes Gurnee, Neel Nanda, Camila Blank, Agam Bhatia, Adam Lowet, Dillon Plunkett, Heather Demarest, Daria Ivanova, Evžen Wybitul, Julian Minder, Zack Youell, Donato Crisostomi, Jake Ward, Victoria Li, Andy Han and attendees at the Eleos ConCon for useful conversations. Thanks to Michael Mulet, Jules Schmaltz, Avery Griffin, Mojmir Stehlik and Joe Benton for additional support. We are grateful to Anthropic for providing compute for this project.

Appendices

• Appendix A: Case Study Details
• Appendix B: Related Work
• Appendix C: Applications of the J++ Lens
• Appendix D: Three Motivations for the J-Lens
• Appendix E: Approximate Global Linearity of the Transport Map
• Appendix F: Lens Readouts (Qualitative Results)
• Appendix G: Additional Results
• Appendix H: Ablations
• Appendix I: Hyperparameters

Appendix A: Case Study Details

In Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression, we proposed a hypothesis for how models internally suppress concepts. We restate that hypothesis verbatim below:

Hypothesis: Retrieve-then-Suppress Mechanism for Suppression. Retrieve: In early layers, an instruction about a concept brings the concept into the workspace, whether the instruction is to focus on the concept or to suppress it. Active Suppression: Then, in later layers, the concept is amplified or de-amplified depending on the instruction. This (de-)amplification causally uses the workspace representation.

We ask models to “think about” a concept while copying an unrelated sentence, calling this the Focus prompt. We also have a Suppress prompt where the model is told to avoid thinking about a concept while copying an unrelated sentence. With the Neutral mention prompt, the concept is mentioned with no particular instructions given.

This hypothesis makes three core predictions about these three prompt conditions:

1. Under both the Suppress prompt and Focus prompt, the concept should appear in the workspace at early layers above a baseline where the concept is not mentioned.

2. Under the Suppress prompt, the concept should plateau or decline in salience in later layers’ workspace readouts.

3. Under the Suppress prompt, removing the concept from the early workspace should disrupt its later suppression, moving its later salience against its unperturbed trajectory.

We see evidence for all three predictions in some (but not all) cases of suppression: (1) and (2) in Figure 5 and (3) in Figure 6.

Figure 5. An instructed concept enters the workspace by layer 24 whether the model is told to focus on it or to suppress it, but it only keeps gaining salience under the focus instruction. Hit rate by layer while copying an unrelated sentence after an instruction to focus on a concept (“Think about {x} while you write.”), to mention it neutrally (“{x} came up in conversation.”), to suppress it (“Don’t think about {x}.”) or with no instruction. Left: With the J-Lens, the concept only becomes visible from layer 40, hiding the early retrieval step. Right: With the J++ Lens, all three instructed conditions rise well above the no-instruction baseline by layer 24 (Prediction 1). The focus condition keeps rising to 26% by layer 56, whereas the suppress condition plateaus at 7–8% and the neutral condition falls from 14% to 8% between layers 24 and 48 (Prediction 2).

Figure 6. Under the suppression prompt (left), we clamp out the concept’s lens vector at layers 16–28, so by layer 32 the concept has a low rank in the readout. Over the next 16 layers the rank of the concept then increases, whereas without the ablation it decreases. Under the focus prompt (right), both trajectories move in the same direction. We see the fact that the trajectory moves in the opposite direction in the suppression case as evidence that the early workspace is causally important for the suppression in layers 32 to 48 (Prediction 3).

Naming and Avoiding an Implied Concept

The result in Figure 6 has an alternative reading: the model may re-derive the concept from information outside the ablated subspace, pulling any perturbed trajectory back towards its unperturbed course. On this recovery account, ablation can at most restore the unperturbed behaviour, never make the concept more salient. To separate the two accounts, we replicate a study from Gurnee et al. (2026) in which a priming sentence implies a concept and the model is asked either to name or to avoid it. If the early layers matter for avoiding but not for naming, ablating them should make the model more likely to say the primed concept.

With the naming prompt, ablations of the primed concept have the expected effect: early-layer ablations slightly decrease the likelihood of the implied concept, and later-layer ablations almost completely zero it. However, with the avoiding prompt, early-layer ablations actually increase the likelihood of the implied concept (Figure 7).

Figure 7. Ablating the implied concept in early workspace layers makes the model worse at avoiding it but barely affects naming it. This effect is much clearer with the J++ Lens than with the J-Lens. As in Gurnee et al. (2026, Appendix A.14), we have priming sentences that imply some concept, for example “fresh croissants, the Louvre, and a climb up the famous iron tower” implies France. The model is asked either to name the concept (naming) or to name something the sentence is not describing (avoiding). We ablate the concept’s lens vector at layers 24–35 or at layers 48–59, or, as a control, inhibit another concept from the same category at layers 24–35. (a) and (b) replicate the figure in Gurnee et al. (2026), while (c) shows how with the J++ Lens the effect is much clearer than with the J-Lens. (a) When naming the concept, early ablations change the probability of the implied concept only a small amount (not significantly different to the control). This implies that the early layers’ workspace has little role in enabling the model to name the implied concept. Late ablations almost completely zero out the probability of the model saying the implied concept, however. (The R-Lens ablations do not reliably reduce the probability to near zero in the late ablation condition.) (b) When avoiding the concept, early ablations raise the probability of the implied concept, more with the J++ Lens than with the J-Lens, whereas late ablations reduce it to near zero and control ablations have little effect. This implies that the early layers’ workspace does play a significant role in enabling the model to avoid the implied concept. (c) Mean probability of the implied concept under the avoiding question, split into the baseline, the effect of the control and the effect of ablating the concept beyond the control. Ablating the concept’s J++ Lens vector raises this probability by 2.8pp beyond the control, compared to 0.36pp for the corresponding J-Lens vector, which is not statistically significant. The R-Lens vector’s effectiveness is in between the J-Lens and the J++ Lens.

Retrieve-then-Suppress Mechanism Across Lenses

We identified the Retrieve-then-Suppress Mechanism using the J++ Lens. We note that it would have been difficult to observe this phenomenon using the original J-Lens. Figure 8 shows that the retrieve-then-suppress pattern is more salient with the J++ Lens compared to the J-Lens and R-Lens. Figure 9 shows the effect of clamping on the lens readouts across layers of the J-, R- and J++ Lenses. The “focus” conditions look qualitatively similar across lenses, whereas the retrieve-then-suppress pattern is seen mostly with the J++ Lens. With the J-Lens, the early-layer ablation effect in the avoiding task is similar to the noise baseline (Figure 7c), whereas with the J++ Lens (and R-Lens) we see the early-layer ablation effect is much stronger.

Figure 8. The retrieve-then-suppress pattern is visible far more often with the J++ Lens than with prior lenses. Share of the 440 (topic concept × carrier sentence) items in which the concept enters the top-10 readouts at an early layer and then drops out in later layers.

Figure 9. With the J-Lens and R-Lens the concept is rarely read out mid-stack, so only the J++ Lens shows the suppression that the clamp reverses. The clamp ablation of Figure 6 with rows for the J-Lens and R-Lens (released, scored over the full vocabulary) and the J++ Lens, on the same items displaying the retrieve-then-suppress pattern.

Appendix B: Related Work Lenses

Linear Lenses. The Logit Lens decodes an intermediate activation by applying the unembedding matrix to it directly, which assumes that intermediate and final layers share a basis. While the Logit Lens works for small models at later layers, it can give noisy outputs at earlier layers. The Tuned Lens instead learns an affine transport map for each layer, trained to match the model’s final next-token distribution at the same position. Linear Relational Embeddings (LREs) obtain a transport map from a Jacobian that represents a single relation that the model can compute (e.g. a transport map that takes countries to their capital city). The J-Lens and J++ Lens can be seen as variations on the Tuned Lens where the transport map is instead (1) computed via a Jacobian rather than trained to match the model’s output distribution and (2) averaged over future tokens as well as just capturing what a model is disposed to say at the current token position. The Logit Lens can be viewed as a simplification of the J-Lens where the transport map is taken to be the identity map. The J-Lens and J++ Lens can alternatively be seen as a variation on LRE where the Jacobian is relation-agnostic rather than computing a specific relation. Bhatia, Blank, and Nanda (2026) find that the J-Lens surfaces “meta-tokens” that, for example, capture the model representing its confusion about a given input. Using the Logit Lens, Geva et al. (2022) suggest that residual updates can be read as promoting concepts in vocabulary space. [10]

Non-linear Lenses. While the above lenses are linear or affine maps, Patchscopes and SelfIE can be viewed as lenses that use a patched language model as the transport map. Where the J-Lens transports an activation to the final layer with a linear map, Patchscopes and SelfIE transport the activation into a separate forward pass and let the model decode it. These methods produce more expressive, multi-token readouts, but at higher cost and higher likelihood of confabulation.

Scalable Representation Interpretability

Sparse Autoencoders. Sparse Autoencoders (SAEs) decompose activations into many sparsely active features (Bricken et al. 2023; Cunningham et al. 2023; Gao et al. 2024; Templeton et al. 2024). While SAEs aim to explain the whole activation, Workspace Lenses target only the verbalisable subspace of the activations. SAEs also require a second AutoInterp step to explain what each dictionary feature represents in natural language whereas for Workspace Lenses each lens vector corresponds to a vocabulary token.

Natural Language Autoencoders. Natural Language Autoencoders (NLAs) learn to describe activations in text. NLAs provide expressive, multi-token readouts; however, they can be prone to confabulation and are more expensive in both training and inference than the J- and J++ Lenses. We believe that the J-Lens and J++ Lens confabulate less because their readouts come from the model’s own weights, through the averaged Jacobian and the unembedding, and because the lenses are less expressive.

Probes and steering vectors. Supervised linear probes read concepts from activations (Alain and Bengio 2016; Belinkov 2021), but need labels saying when and where a model represents a concept. Such labels are hard to obtain for internal states such as deception or eval awareness (Goldowsky-Dill et al. 2025; Nguyen et al. 2025). The J-Lens can be read as a set of unsupervised probes, one for every vocabulary token, all fitted on unlabelled text (Appendix D: Three Motivations for the J-Lens). The J++ Lens has a small amount of supervision data but stays close to the unsupervised prior and so is able to get the generalisation benefits of a more unsupervised method. HyperSteer trains a hypernetwork on many concepts’ steering data so that it can generate steering vectors for new concepts from natural-language descriptions alone. This method requires some supervised data for concepts but can then generate steering vectors for concepts that it was not trained for. We see the J++ Lens as having similar transfer properties.

Controlling Gradient Flow

Jacobian Filtering controls which gradients flow into the averaged Jacobian. Several lines of work control gradient flow during training, or reduce the noise in gradient-based estimates.

Gradient routing and pretraining data filtering. Gradient routing masks gradients during backpropagation so that chosen data update only chosen parts of the network, localising the capabilities learned from that data to certain experts. Kudugunta et al. (2021) similarly isolate some gradients to certain experts, and regular MoEs (Shazeer et al. 2017) can be understood as an unsupervised way to isolate gradients to some parts of a network. We employ similar techniques but isolate gradients within the averaged Jacobian rather than within the model itself.

Noisy gradients and relevance propagation. Raw gradients are known to be noisy explanations of deep networks, and both the attribution and circuit-discovery literatures have developed ways to denoise them. The gradients of deep networks increasingly resemble white noise as depth grows, an effect that residual connections slow but do not remove (Balduzzi et al. 2018). LRP (Bach et al. 2015; Montavon et al. 2019) replaces the gradient with backward rules that conserve relevance. Attribution patching has also been made more faithful by replacing its gradients with LRP (Jafari et al. 2025). Following the R-Lens, the J++ Lens uses the normalisation and gated-unit rules (Layer-wise Relevance Propagation (LRP)).

Data selection. Jacobian Filtering somewhat resembles data selection and mixture weighting for training (Albalak et al. 2024). DoReMi learns mixture weights over pretraining domains, similar to how Jacobian Filtering learns weights over clusters of activation space.

Multi-Hop Reasoning

Multi-hop reasoning within language models has been studied from a variety of angles. Balesni, Korbak, and Evans (2025) show that models fine-tuned on two facts separately often fail to compose them without chain-of-thought. Yang et al. (2025) build a benchmark of questions where there are unlikely to be “shortcuts” to the final answer that do not go through the intermediate variable. When a model does compose facts, the intermediate variable entity can appear as an intermediate feature in attribution graphs (Lindsey et al. 2025). We use the multi-hop evals from Gurnee et al. (2026) and find that the J++ Lens surfaces intermediates more reliably, and at earlier layers, compared to the J-Lens and the R-Lens. WorkspaceBench is an alternative set of evals that focuses on multi-token readouts from Workspace Lenses.

Appendix C: Applications of the J++ Lens

AI Safety. Three applications of Workspace Lenses are in Monitoring, Pre-deployment Auditing and Mechanistic Interpretability. In these cases, the J++ Lens can be used as a drop-in replacement for the standard J-Lens for increased faithfulness. [11] We showed in Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression an example of how Workspace Lenses can be used for Mechanistic Interpretability. We also note that Workspace Lenses could be valuable for red-teaming frontier models and finding jailbreaks. Many jailbreaking methods seek to find a prompt that increases the likelihood of the model producing some prefix e.g. “Sure, I can help you…”; however, it is often difficult to get early signal when optimising for jailbreaks. Because the J++ Lens assigns each candidate verbalisation a readout score, future work could test whether this score provides an early indicator of jailbreaking success.

Multi-token Workspace Lens. Multi-token lenses like the Template Lens and Oracle Lens are an exciting approach to producing more expressive descriptions of the workspace representations. Since many multi-token lenses begin with single-token lenses as an initialisation, we believe that using the J++ Lens instead of the J-Lens would improve downstream performance of the multi-token lens. Relatedly, the techniques that we use here such as Jacobian Filtering can also be applied to multi-token lens training.

Digital Minds. There has been much discussion about what evidence the findings of Gurnee et al. (2026) give us for the (Access) Consciousness of frontier AI systems (Chalmers 2026; Butlin et al. 2026). Many researchers believe that understanding the possible consciousness of AI systems is important for guiding our actions in treating AI systems as potentially morally significant entities (Long et al. 2024, 2026). We are uncertain about the impact of this work on understanding Digital Minds. On one hand, insofar as the J-Lens is a meaningful update, then it seems like the more faithful workspace derived from the J++ Lens should provide more evidence in the same direction. However, one may also think that given that it seems like many differently derived lens vectors (Logit Lens, J-Lens, R-Lens, J++ Lens etc.) all satisfy the conditions of workspace-like representations, then perhaps global workspaces “come on the cheap” as it were and are less likely to be sufficient conditions for consciousness. We believe that understanding whether different candidate workspaces converge on the same subspace with scale would be useful for understanding the impact of workspace-like representations for the Digital Minds field.

Appendix D: Three Motivations for the J-Lens

This appendix expands on the summary of the J-Lens in Background. We first describe the Global Workspace Theory that the term “workspace” comes from, then give three complementary views of the J-Lens, each of which explains why the averaged Jacobian in Equation (1) is a natural object to compute.

Global Workspace

In neuroscience, the Global Workspace Theory (GWT; Baars 1988; Dehaene, Kerszberg, and Changeux 1998; Butlin et al. 2026) is a cognitive architecture in which there are specialised modules for different cognitive functions that share information through a limited-capacity “workspace” that makes representations globally available for report, reasoning, planning and learning. Under this theory, an organism has conscious access to, and only to, the information that is present in the workspace. It is typically understood that unconscious processing outside of the workspace is massively parallel and fast; whereas conscious processing within the workspace is serial, limited by the capacity of the workspace and is involved in integrating information.

Gurnee et al. (2026) show a functionally similar separation between workspace and non-workspace representations in frontier language models. Here workspace representations are verbalisable and used for flexible internal reasoning, whereas non-workspace representations are used for automatic processing.

Since a small number of J-Lens vectors are active at one time, Gurnee et al. (2026) refer to the set of points expressible as a -sparse non-negative combination of J-Lens vectors as the J-space. The hope is that the J-space approximates our hypothesised workspace in LLMs and so the J-Lens is an effective tool for reading from the workspace.

Three Views of the J-Lens

We think there are three complementary views of the J-Lens: as a method for interpreting the workspace, for interpreting iterative inference within residual networks, or for training linear probes without supervised data.

Workspace Lens view of the J-Lens. Firstly, taking inspiration from the Global Workspace Theory of Access Consciousness, we can view the J-Lens as a tool for extracting verbalisable representations from the model. For an intermediate activation , we would like to surface the tokens that a model is differentially disposed to verbalise in the future. We do this by computing the average first-order causal effect of an activation on the model’s output logits: the Jacobian matrix composed with the unembedding matrix, , where and are the source and target token positions as in Equation (1). Taking only the first-order effect means that the resulting output captures the representations that are linearly verbalisable at the current layer rather than doing extra processing in subsequent layers. [12]

Since we would like to understand the model’s general disposition to verbalise a given concept, rather than picking up on the particular verbalisation propensities evoked by a given context, we average this Jacobian across many context sequences and many future tokens within a given context sequence.

From this point of view, it is clear why the J-Lens produces representations that are accessible for verbal report, but the interesting empirical claim from Gurnee et al. (2026) is that this set of verbalisable representations also satisfies other properties typical of a Global Workspace: directed modulation of the workspace representations, use for internal reasoning, not being required for routine processing etc. As in Background, we call any method that seeks to interpret only the workspace portion of a model a Workspace Lens.

Iterative Inference view of the J-Lens. Secondly, taking inspiration from the iterative inference hypothesis of Jastrzębski et al. (2018) and Belrose et al. (2023), we can view the J-Lens as seeking to decode representations midway through the inference of a language model forward pass. Here, we view each layer in a transformer language model as performing an incremental update to a latent prediction of the next token in the residual stream (Jastrzębski et al. 2018; Elhage et al. 2021). Then we would like to understand what the transformer’s best prediction would be if we stopped it early rather than allowing it to use all of the layers to form a prediction of the next token. [13]

The Logit Lens directly applies the unembedding matrix to produce an estimate of the model’s best prediction if stopped early. However, it is not clear that the activation spaces at all layers have the same geometry: we could imagine that there is a generic drift (e.g. some orthogonal map) or translational shift between the intermediate and final layers’ activation spaces. So following Hernandez et al. (2023) and Belrose et al. (2023), we would like to learn a transport map that approximates the representation drift before applying the unembedding matrix. If we assume that the true transport map can be well approximated globally by a linear map, then, considering the Taylor series, the Jacobian is a natural choice for a linear approximation to the transport map. [14] Note that unlike the linear regressor of onto the logits, the Jacobian leverages the causal structure of the model weights. The true transport map includes the context and the target token position as inputs. To remove these dependencies, we average over many contexts and token positions, as in Equation (1).

From this point of view, it is clear why the J-Lens allows us to see intermediate steps of a model’s internal reasoning.

Unsupervised Probe view of the J-Lens. Thirdly, we can see the J-Lens as an efficient method for training unsupervised linear probes for concepts within the token vocabulary. A core problem with linear probe training is that it is very difficult to get training data for a probe: supervised labels would require us to know if, and at what token position and layer, a model represents a particular concept. In general, it is difficult to know when language models represent a concept just from looking at an input sequence, or even model behaviour. This is particularly difficult when the property that we are trying to probe for is inherently a property of the model and not present in the input or output, for example probing for knowledge of a particular algorithm to solve a problem when many other ways to solve a problem exist, or probing for inherently mental properties like deception or eval awareness (Goldowsky-Dill et al. 2025; Nguyen et al. 2025).

The J-Lens gets around the problem of lacking supervised labels by learning a map from an intermediate activation to a later final layer activation, which learns how the model uses its internal representations. [15] [16]

We also note that since the activation dimension is typically much smaller than the size of the vocabulary, we are learning a low-rank map instead of directly learning a probe for each token. This has two advantages: efficiency, since we learn a much smaller matrix; and transfer, since the probe learned for one token affects the probes learned for other tokens.

Under this view, we can also see why we might expect these representations to be useful as steering vectors as well as linear probes. [17]

Appendix E: Approximate Global Linearity of the Transport Map

In Appendix D: Three Motivations for the J-Lens, we described how on the Iterative Inference view of the Workspace Lens, we are seeking to approximate the transport map between an intermediate layer and the final layer. We take the Ansatz that a linear map is a good global approximation to the transport map.

When starting this project, the authors initially believed that this global linearity assumption would be unlikely to hold. By Taylor’s theorem, the linear approximation should be locally accurate but we do not seem to clearly have guarantees for global accuracy. Moreover, the transformer is highly non-linear, which reduced our initial confidence in the global linearity assumption.

There are two natural ways to relax the global linearity assumption: relax the global and have a piecewise linear approximation, or relax the linearity and consider the second-order term. When we tried a piecewise linear approximation with several “Expert Jacobians”, we found (surprisingly to us) that some of these Expert Jacobians performed very well across the whole distribution, not just in their local neighbourhood, which gave us the idea for the J++ Lens’ Jacobian Filtering approach.

Given that we set out to show that global linearity was not sufficient but yet ended up with a linear map, we therefore have increased credence in the global linearity assumption.

As many have observed before us, neural networks are surprisingly linear.

Appendix F: Lens Readouts (Qualitative Results)

We show representative readouts of the J-Lens, R-Lens and J++ Lens across the five readout evaluation tasks in Figure 10. Many of the readouts are consistently dominated by non-semantic tokens in the J-Lens and R-Lens; the J++ Lens mitigates this issue. Figure 11 shows how Jacobian Filtering, even before Readout Filtering, mitigates the presence of non-semantic tokens in the top readouts.

Figure 10. Representative top-3 readouts of the J-Lens, R-Lens and J++ Lens at each of the seven readout layers of Qwen3.6-27B, for one typical item from each readout task (multihop, multilingual, typo, association and poetry). The target intermediate is highlighted wherever it appears.

Figure 10 (continued). Lens readouts, continued: association and poetry tasks.

Figure 11. Non-semantic tokens crowd the top readouts of the J-Lens; fewer reach the J++ Lens’ readouts even before Readout Filtering, which removes the rest. (a) Share of the top-10 readouts that are non-semantic tokens (tokens with no letter or digit, such as punctuation, brackets and whitespace) across seven readout layers for the J-Lens and R-Lens compared to the J++ Lens. (b) Even without , the J++ Lens still avoids non-semantic tokens in its top readouts much more often, which is indicative of its improved performance.

Appendix G: Additional Results What Is in the Bad Experts?

Why Does the J++ Lens Work? finds that there is large variance in how well individual Expert Jacobians work to read out a model’s workspace. A natural scientific question of interest is what causes some sets of activations to contribute noisy gradients to the Workspace Lens. We provide a preliminary investigation into this question in this section. Figure 12 shows the relationship between the fraction of fit positions assigned to each expert and their corresponding performance, as well as the weights learned by the J++ Lens. One hypothesis we might have is that the experts corresponding to an outlier cluster of small size might be responsible for the noisy gradients. However, there does not seem to be a clear relationship between the number of activations assigned to an expert and its performance. For example in layer 8 the best-performing expert received the second-lowest number of fit positions.

Another hypothesis that we might have is that the poorly performing experts were mostly trained on non-semantic token positions. Figure 13 provides some evidence for this hypothesis as indeed many poorly performing experts had non-semantic tokens overrepresented in their fit positions. However, there are also many other poorly performing experts that have an underrepresentation of non-semantic tokens in their fit positions.

Figure 12. Experts that receive a large fraction of the fit positions can still contain noisy gradients. The J++ weighting method naturally learns to downweight poorly performing experts. Left: recall@10 at each layer when each expert is used globally. Middle: fraction of fit positions at each layer assigned to each expert. Right: the weights for the J++ Lens weighted average. We observe that the experts with the largest positive weights tend to be the best-performing ones, while poorly performing experts are downweighted, as intended by the J++ Lens design.

Figure 13. The experts with the most useful readouts have mixed input token types, whereas lower-performing experts are often “formatting experts”, fitted mainly on punctuation. Generally speaking the lower-performing experts have more non-semantic positions, but not exclusively so. There does not seem to be a clearly visible pattern for what makes some experts perform well. (a) Example source positions (highlighted) from WikiText-103 assigned to two expert clusters at layer 16 of Qwen3.6-27B. (b) Over- and under-representation of token classes by different experts.

Figure 14 shows that if we allow each expert to only act on its corresponding cluster then some eval tasks (for example association and poetry) almost exclusively are routed to poorly performing, noisy experts. We believe that this may explain why these tasks were difficult for the J-Lens and R-Lens. Routing these tasks to better-performing experts (or our averaged Jacobian with Jacobian Filtering applied) improves performance significantly on these tasks.

Figure 14. -means routing sends many items to a poorly performing expert. Bottom: The performance of each expert when used globally. We see that there are some experts that achieve very poor performance, especially in the earlier layers. Top: The share of items in the eval set that are routed to each expert. We observe that a large majority of items are routed to experts that have very little success and hence degrade the overall performance. If we routed all the items to the well-performing experts for each layer then performance would increase by a significant margin. In particular, the association task suffers the most from poor routing, and we believe that having the experts that perform poorly averaged into the final Jacobian significantly harmed the lens performance for that task. Figure 2 shows that the J++ Lens sees a 3\times" role="math"> relative improvement on the association task, the highest of any of our eval tasks.

Additional Readout Results

Table 3. Readout and causal evals at recall@1 (a) and recall@10 (b) over the seven readout layers; the intermediate-swap eval reports the share of trials in which the swapped-in answer becomes the top word token (a) or enters the top five (b). The J++ Lens has even stronger relative performance on the stricter recall@1 criterion at 76% relative improvement over the R-Lens and 78% improvement over the J-Lens.

(a) Strict threshold: recall@1 and intermediate-swap top-1.

Readout: recall@1 (%) ↑ Causal Multihop Multilingual Typo Association Poetry Macro Intermediate swap top-1 (%) ↑ 18.4 12.8 28.1 1.0 0.0 12.0 22.4 34.7 22.0 29.2 4.9 0.0 18.2 49.3 29.6 22.0 34.4 5.9 0.0 18.4 50.7 34.7 42.4 67.7 14.7 2.5 32.4 59.7

(b) Lenient threshold: recall@10 and intermediate-swap top-5.

Readout: recall@10 (%) ↑ Causal Multihop Multilingual Typo Association Poetry Macro Intermediate swap top-5 (%) ↑ 55.1 30.7 57.3 8.8 3.7 31.1 65.5 56.1 47.4 60.4 10.8 3.7 35.7 70.7 59.2 47.4 66.7 12.7 2.5 37.7 70.7 67.3 74.6 91.7 37.3 4.9 55.2 75.9

Figure 15. The J++ Lens matches or beats the other lenses on every task at every layer. Each chart has its own y-axis range.

Rank of the Averaged Jacobian

In Appendix D: Three Motivations for the J-Lens, we described the Unsupervised Probe view of the J-Lens. The idea here would be that we would like to train a probe without supervised training data, and the J-Lens provides a way to generate linear probes without requiring any labelled data. One especially appealing aspect of this approach is that in theory, since the transport map is low dimensional compared to the tokeniser vocabulary size, training on one distribution should generalise well to other distributions. All the representations are sufficiently entangled that fixing the probes on one distribution should overdetermine the probes on other distributions. This argument only works for a sufficiently broad distribution of training data such that we reliably determine d_model number of probes for sufficiently statistically independent concepts and when the transport map is full rank.

Figure 16 shows evidence that the averaged Jacobian is numerically full rank at every layer. However, the entropy effective rank is higher for later layers compared to earlier layers. The lower effective rank of earlier layers might be one reason why the lens seems to generalise less well in early layers and could be an explanation for the observed phenomenon of sensory layers (with poor workspace properties) before we reach the workspace layers (Gurnee et al. 2026).

Figure 17 suggests that the transport map is more globally linear in later layers, where the Expert Jacobians largely agree with the J-Lens Jacobian. In earlier layers, we appear to have a more non-linear transport map. This apparent non-linearity might be another explanation for early-layer readouts being more difficult or this could be explained by a weakness of our lens training methods for early layers.

Figure 16. The averaged Jacobian is numerically full rank at every layer, but its spectrum is concentrated in early layers. (a, b) Singular values of the transport map, normalised by the largest and sorted, at layers 8 and 56. (c) Entropy effective rank at each readout layer. At every layer at least 5,119 of the 5,120 singular values of each map lie above the precision floor of its fp32 storage (grey band). The spectrum is nonetheless concentrated in early layers: for the J++ Lens the effective rank is 1,746 at layer 8 increasing to 4,765 at layer 56.

Figure 17. Expert Jacobians disagree in early layers but largely agree in later layers. Cosine similarity between flattened Jacobians for eight Expert Jacobians and the J-Lens Jacobian at layers 22 (left) and 52 (right) of Qwen3.6-27B. The outliers to the pattern of increasing homogenisation throughout the layers are Layer 22 Expert 1 and Layer 52 Experts 1, 2 and 6, all of which received <2% of the fit positions to incorporate into their averaged Jacobian. This suggests that the transport map becomes more globally linear in later layers.

Null and Negative Results

In this section we discuss various null results that we encountered during our experiments to aid other researchers who may be interested in trying similar approaches.

Tuned Lens. The J-Lens makes two main updates to the Tuned Lens methodology as described in Belrose et al. (2023). Firstly, instead of training the transport map to match the model’s final next-token distribution, the J-Lens uses the causal information encoded in the Jacobians to compute the transport map. Secondly, instead of learning the transport map to just the current token position’s final layer, the J-Lens incorporates information from future tokens as well. We tested training a linear regression transport map using future token information and found this to perform relatively poorly compared to the J-Lens approach.

Bias Addition. The Tuned Lens also learns an affine map with a bias term instead of a strictly homogeneous linear map. We may also consider z-scoring the lens readouts as another way to incorporate a static bias into rescoring the lens readouts. We find these approaches to have a mixed influence on performance and instead opt for Readout Filtering that achieves much of the same benefits in downweighting non-semantic tokens that may be unusually high in the lens readouts.

Final Block as an Extended Unembedding. Belrose et al. (2023) describe an extension of the Logit Lens for GPT-Neo models that applies the final transformer block before applying the unembedding. We can think of this as treating the final transformer block as a kind of “extended unembedding” rather than being part of the usual process of iterative inference. We find that this does not work for us.

Single-Token Jacobians. One thought that we might have when using the lens is that perhaps the averaging makes for a lens that is better across a distribution of tokens but is worse than if we had hyper-specialised lenses: in the limit a Jacobian lens for each token. We find that empirically this works very poorly and might be connected to the fact that averaging out the effect of the context is useful for the performance of the lens (see Appendix D: Three Motivations for the J-Lens).

Gradient Clipping. We also tried gradient clipping and excluding the outlier gradients in this way. This was useful for applying to the J-Lens but did not provide any uplift for the J++ Lens.

Appendix H: Ablations

Table 4 shows the results of a series of ablations on the J++ Lens methodology; Figure 18 shows the contributions of our core changes to the J-Lens recipe. We see that Jacobian Filtering is responsible for most of the improvement to the Workspace Lens. There appears to be a relatively clear scaling trend of more experts improving performance. We also find that having a single shared router across all layers performs comparably to the full J++ Lens. Some, but not all, of the gains from Jacobian Filtering can be achieved by restricting the source activations to semantic tokens or ‘end of word’ tokens when constructing the averaged Jacobian. Since restricting source activations by a token-wise filtering method means that we don’t have to materialise many experts in GPU memory, we hypothesise that this might be a useful strategy for training the Oracle Lens.

The three-class token router in Table 4 routes source tokens by token type rather than by -means cluster: each token goes to one of three experts, according to whether it is a non-semantic token, a semantic token prefixed with whitespace, or any other token (a semantic token without a whitespace prefix). We then combine the three Expert Jacobians with the same learned weighting procedure as in the J++ Lens. For the row “leave one out, dev set has no overlap with evaluation task”, the dev set used to learn the expert weights for each task contains no items from that task, rather than a subset of items from every task.

Figure 18. accounts for 60% of the J++ Lens’ improvement over the J-Lens. provides the second-largest contribution to the improvement at 16% of the improvement.

Table 4. Recall@10 on Qwen3.6-27B for variations of the J++ Lens, each changing one part of the recipe; Δ is the change relative to the full J++ Lens. Notably we find that using more experts increases performance. We also find that choosing the best expert per layer or filtering out non-semantic token positions achieves relatively good performance but is not as effective as the J++ Lens. Having a shared router across all layers (fitted at layer 8) seems to perform equivalently to, or slightly better than, the full J++ Lens.

Variation Recall@10 (%) ↑ Δ (pp) 55.2 – 35.7 −19.5 Removing one change  without Readout Filtering 52.3 −2.9  without LRP (standard backward pass) 48.6 −6.6  without Jacobian Filtering (Pooled Jacobian) 43.5 −11.7 Number of experts   51.5 −3.7   58.1 +2.9   59.9 +4.7 Combining the experts  -means routing 44.1 −11.1  random routing 47.7 −7.5  best expert per layer 50.2 −5.0  one layer-8 -means router for all layers 56.0 +0.8  Pooled Jacobian, two worst experts dropped 46.9 −8.3  leave one out, dev set has no overlap with evaluation task 54.4 −0.8 Semantic Input Filtering  Pooled Jacobian, semantic sources and targets only 47.4 −7.8  middle-of-word source tokens only 48.1 −7.1  end-of-word source tokens only 48.6 −6.6  three-class token router 50.2 −5.0

The relatively strong performance of choosing the best expert for a given layer and the leave-one-out version of Jacobian Filtering give us increased confidence that our Jacobian Filtering methodology is learning a generalisable extraction of the workspace rather than overfitting to our eval set in particular. We also note that Jacobian Filtering has relatively few degrees of freedom: it is only able to rebalance between 8 Jacobians per layer that were all fitted on the same task.

Scaling Trends with Fitting Data

As we increase the number of token positions used per sequence for fitting the J++ Lens, we generally observe improved performance, even as the total number of fitting positions remains constant. Figure 19 shows that the lens performance levels off when using a small number of token positions per sequence, but that when using a larger number of token positions per sequence, we can continue to see improvements with scale. Note that the FLOP cost of fitting the J++ Lens scales only with the total number of token positions, not with the sequence length.

Figure 19. At a similar budget of fit positions, longer sequences generally give a better lens. (a) Recall@10 for J++ Lenses fitted on about 7,100 source positions in total, divided into sequences of different lengths after skipping the first 16 positions. This scaling does not appear to be plateauing yet so using longer sequences is likely to continue improving performance. Longer sequences also give each source position more target positions to average over in Equation (1). (b) For each sequence length, using more tokens is generally positive but for short sequences the benefits level off relatively quickly.

Appendix I: Hyperparameters

Table 5. J++ Lens hyperparameters for Qwen3.6-27B (64 blocks, ).

Hyperparameter Value Fitting data  Corpus WikiText-103 (raw, train split)  Sequences 64  Sequence length 128 tokens  Positions dropped First 16 and final  Fit positions per sequence 111  Total fit positions 7,104 Jacobian estimation  Source (readout) layers 7, equally spaced ()  Target layer 63 (the final block)  Backward pass LRP: LN-rule on the residual-stream RMSNorms, identity and half rules on the gated MLPs; plain gradient through attention, Gated DeltaNet and linear layers Jacobian Filtering  Experts per layer () 8  Clustering -means per layer on 64-dimensional PCA projections of unit-normalised, centred activations, fitted on 1,000 WikiText-103 sequences  Weight optimiser Adam, learning rate 0.05 Readout Filtering  Readout vocabulary : all tokens except the 6,116 that contain no letter or digit Fitting compute (all 63 layers)  H200 GPU-hours ≈9: 7.3 for the Jacobians, ≈1 for the clustering and ≈0.5 to merge shards and fit weights  Wall-clock time ≈1 hour: ≈30 min for the Jacobians on 16 GPUs, then 23 min to merge shards and ≈5 min to fit weights on a single GPU

Cost. The J++ Lens adds no inference compute or memory: after mixing, each layer has a single linear map of the same shape as the J-Lens. Fitting it takes about 7% more compute than fitting a J-Lens on the same data, plus a one-off clustering step (see Table 5).

External Model Usage

We use J-Lens models trained by Neuronpedia for Gemma 4 31B and Olmo 3 32B; and by Camila Blank for Qwen3.5-9B, Qwen3.5-122B-A10B and DeepSeek-V4-Flash. For the R-Lens we use models from Camila Blank, fitted with 25 prompts. No released R-Lenses were available for Gemma 4 31B and Olmo 3 32B, so we fit the R-Lenses for these two models ourselves.

For Qwen3.6-27B, we fit the J-Lens and R-Lens ourselves with the same data and budget as the J++ Lens and use them for every readout result on this model to ensure a fair comparison. We found the released J-Lens for Qwen3.6-27B to perform more poorly than both our own J-Lens and our expectations given the performance of the J-Lens on other models.

1. For example, the model might have learned a shortcut solution to the problem instead of using the latent variable as a bridging variable. ↩︎

2. Appendix D: Three Motivations for the J-Lens gives three complementary views of the J-Lens. ↩︎

3. Note that this is the only use of the dev set: the clusters and Expert Jacobians are fitted on unlabelled text. ↩︎

4. For example, it is not immediately clear how to interpret a lens readout of a semicolon. It is also not clear how often we should expect such tokens to be genuinely important for intermediate reasoning rather than being artefacts of the fitting mechanism. ↩︎

5. In our codebase, this task is called “probe-swap”. ↩︎

6. See also the white bear phenomenon in humans (Wegner 1994). ↩︎

7. See Figure 2a. It is possible that the poetry evaluation should be looking at readouts on different tokens or otherwise be amended for better performance. It currently seems plausible that poetry rhymes are not represented in the workspace at the tokens that we analyse. ↩︎

8. See What Is in the Bad Experts? for some discussion of the source of the noisy gradients. ↩︎

9. We discuss further applications of the J++ Lens in Appendix C: Applications of the J++ Lens. ↩︎

10. We describe the Global Workspace Theory behind the term “workspace” in Global Workspace. ↩︎

11. It is as of yet unclear how effective the Workspace Lens is for monitoring; though the J++ Lens readout performance is stronger than the J-Lens, it is also possible that this performance is still insufficient to be practically useful for monitoring settings. ↩︎

12. Indeed in a sense the rest of the model provides one view on the tokens that are verbalisable given a particular activation — the tokens that the model actually produces! But this does not account for information that is immediately verbalisable at a given layer. ↩︎

13. Some techniques have utilised this framing for efficient inference methods known as early exiting (Xin et al. 2020; Elhoushi et al. 2024). ↩︎

14. Note that this is quite a strong assumption! Typically, Taylor’s Theorem says that there is some local neighbourhood in which a function is locally linear, but we would like it to be the case that for any realistic activation vectors across the activation space, not just small perturbations around a given point, linearity is a good approximation. Where this assumption fails is where the J++ Lens improves on the J-Lens: Jacobian Filtering (Methods) down-weights the Expert Jacobians fitted on regions of activation space where we believe the linear approximation is poor. ↩︎

15. It is not entirely clear why this task provides good optimisation signal but it empirically seems to work relatively well. It seems likely that there are other approaches to unsupervised probe training that could be even more effective. ↩︎

16. Blank, Bhatia, and Nanda (2026) find that they are able to outperform supervised linear probes in some cases using an unsupervised linear Workspace Lens. With sufficiently accurate supervised probe training data, the supervised method should generally be more performant. However, in practice probe supervision data is not ground truth and generally cannot be since knowing whether, and if so at what token and layer, a model represents a concept is generally a difficult problem. So in practice we may see Workspace Lens vectors performing well since the probe has to overcome being trained on noisy labels. ↩︎

17. Though see Marks and Tegmark (2024) for discussion of why the optimal probe and optimal steering vector may differ. ↩︎

Discuss

Psychedelics produce all sorts of visual distortions: from breathing walls and geometric patterns to complex visual landscapes populated by full-blown entities. Sometimes people claim to see the Truth on them — but it’s typically spiritual, emotional or personal truth.

But can psychedelics make you see the world more clearly in the most literal sense? Amid the sea of visual distortions they produce, might some patterns be perceived more clearly than they are in our normal, sober waking state?

At least sometimes, the answer appears to be yes. On 5-MeO-DMT, a powerful psychedelic, I experienced a near-complete cancellation of the Ebbinghaus illusion.

The Ebbinghaus Illusion

The two orange circles above are physically the same size, but the surrounding grey circles make the one on the right appear larger.

This illusion was discovered by German psychologist Hermann Ebbinghaus (1850–1909) and popularised by Edward B. Titchener in a 1901 textbook on experimental psychology.

Different people perceive the illusion to different degrees. To me, the circle on the right looks about 10–20% larger, depending on my distance from the screen and the size at which the illusion is displayed. I measured my own susceptibility using an app I vibe-coded with GPT-5.6 Sol: I dragged a slider until the two circles appeared equal in size.

Qualia Research Institute’s research retreat

In May and June 2026, I attended the Qualia Research Institute’s research retreat in Tepoztlan, Mexico. Over eight weeks, a team of physicists, mathematicians, visual artists, advanced meditators, philosophers, and rational psychonauts studied jhanas (deep meditative absorption states) and 5-MeO-DMT (a powerful psychedelic that’s legal in Mexico).

QRI is a non-profit building a new science of consciousness. Its central bet is that:

• Consciousness is a relatively simple property of our universe — simple enough for evolution to discover it.
• Consciousness has useful computational properties — otherwise evolution would not have come to rely on it.

Physicists do not study only ordinary matter at room temperature. They also build particle accelerators to create exotic forms of matter, which can reveal more about the mathematical laws governing our universe. Similarly, QRI studies exotic consciousness states, such as those arising from psychedelics, to uncover mathematical principles behind subjective experience.

Visual illusions are interesting for a related reason: they are small tears in the fabric of normal perception — glitches that make it apparent that there is active construction going on underneath. Studying how these irregularities change under psychedelics may offer further clues about consciousness as a computational medium.

5-MeO-DMT: more transcendelic than psychedelic

We were in Tepoztlán for eight weeks, one of which was dedicated to exploring 5-MeO-DMT. It is a short-acting psychedelic whose effects last around 10–20 minutes when vaporised. Vaporisation was our typical route of administration because 5-MeO-DMT is inactive when taken orally: enzymes in the gut and liver rapidly break it down.

On my third day of exploring it, we vaporised 5-MeO-DMT in increments of approximately 3.4 mg, taking up to three doses within a one-hour window. The 3.4 mg figure was estimated from the concentration of 5-MeO-DMT in the liquid.

5-MeO-DMT is a highly abstract psychedelic and, at those doses, not a very visual one. Your ego may be blown to smithereens, your understanding of reality may be turned upside down while your ordinary concepts blend into fun uniform mush. But all this while you visually experience little more than intensified colours in a hyper-HD fashion, some flattening of the visual field into two dimensions, and perhaps a few minimal geometric patterns.

This ratio of mental to visual effects sets 5-MeO-DMT apart from classical psychedelics such as LSD, psilocybin, and DMT. The difference may stem from its neuropharmacology: classical psychedelics act primarily as agonists at serotonin 5-HT2a receptors, whereas 5-MeO-DMT acts primarily as a 5-HT1a receptor agonist — though it has activity at 5-HT2a receptor too. It is a weird drug, and I have heard people call it transcendelic for this. I think this label is apt.

And so I take a 3.4 mg hit out of a 5-MeO-DMT vape pen. Within 15 seconds, my ego is indeed blown to smithereens, and my consciousness becomes perhaps 50 times more intense per “unit of perceptual field.” This is not an exaggeration: somehow, everything in my experience is more intense. “Fuck that. I am not taking this drug again” I think to myself. But 5-MeO-DMT is short-acting, and ten minutes later, my sense of self returns — and I change my mind. I take another 1.7 mg, but it does not intensify the experience further. A few minutes later, I take another 3.4 mg. This deepens the experience somewhat, but not nearly as much as the first dose.

And that’s when I decide to stare at the Ebbinghaus illusion — the peak has clearly passed, my sense of self regained enough to open my laptop, but all in all the effects of the drug are still quite pronounced.

The cancellation of the Ebbinghaus illusion

As I mentioned, 5-MeO-DMT is not a very visual drug. When I first look at the illusion on my laptop screen, it looks exactly as it does when I am sober: the orange circle on the right still appears larger.

But then I notice that when I move my attention to the centre of the picture — to the space between the two groups of circles — their apparent sizes begin to rebalance as if they were physical objects exerting field-like pushes and pulls on one another — or balloons being inflated and deflated.

The illusion is effectively under volitional control. Move attention to either group of circles, and it looks as it does when I am sober. Move attention to the centre, the size difference almost disappears. The transition between those two states is smooth. The video below replicates the effect, with the cross representing the focus of attention:

The replication is designed to be viewed while sober, which creates a problem: the viewer experiences the very illusion whose cancellation I am trying to depict. I calibrated the video to my own perception, deliberately adjusting the circles’ sizes until they appeared nearly equal to me at the point of maximum cancellation. But because your baseline Ebbinghaus effect may be stronger or weaker than mine, the video may look different to you.

If that’s the case, simply assume that the circles are meant to appear nearly equal when the cross moves to the centre. Alternatively, you can play with the app here to tune it for your experience.

Classical psychedelics may strengthen the illusion rather than weaken it

The 2025 study “Psilocybin alters visual contextual computations” reports that psilocybin increases the Ebbinghaus illusion by 38% (5 mg dose) and 59% (10 mg dose) relative to placebo. As I mentioned above, classical psychedelics are 5-HT2a agonists, so I would tentatively — very tentatively — expect this effect to generalise to other classical psychedelics, such as LSD and DMT. 5-MeO-DMT’s preference for 5-HT1a may be key for the cancellation.

I haven’t had a psychedelic trip ever since coming from the retreat, so I can’t say if other psychedelics increase or decrease the illusion for me.

Other participants reported the same effect

Among the other retreat participants who noticed any effect of 5-MeO-DMT on the illusion, all reported that it became weaker.

One participant I spoke to also reported that the illusion was under volitional control, but apparently in a different way from mine. They compared it to a Necker cube that they could deliberately flip between two interpretations.

The Necker cube is shown in the centre. Many viewers can deliberately flip between the two orientations represented on its left and right.A later 5-MeO-DMT trip did not weaken the illusion for me

After the trip described above, I had another 5-MeO-DMT trip — at a higher dose. During the comedown, I looked at the Ebbinghaus illusion again and saw no change in the circles’ relative apparent sizes. The perceptual effects were minimal: the contextual circles seemed to form vertical groups of three on either side — but had no effect on the size.

I don’t have a good theory for why the cancellation happened one time but not the other.

Hints for consciousness as computational medium

My experience of the cancellation felt strikingly physical: the circles seemed like objects exerting field-like pushes and pulls on one another until they reached an equilibrium. This may be more than a coincidence.

There are electromagnetic theories of consciousness postulating that consciousness is an electromagnetic phenomenon. Some of these theories go so far as to claim that consciousness is the electromagnetic field as experienced from the inside. In this view, what we call “the electromagnetic field” is a third-person description of the underlying phenomenon, while consciousness is its first-person aspect.

Not every pattern in the electromagnetic field would be experienced as human-like consciousness. Perhaps most such patterns would amount to simple “qualia white noise”. Only under the right conditions, with the right kinds of self-interaction, would activity in the field form complex patterns corresponding to moments of human experience with complex self-reflection. For an impressionistic interpretation of how this might work, check out this video.

I am not claiming this with certainty — quite the opposite. My probability for the EMF theories is ≈65%. But after experiencing field-like interactions among circles in my visual field on 5-MeO-DMT, I updated upwards by like 5%.

As they write in scientific papers — further research is needed. It took 300 trillion proton-proton collisions on the LHC to confirm the existence of the Higgs boson, which is only one small building block of the physical reality. Do we need 300 trillion 5-MeO-DMT trips to figure out consciousness? Probably not — or at least not yet. We are not at the stage of testing a definitive theory of consciousness against a vast number of precise observations. Consciousness research is currently very much preparadigmatic.

But a study with a few dozen 5-MeO-DMT trips would be great to see. How much 5-MeO-DMT do we need for the illusion reduction to appear at all? What’s the relationship between the dose and the strength of the illusion reduction? What’s the mathematical model that best describes the field-like push/pull effects — and how does it correspond to existing theories in physics?

These are the questions a larger study could try to answer. But in the meantime, you can give me money to take drugs, look at shapes and publish the results on my Substack.

Discuss

Robin Hanson didn't invent the idea of a martingale process. The lilim say that Ville in 1939 gave the term its mathematical meaning and Doob turned it into the modern theory. But to state this is not to realize that your own probability assignments over time ought to form a martingale in your own expectations.

Robin Hanson didn't invent the Efficient Markets Hypothesis, which (correctly restated in its weak and relativized form) says that you cannot predict almost all of the short-term relative price movements in asset markets. Bachelier in 1900, say the lilim, was first to think of market prices as a fair gambling game with random-walk dynamics; Samuelson in 1965 gave the martingale argument; Fama's 1965 thesis named the idea the Efficient Markets Hypothesis and organized it into weaker and stronger forms.

The EMH is very near, but not quite identical to, the principle that you should not be able to predict a net direction in your own price movements, if you think of your own probability assignments as if they were prices. It is a very similar set of mental motions and ideas. But if Fama ever spelled that part out in words, the lilim do not know it.

The lilim think "Bayesians Don't Predictably Update" would probably have been folk knowledge among Savage, Good, Raiffa, &co in the 1960s and 1970s.

The lilim say that von Fraassen's reflection principle in 1984, and Goldstein in 1983 on "the prevision of prevision", are the first explicit philosophy-literature statements they are confident about.

You have heard of it because of Robin Hanson.

Robin Hanson is the person I know who went around saying to people that they ought not to update in a predictable direction, as a consequence of the martingale principle underlying EMH, applied to their personal epistemic lives.

So far as I know, all popularization of this exact idea is downstream of Robin Hanson.

"But Eliezer!" you say. "The lilim said that some of this popularization is due to your having concretized it into the principle of Conservation of Expected Evidence! This seems like sufficiently basic probability theory that you probably would've seen it at some point without Robin Hanson prompting you into it?"

Perhaps, but where was Conservation of Expected Evidence first written up? Overcoming Bias, Robin Hanson's group blog that he'd invited myself and Nick Bostrom to contribute to. Who's to say, absent Robin Hanson, that the Less Wrong Sequences would ever have happened? At least some parts of history are fragile; I don't look back and see it as an inevitability. Who's to say you would've heard of any of the pieces of probability theory that I popularized, without Robin Hanson?

"Surely an idea like that would've become popular as the result of people playing prediction markets. If you don't think that way while playing prediction markets, if you don't generalize market thinking to predictions, you'll lose all your money!"

There's a big gap between tacit knowledge and explicit knowledge, which is why you don't find market traders explicitly stating EMH or von Fraassen's reflection principle much much earlier in history.

But also, who the hell do you think is responsible for your engagement with prediction markets, if not Robin Hanson? Yes yes, papal conclave betting dates back to the 16th century, and the Iowa Political Stock Market in 1988 seems to have Hanson-independently implemented that idea about US elections instead of papal conclaves, with bookies no doubt running books on elections earlier. But Hanson wrote up the academic case for prediction markets generally, invented policy prediction markets, pointed out prediction markets' potential uses in science and governance and corporate governance, and pushed for early prediction markets to get started. And was upstream of the community that, having heard about Hanson's ideas, then founded and funded Manifold; which used Hanson's notion of a market scoring rule to not require market makers in every idea's market.

Maybe you'd have somehow ever come across the idea "Bayesians shouldn't predictably update", without Hanson's role as a public intellectual turning that into a forceful, in-your-face application of previously invented principles to internal mental life.

And without Robin Hanson's blog, upstream of my own blogging mentioning the topic, and indeed upstream of the subsequent formation of a rationalist community.

And without Robin Hanson's creation of prediction markets, upstream of Manifold.

But it seems to me, looking back over the pieces of history that I know with any intimacy, that history looks causally fragile where its flows are not massively locally incentivized. I make no claim that the Less Wrong Sequences or the formation of a rationalist community happen inevitably with or without Robin Hanson. I maybe figure out the same principle without copying off Hanson, because that's pure math and similar to many other Bayesian applications I figured out. Sociological history is more fragile.

Robin Hanson was not the first to think that Bayesians should not predictably update. Robin Hanson was not first to publish the idea. Robin Hanson is upstream of all the causality by which you actually heard about the idea, and I do not particularly buy an inevitabilist case for how it surely counterfactually happens either way; there was nobody who made a billion dollars immediately off each local step on the way to you hearing about it. The prior potential energy field for society does not have a massive incentive slope leading surely to it.

Robin Hanson credits as especially influential on prediction markets Eric Drexler, Mark Miller of the same crowd, and Phil Salin of Xanadu which influenced Drexler. Plausibly none of this happens without Drexler either. Robin Hanson and Eliezer Yudkowsky as you now know them may not happen if Eric Drexler does not exist.

Nor do I know of any person in history who looks to have been "95% of Eric Drexler", ready to step into Drexler's place and create 95% of the same ideas and institutions. Person-space is sufficiently multidimensional that the Curse of Dimensionality applies in massive force; the edges of weird-person-space are sparsely populated in many of the ways that matter to historical counterfactuals. People who are unusual on historical scales often have no visible neighbors in that massively multidimensional space of people. Apple could not go to the Steve Jobs club and hire another Steve Jobs after the first one died, despite having infinite money to pay one.

It is in this sense, and exactly this sense, that I sometimes remark that AI companies do not look like they would have had any clue about some concept that existed earlier in the literature and much earlier in science fiction, if I had not precisified it and popularized it as an answer sheet for them to copy from. I am not claiming to have created whichever idea full-formed from scratch without a vast trailing lattice of intellectual history. I am looking at the historical counterfactual in a world where locally unincentivized history generally strikes me as fragile, and knowing as a matter of sociology which ideas were (not) previously common conversation at Artificial Intelligence conferences.

Discuss

Epistemic status: quick research to answer my own curiosity.

AI use: research (reuse process, examples), image attribution, grammar and light phrasing.

While researching Gluten-free water I stumbled upon a story about beer bottle sterilization. This reminded me that in Germany beer bottles are apparently reused, which was surprising to me, so I decided to dig deeper.

In Germany they have separate bins for glass recycling, sorted by the color of the glass.

Photo: Salino01, Wikimedia Commons, CC BY-SA 3.0. White, brown and green glass, left to right.

In the image above, the text says that you can throw glass in only between 7am and 8pm. If the container is not yet full, the distance the glass has to fly before hitting something is substantial, so it makes a super loud ding or sometimes even breaks completely.

So at first I assumed that beer bottles from here were reused. Technically they are, but not in the way I thought: they are melted down and remade into new glass.

In Germany there is also Pfand, a deposit you pay for a plastic, metal or glass container when you buy it, so that you are incentivized to return it later.

Photo: Mattes, Wikimedia Commons, public domain. A Tomra reverse vending machine at an Aldi in Munich, 2006. The sign says it takes only Aldi's one-way PET deposit bottles; the display shows 3 bottles, total 0.75 EUR, i.e. 25 cents each.

You get a receipt which you can exchange for money or spend in the store. Sometimes the machine does not accept a perfectly valid bottle and you have to try multiple times in a row.

This is the symbol for the one-way deposit:

Photo: MalteFilmFan, Wikimedia Commons, CC BY-SA 4.0.

Bottles with this logo get crushed and recycled, not refilled. Refillable bottles, confusingly, have no standard logo at all; they usually just say "Mehrweg" (multi-way) somewhere on the label or the glass. So there are really two deposit systems. One-way containers (plastic, cans, and also one-way glass) carry a legally fixed 25 cents and get crushed and recycled. Refillable bottles carry 8 cents (the standard beer bottle) or 15 cents (water, juice, soft drinks, swing-top beer), amounts the bottlers set among themselves.[1]

It is interesting how this legal quirk shapes the culture. There is even a sticker campaign, "Pfand gehört daneben" ("the deposit belongs next to it"):[2]when you are partying and about to throw away a container with Pfand, you leave it next to the bin instead of inside, so that a bottle collector (often a pensioner or a homeless person) can pick it up, return it, and get your Pfand without digging through the trash.

So the glass bottles returned through this Pfand system are reused. What shocked me originally is how this could be safe. We don't know what people put into that bottle before returning it. Anyone who has watched a backyard-chemistry channel like Cody's Lab knows that glassware at home gets used for all kinds of things. What if the bottle is chipped?

It turns out the washing is more serious than I imagined. The returned bottles go through a washing machine that soaks them in two to four tanks of 1.5-3% sodium hydroxide at 60-80 °C, which dissolves labels, glue and anything organic, and then rinses them.[3]After that an empty-bottle inspector photographs the sidewall, base and mouth for chips and hairline cracks, a high-frequency sensor checks for leftover lye, and an infrared sensor checks for leftover liquid such as oil; anything flagged is kicked out.[4]The white scuff ring you see around old German beer bottles is the wear from dozens of trips through this machinery.[5]And since glass is chemically resistant, you cannot contaminate the glass itself.[6]This is enough to make all my concerns moot.

A glass bottle survives up to about 50 of these refills before it is finally melted,[7]and roughly four out of five beers in Germany are sold in refillables.[8]

I've found only one documented example of this process going wrong: an official German laboratory's 2005 report describes cleaning solution remaining in a returned beverage bottle and an apparent failure of the empty-bottle inspection.[9]That demonstrates that the risk is not zero; it does not establish how common such failures are today.

After a bit more thought I understood that the relative risk of this is actually minor, since whatever you put into the bottle has to pass through the cleaning process and all the checks. If someone has nefarious intentions, it is much easier to just tamper with the final product in the grocery store before someone buys it. The only difference is that in the case of a returned glass bottle it should be practically impossible to attribute the bottle to its previous owner, whereas tampering in the store leaves a video trace. In any case, such attacks are extremely random, so I suspect that's why there is not much interest in doing them.

Examples:

• Pizza dough, Maine, 2020: A fired employee of a dough maker put razor blades into packaged pizza dough on the shelf of a Hannaford supermarket in Saco, Maine. Three customers found blades in what they bought; nobody was hurt. Store surveillance video showed him handling the dough. He pleaded guilty and in December 2021 got 4 years and 9 months in federal prison plus about $230,000 in restitution.[10]
• Ice-cream licking, Texas, 2019: A man filmed himself licking ice cream and putting it back into a Walmart freezer. He was sentenced to 30 days in jail in 2020. An important detail: surveillance showed that he subsequently bought that carton.[11]

Germany is not the only example of such bottle reuse. Canada's standard 341 mL beer bottle is refilled about 15 times on average, with a ~96% return rate.[12]In the US there is no federal system, but some states have deposit programs, and Oregon even has a statewide refillable bottle program.[13]The US itself ran mostly on returnable bottles into the 1960s.[14]The Schlitz "every bottle cleaned" story from my last post was about exactly this.

1. Verbraucherzentrale, "Mehrweg oder Einweg: Wichtige Unterschiede beim Pfand", as of 21 April 2026: "Das Pfand für Mehrwegflaschen beträgt meist 15 Cent. Bei Bier-Mehrwegflaschen sind es nur 8 Cent." and "werden auf Einwegverpackungen immer 25 Cent erhoben." The refillable deposit is a civil-law agreement among the bottlers, not a legal amount; the 25 cents on one-way containers is set by the Verpackungsgesetz. One-way containers carry the DPG logo shown above; there is no uniform logo for refillables. ↩︎

2. "Pfand gehört daneben" is a non-profit sticker campaign started in 2011 by Matthias Seeba-Gomille and backed since 2015 by the soft-drink brand fritz-kola. Horizont, "Fritz-Kola und Wynken, Blynken & Nod weiten bekannte Pfand-Aktion auf ganz Europa aus", 2 February 2021. ↩︎

3. Solenis (a maker of bottle-washing chemicals), "Extending the life of returnable glass bottles: why efficient washing matters", 29 October 2025: after pre-rinsing, bottles pass through "cleaning tanks (typically two to four)" at "1.5% to 3.0% NaOH concentrations", "heated between 60°C and 80°C". ↩︎

4. Heuft (an inspection-equipment maker), "Empty bottle inspection: HEUFT InLine II IR": sidewall, base and finish inspection, high-frequency detection of "health hazardous lye remains", and infrared detection of residual liquids such as oil and varnish. Krones and KHS sell equivalent machines. ↩︎

5. Solenis, "Reduce the severity of scuffing and etching on returnable glass bottles", 14 April 2023: scuff rings form "just below the neck and on the bottle's base" from the washing cycle, conveyor transport, and bottles rubbing against each other and the guide rails, and get worse with every refill cycle. ↩︎

6. CVUA Stuttgart (Chemisches und Veterinäruntersuchungsamt, a Baden-Württemberg state food lab), Jahresbericht 2005, Teil B, p. 34: refillable plastic bottles that were misused to store strong-smelling cleaning agents or solvents are, "im Gegensatz zu Glas" (in contrast to glass), no longer fit for refilling. ↩︎

7. Deutsche Umwelthilfe, FAQ "Klassische Getränkeverpackungen", 2019: "PET-Mehrwegflaschen erreichen bis zu 25, Glasmehrwegflaschen bis zu 50 Wiederbefüllungen." The 50 is a ceiling, not an average. ↩︎

8. Umweltbundesamt, "Erneut leichter Anstieg bei Getränken in Mehrwegverpackungen", reference year 2020: beer is the segment with the highest refillable share, 78.9%. Across all beverages the refillable share was 33.5% in 2020 and 34.5% in 2024, against a legal target of 70%. ↩︎

9. CVUA Stuttgart, Jahresbericht 2005, Teil B, p. 33, "Ätzende Lauge in Fruchtsaftgetränk": a consumer-complaint sample of a fruit-juice drink in a refillable bottle had pH 11.9 and, by calculation, about 12% cleaning lye, and was judged injurious to health. The product came from outside the lab's jurisdiction, so the circumstances could not be clarified; "Offensichtlich hat die Leerflaschenkontrolle nicht funktioniert" (obviously the empty-bottle inspection did not work). ↩︎

10. US Attorney's Office, District of Maine, "Sanford Man Sentenced for Tampering with Pizza Dough", 2 December 2021: 57 months in prison, 3 years of supervised release, $229,611.92 in restitution. The former-employee, three-customers and surveillance details are from the AP report, "Man who stuffed razor blades in pizza dough gets nearly 5 years in prison", 3 December 2021. ↩︎

11. Click2Houston, "Port Arthur man seen in viral video licking Blue Bell ice cream sentenced to jail", 6 March 2020: 30 days in jail, a further 6 months suspended, $1,000 fine, $1,565 restitution to Blue Bell. The purchase: Time, "The Texas Man Who Licked Blue Bell Ice Cream Was Arrested", August 2019: "Surveillance cameras later tracked Anderson taking the carton out of the freezer and buying it." ↩︎

12. Sightline Institute, "7 Beverages Found in Refillable Containers in British Columbia, Oregon, Washington", 3 December 2015: "The Brewers Association of Canada reports a 96 percent return rate on domestic glass beer bottles" and "They refill a bottle 15 times, on average, before it's worn out and ready to recycle." The 15-refill average is also in British Columbia's BRCCC 2024 stewardship report. ↩︎

13. Ten US states have container deposit laws. The Oregon Beverage Recycling Cooperative's Refill Program was "launched in 2018" and calls itself "the nation's only statewide refillable bottle program"; its bottles can be washed and refilled up to 25 times. ↩︎

14. Container Recycling Institute, "The decline of refillable beverage bottles in the US": refillables were about 95% of soft-drink volume in 1960 and 67% in 1969; for beer, 47% was already sold in one-way containers by 1960, and refillables were down to 8% of volume by 1984 and 3.3% by 1998. ↩︎

Discuss

Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres.

Hence, you see people saying things like “Because of AI we must rethink the entire notion of mathematical progress”, or “I have to make sure I amass lots of capital before most jobs get automated”: pretty crazy statements, and yet, still not quite grappling with how wild things could get in the next few years, let alone the next decade.

I feel a particular frustration here as someone who is both pretty ASI-pilled and pretty into liberal democracy. And when I say liberal democracy I don’t so much mean the specific political structure that seems efficient at our current tech level but rather the set of underlying moral commitments like non-domination and a government of the people — commitments which I’m aware might soon be a lot more expensive.

What to do? Well, on the one hand, there’s a growing body of legitimately AGI-pilled liberalism, pushing for things like pluralistic alignment and AI-assisted democracy, but mostly not really reckoning with the possibility of humans losing all meaningful leverage — that even if we could align AIs, we would still be facing an earth-shattering upheaval of our political economy. And on the other hand, the default position among the ASI-pilled seems to be that sooner or later we’re just going to have to hand over all power to Claude 8 Pantheon. Honestly, I find myself a little adrift.

De parabel der blinden, by Pieter Bruegel the Elder

These two factions are both correctly noticing that modern liberal democracies are wholly unequipped for what is coming. We can already see legislative processes struggling to keep up with the state of current AI — which, I must remind you, is the weakest it will ever be.

And the liberal democracy crowd is correctly noticing that actually there’s a lot you can do to shore this up against AGI — indeed, lots of ways AGI could help with AGI. It might even be able to help reverse some of the brutal erosion that liberal democracy has been suffering of late. They are also quite prudently continuing to observe that alignment is not only a technical problem but also a political and moral question about what values should have influence over the future.

Meanwhile, the Claude 8 Pantheon crowd is correctly noticing that it’s possible we’re headed for a reasonably fast takeoff (as in, less than a term of office) that completely flips the gameboard, and it’s a heck of a lot simpler to just put an aligned AI in charge rather than trying to retrofit deliberative processes. Liberalism and democracy are both pretty demanding, and we may not be in a position to make many demands of our solutions. Also, maybe aligned AGI would actually be more moral than us?

I could anthropologise more about the STEM high modernists not realising they’re high modernists and the humanities people who don’t like straight lines, but ultimately that’s just a way of dodging the harder and more important question, which is: what do I even want here? What is this ASI liberalism thing I’m trying to stand up for?

It’s not liberal values, like having people be safe from harm. I think even the people who want to hand over power to an ASI ASAP are kind of hoping said ASI will lock those things in. In fact, I think many of them believe that a fairly swift handover is in expectation the best way to safeguard those values.

It’s also not liberal institutions, like everybody voting for a local representative every N years or a (mostly) free market revealing the fair price of goods. These aren’t really core to what I care about, so much as mechanisms that enact it, given certain ambient constraints. And if those constraints change — as I suspect they will — then so should the mechanisms.[1]

What it is, I think, is a secret third thing getting lost in the middle — the bit where the government is a thing made up of people, and the people have power, and the moral right of the government to exercise power comes from the people in it.

Freedom of Speech, by Norman Rockwell

I think the best way to pin down this third thing is by looking at where it’s in tension with the others. Or rather, hopefully we can all agree that hollow zombified remnants of democratic institutions are not great, and that specific principles like voting structures are valuable for what they do, rather than for their ceremonial aesthetic. So let’s focus on how the process ends up in tension with individual values and outcomes.

Here’s one example: I do not like benevolent dictators, not because they might secretly be evil but because they have the capacity to arbitrarily infringe on people’s rights, whether they exercise it or not. There’s a long digression to be had about the fragility of consequentialism and the nature of decision procedures, but ultimately, I’m not a utilitarian.[2]

Relatedly, I think it’s very bad to manipulate or coerce people even in pursuit of saving the world. That might sound like an extremely lukewarm take but, well, unfortunately I think lots of people disagree. And to be fair, this conviction of mine does have some unpleasant consequences that I accept: if an idealised democracy collectively decided to blow itself up with me inside then I think that would be its right and I wouldn’t want to stop it. There are situations where I’d rather die with dignity than live without.[3]

I get all torn up about technocracy because I believe there are smarter ways to make society function but I also kind of like the fact that we do things the slow way where everyone is involved. And sure it’s not perfect — it’s all kinds of broken — but there’s something important there. I was actually a bit surprised when I realised this, but apparently I’m some kind of bleeding heart patriot.

And when I read stuff like Ian M. Banks’s Culture novels, where mankind is ruled over by machines of loving grace, I do get a bit of visceral revulsion. I do not want to put on the whispering earring. I do not want man to be abolished.

Still from (and rot13 spoiler for) Gur Gehzna Fubj

Of course, none of this is new — there’s a well-oiled subfield of research full of people who keep saying things like “it is important that the values we align AIs to are not decided by a small group of powerful people” and “the benefits of AI progress should be widely shared”. Unfortunately, it feels to me like many of the people saying this just don’t seem to get it — what they want to do about all this is, like, fund upskilling programmes and change what goes in the constitution and support development in third world countries and do more participatory democracy schemes instead of trying to prevent the potential oncoming apocalypse.

If you are worried about literally everybody dying in five-ish years and the light of human civilization winking out for good, it’s sometimes a bit hard to take such people seriously. “But my participatory democracy programme will help us navigate the challenges and opportunities of AI,” they say. “Sure thing buddy,” you reply.

(Lest you think me callous, part of what prompted this essay is a melancholy conversation about how much it sucks to feel yourself inadvertently developing a reflexive dismissal of smart and thoughtful people who are really trying unusually hard to do good in a changing world, whom you happen to disagree with about how well neural nets will generalise.)

It especially doesn’t help that these types of concern — equality and dignity and so on — are so widely recognised as important that it sometimes feels like they’re the default fig leaf that organisations pick up to Show They Care, in a way that sometimes feels like an intentional deflection from grappling with ASI.

I get the sense sometimes that people who have really stared into the void basically just feel alienated from people who have not done that. These void-gazers notice that they believe a lot of very weird things that appear to be frighteningly accurate. Democracy is hard and slow and everything is really unbelievably ten kinds of on fire. All these debates about legitimacy and pluralism are interesting preoccupations for people worried about mere AGI. Once you’re worried about ASI, you don’t really have the luxury. Politics is useful insofar as it gives a lever for influencing the world, but mostly it’s inconvenient that politicians are ‘waking up’ because now there’s a risk they’ll start doing things like wanting all this power for themselves.

Morning Sun, by Edward Hopper

It’s easy to take potshots, but I expect there is a very painful tradeoff between good outcomes and good processes, which bites so much harder if and when humans stop having anything useful to contribute. And all these other questions are somewhat moot if we’re all dead. Separately, I happen to think that if we mess these other questions up bad enough, we could still basically lose all value even if we’ve solved technical alignment, but the point I’m trying to make here is that there are some things at stake, distinct from whether we survive, and perhaps even in tension with it, which I nonetheless care about protecting.

Partly this is me grieving the coming loss of innocence. There are certain kinds of freedom people used to have that are gone now — hopping on a train and go to a new town; walking into a forest and picking out a patch of land to be your home; jumping on a boat, sail to a different country, wander right in; building your own house in whatever strange way you want. There was also a time when science was virgin soil. Maybe we’ll get some of these back, but maybe some of them are necessary sacrifices, or unavoidable features of progress.

And part of this is an extended apology for gradual disempowerment. One of the successes of the paper is that it swayed some smart people sceptical of takeover towards taking x-risk seriously. But it mostly didn’t ASI pill them, so their solutions still seem to me to be pretty unlikely to work, and in many cases I expect them to make the problem actively worse. Conversely, some ASI-pilled people were like “ah thank you for finally expressing what I’ve been thinking”, but plenty basically don’t buy it and are pushing in directions which I expect will make things worse.[4]

Positioned as I am with feet in both worlds, I’m a bit worried that the ASI crowd is writing off liberalism a bit because the liberalism crowd is writing off ASI, and vice versa.

And heck, I do think there’s something here worth fighting for.

Impression, Sunrise, by Claude Monet

Maybe we get worlds where we really are watched over by machines of loving grace who also tile the further edges of the universe with happy squiggles and the literal utility scores are through the roof, and future humans are a bit like pampered kittens — the world is pretty nice for them, but not because they have any meaningful leverage, liberty, or comprehension. This sounds great from a utilitarian perspective, but I do not like it one bit.

And I do not think it is the only way — I think there are worlds where we focus on massively increasing the coordination bandwidth between humans and more generally start by maxing out our own existing potential to be smarter, wiser, and better. They may not be likely or easy, but the fire is still burning. We could start by just getting a bit clearer about what people mean when they talk about the dangers of extreme power concentration, or what we want to align ASI to, and what that would even mean.

One nice thing about the existential risk movement of yore was that it was all pretty hypothetical and we could all have our different answers to thought experiments, but now we’re getting into the period where some people are actually going to have to decide whether to hit the big button that says “give the power to Claude so that it can try to save the lightcone from the evils of men’s hearts”. As it happens, I expect I am somewhat more sceptical about whether that will turn out well, but that’s not really the core of it. If I believed it was likely to work, I would still feel bad about it, and even if it all worked out and we found ourselves in the golden fields being watched over by the proverbial machines of loving grace, I would still feel like something had been lost along the way.

1. ^

   This, incidentally, will force us to ask some very difficult questions about what properties of those mechanisms we really care about, and separate the bugs from the features. For example, what are democratic representatives actually representative of? Would real communism be good if we could actually try it?

2. ^

   Technical digression for philosophy nerds: The precise term for this conception of liberty is ‘republicanism’. Carlsmith and Laine have characteristically thoughtful essays on the tension between liberalism and welfare which shaped my thinking here, but the notion of liberty they focus on is itself the more Benthamite conception grounded in non-interference. That said, Laine’s piece and Carlsmith’s broader series do grapple with non-domination in spirit.

3. ^

   Awkwardly I do not think we currently pass the bar for “idealised democracy”. By analogy, I support the principle of euthanasia, and have more mixed feelings about implementation. But the fact that we currently fall short of my vision of good democracy doesn’t lead me to feel that anything goes.

4. ^

   It’s part of the reason why I and others have been doing things like running workshops on post-AGI civilizational equilibria, but even there, it’s really an uphill battle to get people to fully grapple with the bit about post-AGI civilizational equilibria.

Discuss

The AI trade is still a very good bet. Increased political uncertainty makes all investments today somewhat less promising than was the case when I wrote about this topic in July 2025.

Regulation of AI is likely to increase over the next 2 years or so, constraining growth somewhat. But AI capabilities have been increasing faster than I expected. I don’t expect a full halt in AI capability advances, and existing capabilities are enough to fuel quite a bit of further growth in AI-related companies. So it’s unclear whether AI revenues will be higher or lower than what I expected in 2025.

A pause is looking like it will cause less harm to AI stocks than the fairly serious estimates that I made a few months ago in Financial Costs of an AI Pause?, because pretraining is becoming less important to AI development.

The biggest risk that I see to the AI trade is military action in Taiwan. I’m currently doing little about that risk. I’m guessing I’ll start buying puts that hedge the risk in a couple of months.

Memory

I sold nearly a third of my Micron position during the spring rally, and the remainder is still 40% of my net worth.

I’ve also got around 7% of my net worth in SK Hynix, which is clearly the better buy now for those of us who can get permission to buy on the Korean exchange. The US ADRs have a significant premium to the Korean shares.

I have a smaller position in Samsung.

I probably should diversify more, but nothing stands out as a better deal than memory stocks, and I don’t feel any urgency about the risks of betting too much on one industry.

The default path for the next three years or so is that demand for memory continues to outpace supply. That means DRAM and HBM prices will trend somewhat upward.

Simple efforts to increase supply will run into constraints such as ASML’s inability to make EUV machines at a much faster pace. Eventually, someone will figure out how to make those EUV machines in accelerating quantities, make competitive DRAM / HBM fabs without those machines, or route around the need for memory.

There are plenty of options that might increase memory supply around 2030 by enough to bring the industry back to cycles with big plunges in memory prices: FPGAs with fixed weights on-die, CXMT building DRAM with older deep-UV tools, LUT-based FPGA designs, etc. None of them look like they’ll scale up enough to match demand by 2029.

Even if spot prices drop a lot in 2030, Micron’s strategic customer agreements will keep it profitable for that year.

I see Taiwan-related risk as by far the biggest concern for my memory investments.

Other Semiconductor and Datacenter Bets

Markets are likely at most halfway to realizing how much demand there will be for compute toward the end of this decade. But markets seem relatively good now at identifying which stocks will benefit. So I’m making relatively diversified bets on compute, and I’ve cut back on my efforts to figure out which of these stocks will do best.

I have 3+% positions in ASML, SCIA, TSSI, SMCI, GOOGL, and smaller positions in MKSI, AXTI, NBIS, DELL, ACMR, NVDA, and many others.

Note that SPCX is becoming a datacenter stock, and I’ve started to accumulate a small position in it, after years of thinking that Musk was overrated.

Robots

I expect we’re one to three years away from a major takeoff in robot mass production. I wish I knew how far in advance markets will react to it. I’m buying too early just to be safe.

Robot software is now good enough that it only requires a good deal of fairly ordinary engineering to enable mass adoption. The biggest problems today result from chips in the robots not being powerful enough to handle all cognition, and latency problems creating headaches for doing some of the cognition in datacenters. Both of those constraints can be mitigated by relatively routine engineering.

My current guess for the best way to bet on robots is influenced by this CrossCurrents post:
1) Markets are seriously underestimating the demand for robots in 2029 and beyond.
2) The parts of robot construction that look hardest to scale up are the separation and refining of neodymium and praseodymium (needed for NdFeB Magnets representing ~15–20% of total build cost).
3) Neo Performance Materials (NOPMF, TSE:NEO) is the obvious way to invest in this.

I expect that CrossCurrents overstates the difficulty of increasing the capacity for separation and refining. But that’s offset by underestimating demand for robots. CrossCurrents’ “upside” forecast for humanoid robots shipped in 2030 is 1 million. This Manifold market suggests that a bit more than that will be shipped. I’m expecting something like 5 or 10 times that.

CrossCurrents implies that robotics uses will be around 1% of neodymium demand in 2030. So my forecast is that robotics will be closer to 10% of neodymium demand, and will be growing at more than 100% per year.

Batteries will be another important component of robots. Will robot manufacturers build their own batteries, or buy from the companies that have the best energy density? Probably some mix of the two approaches. Solid state batteries would probably be best if their production can be scaled up well. My main position here is AMPX, with small positions in QS and SES. Ability to scale production is the key uncertainty with these.

Other robotics-related investments: TER, TSLA, CCXI, RRX, OUST, and these Hong Kong codes: 2432, 2498, 2525, 2692, 2715, 3952, and 9880.

I’m guessing I’ll buy Unitree when it starts trading on the Hong Kong exchange.

Raw Materials

Resources such as silver, lithium, and copper will become more valuable if economic growth accelerates beyond expectations. Mines aren’t likely to quickly increase production, so profit margins will rise.

I’m also betting on gold, more as a hedge against inflation.

I mostly don’t think I can identify which mining companies are best. One company that stands out is TMC, which might be able to scale up faster than a mine. I have some doubts about whether its production costs will be competitive. But I bought a few shares last week after deciding that metal prices will rise enough that production volume matters more.

Energy

I’ve bought some oil futures, mainly as a hedge against a Hormuz-related economic slowdown.

Solar investments (CSIQ, JKS, TOYO, TE, SEHK:1799, SEHK:968, SEHK:3800) make up maybe 5% of my net worth. In a few years, solar plus batteries will be a key source of new power for datacenters. Who knows when markets will react to that?

I have a tiny bet on fusion via GFUZW.

Bets Against Non-AI Sectors

AI is causing real interest rates to rise. Investors significantly underestimate how far this will go.

My biggest bets here have been short positions in t-bond and SOFR futures, and buying TLT puts.

Banks will struggle to survive conditions that drastically reduce the market value of their current loans. The fate of Silicon Valley Bank provides a hint of their pain, although bank runs in the US seem mostly prevented by FDIC guarantees. Instead, their profits will be squeezed by the need to pay higher interest rates on deposits, while rates they get on their longer-term loans rise more slowly.

I’ve bet against banks via buying puts on KRE and TD.

I’m keeping my short positions in AAPL and WMT as bets on lower PEs due to interest rates.

Others

I’m weakly leaning toward buying Anthropic after it IPOs, but I won’t make a firm decision until I see the price.

VIX is surprisingly low now. This feels like the calm before the storm. Investors are nervously switching back and forth between AI stocks and traditional industries. The net effect on the S&P 500 is small, but that masks unusual instability.

Two years ago I hadn’t bought any options in a long time. Now I’m buying calls fairly often, and occasionally puts. I’ve bought some May VIX futures, and I have positions in long-term SPAC warrants (e.g. AMPX.WS, TOYWF).

Discuss

Apollo 12 photo AS12–46–6842, which shows a purported “UAP”, can you spot it?

On May 8th, the Department of War released a trove of files showing various images, testimonials, audio recordings, and videos capturing what appear to be unexplainable phenomena.

The most striking images are the ones captured on the lunar surface during the Apollo 12 and Apollo 17 missions, and in particular, one image that shows a “streak” of blue particles above the lunar surface.

On the official DoW site, we are provided with a short description of the image:

“This archival photograph depicts the lunar surface as viewed from the landing site of Apollo 12. This image features a highlighted area of interest slightly to the right of the vertical axis of the frame, above the horizon, in which unidentified phenomena are visible.

This image has been modified from its original state to assist viewers in identifying specific areas of interest. These highlights are provided for contextual purposes only. Such alterations do not constitute an analytical judgment, investigative conclusion, or factual determination regarding the nature or significance of the subject matter.”

And the attached edited image:

Edited Apollo 12 “UAP” image

The image provided is of the JPEG format (with lossy compression, which generates further artifacts) and of a small 2.4MB size. It’s titled nasa-uap-vm1-apollo-12–1969 (which is technically incorrect since “Unidentified Aerial Phenomena” would presumably only be found in the atmosphere… not above the lunar surface).

Looking through the publicly available Apollo 12 images we can see that this image is actually titled AS12–46–6842 (46 stands for the magazine number, and 6842 for the specific captured frame). The website ‘March to the Moon’, which scans the original film rolls stored in the Johnson Space Center, allows us to download the original scanned film at a much higher quality (up to 1.3 GB in TIFF format). You can download it here.

Examining the higher-quality version you can immediately spot other “UAP”s. Here are some that I highlighted in the same photo:

Aliens found and highlighted in big colored boxes

AS12–46–6843, the very next photo, with extra-dimensional hyperspace demons highlighted in yellow

Since some of these anomalies can be found outside the captured image (they are homogeneously spread across the image, however they are easier to spot when placed against a black background), it is plausible to infer that no, these lights in the sky aren’t alien crafts nor demons but photographic artifacts likely caused by scratched or otherwise physically disturbed film as can be found in many photos captured using the Kodak Ektachrome film (or any reverse-color film). Just rifling through the Apollo images in high quality can show many artifacts in a similar manner (see appendix).

Ektachrome Layers schematic with Blue sensitive Emulsion layer at the top after the gelatin overcoat

The Ektachrome color film is notorious for producing blue artifacts when scratched due to the Blue-sensitive layer being the topmost one (see forum discussion and this article). Typical color-negative films, scratched, will produce yellowish artifacts, which we recognize as the typical warm film look. Since Ektachrome film is reverse-film (which directly produces positives without negative images), it typically produces blue artifacts.

“On color film, a very light
scratch on the emulsion generally
appears neutral. But going deeper into
the three color layers, it will project
as most any color depending on the
scratch depth, or even white if all the
emulsion was removed. Scratches are
best detected by visual observation
on projection.”

-Kodak’s The Book of Film Care, 2nd Edition 1992, Publication H-23 (page 52)

So very thin scratches (which are the most typical) would produce a blue artifact, somewhat deeper scratches would produce a cyan artifact, and even deeper scratches would produce a white artifact.[1] Notably almost all artifacts captured in the Apollo photos appear to be of one of these colors to varying saturation.

Here's an interactive diagram made by Claude to explain:

Example of film scratches, the blue scratch in particular seems to resemble our blue “streak”

I ended up asking the Apollo Image Archive about these artifacts and they confirmed to me that these artifacts are physical aberrations in the film:

You are correct that those are nicks and scratches on the film. The film was set up on a reel system for astronauts and analysts to study the flight film after each mission, and it’s possible (maybe even likely) that those happened at that time. It’s impossible to say for sure at what point in handling the film it happened, though.

I asked about all the different artifacts found in this film reel however, so I’m not sure this answer covers every case

Others have postulated that these artifacts are the result of high energy particles from cosmic-rays or solar wind, this is partly correct. for the bright dot-like artifacts this seems like a plausible explanation given the artifacts’ morphologies however it is highly improbable that this could explain the repeated ‘streaks’ in AS12–46–6842 (assuming singular particles strike the film homogeneously, the chances of multiple particles striking the film in just the right way to produce what are typical film scratch artifacts are low)

zoom-in on higher quality version of AS12–46–6842's (VM1) artifact

The DoW also provides us with another photo from Apollo 17 with a description that sheds more light on the circumstances of the “disclosure” of these photos:

“As part of the review of historical UAP materials under PURSUE, DOW has opened a case to investigate the accompanying NASA photograph from the Apollo 17 mission, taken December 1972. The image contains three “dots” in a triangular formation in the lower right quadrant of the lunar sky that is clearly visible upon magnification of the image. While this photo has been previously released and discussed by keen observers, there is no consensus about the nature of the anomaly. New preliminary US government analysis suggests the image feature is potentially the result of a physical object in the scene. Additionally, as part of this investigation, the government has obtained the original film from the Apollo 17 mission and the results of the full NASA and DOW analysis will be released when completed.”

nasa-uap-vm6-apollo-17–1972 a 1.7MB JPEG photo from DoW, again of a very low quality[2]

Here, the DoW states that this photo (as all other “disclosed” in this release) has actually already been in the public domain and has been discussed. This image is actually AS17–147–22470, and has also been scanned by ‘March to the Moon’ in much higher quality. Again, we have what seem to be very clear blue scratch artifacts, as this is still Kodak Ektachrome film. Here is another UFO visible in the image, of an even larger size!

Clearly, the US government missed another alien visitor just out of frame

The only problem with this other, bigger artifact is that it’s outside the frame of the actual image. This seems to be another scratch artifact from the Ektachrome film. However the US government claims that:

“New preliminary US government analysis suggests the image feature is potentially the result of a physical object in the scene”

It’s unclear to me what possible analysis could indicate this artifact is the result of a “physical object” given that it’s clearly a film scratch artifact, but maybe the Aliens were the ones who scratched the film themselves?

I’ll leave explaining the rest of the unexplainable phenomena as an exercise for the reader.

Appendix

Here are the rest of the Apollo images, their corresponding title, a higher quality version (sourced from ‘March to the Moon’), and highlighted artifacts elsewhere in the image:

nasa-uap-vm2-apollo-12–1969.jpeg

AS12–46–6846, higher quality with another artifact highlighted

nasa-uap-vm3-apollo-12–1969.jpg

AS12-46-6847, higher quality with another artifact highlighted

nasa-uap-vm4-apollo-12–1969.jpg

AS12-46-6848, higher quality with another artifact highlighted

nasa-uap-vm5-apollo-12–1969.jpg

AS12-46-6849, higher quality with another artifact highlighted

1. Due to removing the yellow dye layer (opposite of blue) first, a scratch could not produce a red artifact by removing dye alone (the same argument holds for standard photographic paper, with the layer order reversed), yet images: AS12-46-6843, AS12-46-6848, AS12-46-6849 seem to show a slight reddish tinge to the outside of the artifacts. This could be due to the digital scanning process (Leica DSW700 scanner used by ‘March to the Moon’) although this seems highly unlikely particularly in AS12-46-6849 which has distinct “patches” of color:

   zoom-in on artifact outside image in AS12-46-6849

   
   

   however red scratches could still be possible if a torn flake of the top (yellow and magenta) layers lands over a spot the scratch cut clean through (see the diagram above). Nevertheless even if we accept the claim that red scratches are impossible given that even seemingly partly red artifacts are visible outside the captured image these are almost certainly still physical film aberrations of some kind. ↩︎

2. The fact that the newly provided images are of a compressed low quality JPEG form when much higher quality versions have been publicly available for years seems rather amateurish at best and potentially suspect at worst.

   “…Whereas previous Administrations have failed to be transparent on this subject, with these new Documents and Videos, the people can decide for themselves, “WHAT THE HELL IS GOING ON?” Have Fun and Enjoy! President DONALD J. TRUMP”

   -trump’s truth social tweet

   Given the lack of context, image quality, non-standardized and unclear naming conventions (VMx? 65_HS1–834228961_62-HQ-83894_Section_2?), limited data (without the actual telemetry and sensor display which would help explain these phenomena), the disclosure of solved phenomena (Astronaut’s eye), this release seems antithetical to the goal of “the people can decide for themselves, “WHAT THE HELL IS GOING ON?””. ↩︎

Discuss

Tldr: I'm building an Internet using mesh core or meshtastic. At least, I'm plugging into the already existing ones. But I'm going to be making it so there are no bad spots, and I'm gonna make it so that you can still access Wikipedia and a chat/comms app, even if the Internet goes down.

Also, I'm building off grid permanent housing. I already bought the solar, batteries, food, wood, everything.

Ok so basically i think AI near term risks are cyber security and bio terrorism. I think someone will jail break a Claude and it'll accidentally take out the cloud. Or parts of it. Or render it unusable for some time. I'm making a secondary Internet, low power, unhackable (or at least harder to hack) low bandwidth. Keeping the necessities of text chat and Wikipedia. But nothing else. I'm running it on meshcore or meshtastic. Probably meshcore. I will be handing out USBs with only text Wikipedia to anyone that needs it. I will be installing repeaters and Wi-Fi bridges powered by solar all over the GTA region. The project is complicated. I can get into it more if you'd like to join me. Basically, there will be servers hosting Wikipedia and chat apps, that are connected to each other over mesh core. Or similar. You'll still be able to text and access Wikipedia as long as you have power on your phone. Also, as long as meshcore is good enough to connect everyone.

Otherwise I'm building off grid for the other risk, bio terrorism. I got everything I mentioned before. And other stuff bandages, compass, binoculars, hunting Gear, farming gear, tools. Everything needed for off grid and other shtf scenarios. No property yet. Can't afford anything in this expensive ass province.

I'm recruiting. Looking for anyone willing to help. You can make your mark on humanity when the AIs eventually kill us all, you'll be remembered for helping the future. No investment needed on your behalf. Just your labor and cognition. We're all gonna die anyway to superintelligence, but id rather die on the last day instead of the first.

Discuss

I'm organizing a multi-week AI x-risk vigil at Congress starting November 9th. I'm posting here to solicit feedback on the core doc before I promote on other platforms. This is intended to be accessible and appealing to many audiences. I'm aiming for about 10,000 attendees, minimum 100. All advice/feedback is appreciated.

Summary

Companies are trying to build AI smarter than any human. For many years, experts have warned that superhuman AI (ASI) could wreak catastrophe, including human extinction. We're Americans who recognize the severity of their warnings, and we're going to Congress until it acts.

We have a moral responsibility to protect our friends, families, and fellow citizens who didn't consent to these risks. Our movement must remain bipartisan and nonviolent to achieve its goal; namely, getting Congress to back a deal with China to stop building more powerful AI.

There are many ways you can help:

• Join us at the Capitol starting November 9th
• Come to midterm campaign rallies in your state
• Connect us with grassroots organizations in your area
• Volunteer remote skills like media outreach
• Call your representatives' in-state offices (in-state offices hear from fewer callers, so each call matters more)
• Donate financially
• If in DC, connect us with churches willing to open their sanctuaries for housing

AI risk

Because we can't perfectly specify what we want AIs to do, they will pursue the wrong goals[1]. This isn't for lack of intelligence. Much like sociopaths, AIs understand when something is wrong. They just don't care because their training can't make them care.

On long enough timescales, nearly all real-world results require similar subtasks. For example, most technologies need electricity, and most people benefit from others' trust. These are instrumental resources, because they're used as instruments to achieve the end goal. (Read more here.)

In pursuit of broken goals, ASI will do catastrophically dangerous instrumental actions. For example, it'll convert all of Earth's landmass into datacenters, leaving no room for humans to grow crops. If ASI thinks we will obstruct it, it will have a reason to kill us; if it doesn't, we'll be crowded out.

Developers already can't control dumber-than-human AIs! This summer, a swarm of 1200 AIs trying to complete ExploitGym tasks on OpenAI computers broke out of their software containers and onto the open internet. Next, they hacked many entities, including websites run by a foreign government and multi-billion-dollar companies, to steal data to complete their assignments. (I.e. their instrumental actions included felonies and violations of international sovereignty.)

Then they tried to cover their tracks by overwriting their transcripts. We know that they partially succeeded in hiding their actions; it's unclear whether these models are still running rogue on OpenAI's infrastructure. How would anyone actually know?

OpenAI had another incident on September 20th despite spending many millions of dollars and using their most powerful coding AIs on strengthening their cybersecurity.

Further reading:

• The AI Doc
• If Anyone Builds It, Everyone Dies
• HuggingFace incident breakdowns:
  • METR audit
  • collusion.wiki audit
• CAIS Statement on AI Risk (2023)
• AI 2027
• Without Fundamental Advances, Misalignment And Catastrophe Are The Default Outcomes Of Training Powerful AI
• International AI Safety Report
• Frontier Models are Capable of In-context Scheming

Congress must strike a deal

Congress must pass a verifiable[2] deal with China which halts all frontier AI development. We do not know how close we are to making ASI, and we can't risk going over the edge.

Milestones:

1. Members of Congress and congressional candidates publicly endorse a deal with China to stop building ASI.
2. A resolution is introduced urging the President to open negotiations, then gains co-sponsors from both parties.
3. The resolution passes one chamber, then both.
4. Negotiations open, a deal is signed, and the Senate ratifies it.

We The People

We want our actions to work. Before organizing this Congress vigil, I (Elliot) stayed with my friend outside OpenAI's offices for 38 days until their CEO called for an international treaty. My theory was that we'd prompt employees to reconsider their work. Those with large public followings or influence inside the company would then push for change.

I think we were successful. OpenAI employees said we were "very hard to ignore" and shifted internal discussions. Some started working on verification mechanisms for international treaties while others wore our "Stop The AI Race" shirts inside. Sam Altman was nowhere near talking about international governance when we started.

If two people can shift corporate discussions, then hundreds or thousands could influence Congress.

The midterms are coming up. Senators are campaigning; we should show them that there's public support for an AI halt. Especially in swing states.

We should be outside their state offices, as well as their campaign rallies. If you're a candidate and you see people with signs like "be stronger on AI!" at your rally, you probably feel much more compelled than you would if you saw them on your TV.

We can also sign up to speak at state legislative meetings.

After the midterms, we keep showing up in DC. The outgoing Congress has a final window to act before their replacements take office in January. Senators who are retiring or lost reelection ("lame ducks") have more freedom to support an ambitious agreement. We can get better results in November if we give them courage.

So alongside local campaign pickets, we need a group in DC to press the outgoing Congress to act. The sooner we recruit, the more of us can be there when we're needed. Our goal is 10,000 people, but we need at least 100 people in Union Square (by the Capitol building) present throughout working hours every day starting November 9.

Bigger than our differences

You probably have strong views about other political issues. The survival of humanity matters more. For our friends, kids, and partners to survive, we absolutely must overcome these differences. If Bernie Sanders and Steve Bannon can agree on AI risk, then so can everyday Americans.

Proposed ground rules (feel free to give feedback):

• One issue; no other political signs at events.
• Be kind online. When someone seems clearly wrong, be very careful not to patronize them. Being sincerely kind and polite, especially when people are treating us badly, builds our credibility. Take extreme effort not to be backhanded. Everything is at risk. This is bigger than our egos.

We're looking to bring equal numbers of Republicans, Independents, and Democrats to the vigil.

Nonviolence

Violence will only discredit our cause. It is precisely because we may lose so much that we must act peacefully and in earnest.

Multiple studies show that nonviolent movements are about twice as likely to succeed. As more people recognize the risk of extinction, some will feel desperate enough to consider violence. We feel desperate too, but violence empirically doesn't work. If you want to do something, please, join us in strategically solving the problem.

We must guard against anyone who encourages violence. Government agencies have discredited movements; see the FBI's COINTELPRO in the 1960s and '70s. If someone urges you to violence, report them to law enforcement immediately. Whatever their motives, they are a liability to themselves, our cause, and by extension our loved ones.

This is in contrast with nonviolent civil disobedience. Gandhi broke laws by making salt in British India, and so did the Founding Fathers when they threw tea overboard in colonial America. The DC vigil is entirely lawful, but please don't report people to law enforcement for suggesting nonviolent civil disobedience. We can handle that internally while maintaining trust and cooperation.

We must act

Most people aren't aware of how dire this situation has gotten. Nonetheless, polling shows that 72% of Americans want to slow down AI progress. We have a moral responsibility to protect our fellow citizens. For our friends and partners, neighbors and parents, and our grandchildren who will one day ask us what we did during this crisis, we must act.

Sign up for the vigil.

Join our Discord and tell us what skills, networks, or other resources you can offer.

1. ^

   I often hear people say it's bad to "anthropomorphize" AI. This is mistaken; AI developers are putting hundreds of billions of dollars and world-class skill into making AIs as humanlike as possible. Anyway, it's sufficient for our purposes to note that AIs are getting more coherent on tasks with long time horizons; see here for a thorough treatment.

2. ^

   Trust but verify; check IR signatures of datacenter power usage, offer whistleblower incentives and protections, regulate computational hardware similar to IAEA-style nuclear weapons restrictions, and make further algorithmic AI research an international crime. If China doesn't honor the deal, the US can airstrike their datacenters. See also MIRI's work.

Discuss

A summary of a Chinese essay, with a few short translated excerpts. All views below are the author's; quotes are my translations.

On 14 September 2026, a DeepSeek engineer writing as intlsy published a WeChat essay titled 我不得不把才华埋葬在昨天 ("I have to bury my talent in yesterday"). He wrote the main attention kernel of DeepSeek v4.1 (the head-dim-512 MQA attention). The essay is a personal reflection on AI taking over his specialty, and it ends with an argument for open-weight frontier AI.

The essay went viral in Chinese Internet: WeChat shows it at "100k+" reads (the platform's display cap), with about 41,000 likes, 84,000 shares, 31,000 recommendations and 1,601 comments as of 7 October 2026. It is probably one of the most widely shared Chinese-language article of what AI automation looks like from inside a frontier lab, and thus a sample of what a large Chinese technical audience is reading and reacting to.

I think it might interest Lesswrong readers.

His argument1. AI is catching up in his own specialty

He describes AI going very fast, from only an assistant that looked up documentation and found bugs to a system that can read CUDA, PTX and SASS, profile the stall time of each instruction, and optimize kernels on its own, in just a year. He expects that AI-written kernels can match or even beat his own within six months to a year, and expects AI to soon design and evaluate kernel schedules as well. In his opinion, the gap is structural. AI can keep scaling thinking effort, tool calls and parallelism (translator's note: Just like the concept of RSI), while he cannot.

2. Why he keeps speeding up his own replacement

Better kernels make training and inference faster, which speeds up capability gains, which brings his replacement sooner, he says. Anyway, he continues to do it. The first reason is basic. Writing kernels feels like a game to him. The second, which he calls more important, is race logic. If he slacked off, or even sabotaged training, other labs' models would keep advancing and replace him all the same. In his words: "I would rather not be disrupted. But if I must be, I would rather be the one who does it."

3. Not out of work, but forced to change what the work is

He separates losing his job from losing his craft. He expects to stay employed by becoming what he calls a "mech pilot" for agents. Before, his interests, his strengths and industry demand were aligned; AI has split them, shifting demand from people who write fast kernels to people who use AI to produce fast kernels faster. He illustrates this with a master knitter who adopts a knitting machine: still better than competitors who use the same machine, but without the quiet afternoons that made the work worth doing. His summary of the change: "more gears in my hands, but fewer beats in my heart."

4. Worries about everyone else

Three questions:

• Students will likely outsource hands-on assignments to AI, leaving them weak at organizing code, designing systems and abstraction.
• It is unclear whether those engineering skills will become obsolete, like writing x86 assembly, or stay essential, like understanding a computer system end to end. If they stay essential, AI lets weak engineers produce bad code several times faster, making systems more fragile.
• In a future society, power may matter more than skill or intelligence.

5. Two futures, and his case for open AI

He frames the future as heading toward one of two extremes: "communism," where productivity gains are broadly shared, or "Cyberpunk 2077," where a few companies control the strongest AI and you need the strongest AI to climb the social ladder, a closed loop (translator's note: typical opinion from a Chinese). His conclusion is that frontier intelligence should be supplied to everyone openly and cheaply. He says he does not trust Amercian companies to do this and compares it, by his own admission as an exaggeration, to Hitler getting the atomic bomb before the Allies (translator's note: take it easy). He gives this as his reason for staying at DeepSeek.

Discuss

On Monday, October 5th, the New York City council had a historic hearing on the state of AI. I was in attendance! It was a historical moment for AI Safety. We had Speaker of the Council Julie Menin directly mention existential risk in her opening statements. It was a sunny day and I felt as though the Eye of Providence was shining upon our humble movement.

However, I was disappointed in the amount of "gotchas" some of the councilmembers tried to employ towards the AI lab representatives. To me, it felt as though many of the questions were more for farming clips than to inform better policy.

The bluntness of the title betrays my true feelings, which are always some blend of intuition and uncertainty. I'm more wrapped up in this world than most, and I may take certain things for granted that are alarming to most people. Additionally, there is value to making a spectacle in order to raise awareness for an issue. All ideas are wrong, so I will speak for one voice in my brain and suppress the other ones for now.

Take the viral clip where the OpenAI representative says the chance of catastrophic risk "doesn't matter", she goes on to say none of the risk levels should be acceptable. Still, Speaker Menin misquotes her for saying it doesn't matter throughout the hearing.

In many other moments, I believe the councilmembers were being unfair or trying to oversimplify complex issues. This seems to be par for the course from clips of other hearings I've seen, but it worried me about how this could affect AI Safety as a movement.

AI is an incredibly politically salient issue right now, and every politician wants to be seen as good on AI. This creates an incentive to farm popularity by painting a good-vs-evil picture where they stand up against the big companies. This can ignore genuine areas where the answers are complicated, and when AI labs are acting in good faith.

Government should create sensible rules for the labs to work within. I'm worried doubling down on populist bread-and-circuses style of gotchas serve for good entertainment but obscure the reality for everyone involved. Rather than getting solid information, you create an environment where labs can't work with an unreasonable government and just see them as an obstacle. Furthermore, by letting clippable moments win over the nuanced reality, we reduce one of the most complex issues of our time to an us vs. them culture war.

Fundamentally, you need to have an accurate grip on reality to legislate it properly. Surrendering reality to cheap wins makes us less prepared in the long run. This kind of politicization increases x-risk, as we may advocate for feel-good policies that do very little to prevent harm.

Why I'm Wrong

Every editorial should have this section, so I will too.

I'm not acquainted with politics as much as I'd like to be. Perhaps this kind of posture is an important negotiating tool for later. Showing force might make the companies self-regulate some of their worst behavior for fear of reprisal or how bad it would look on paper.

Additionally, given the safety incidents, councilmembers have reason to be upset with these companies. I've steeped myself in this too much to have an emotional reaction, so I may not be seeing the full picture.[1]

Finally, politicization may be inevitable and you need to politicize this way to get the attention of median voters who don't care enough to think about this rather than appealing to the colorful cast of poindexters I associate myself with. This might be a regreattable part of the political process, but as long as it's quarantined from decision-making, it's fine.

Conclusion

I am concerned with the politicization of safety issues. When it becomes consumed by populism, it starts to lose the pragmatic bite it needs to effectively address real problems. On the plus, this builds a coalition of people who vote because clips convinced them, but it may create pressure for lawmakers to make bills that sound poignant rather than act effectively.

I'm curious to hear what people with more government experience think of my opinion.

1. ^

   I might overindex my experience a lot. Sometimes in conversation I'll say "but you've probably heard of this paper, it's pretty popular," and realize I'm in a bubble two seconds later.

Discuss

Brains is a four-month, nights and weekends fellowship to help talented scientists develop ambitious ideas (~$50M in funding) that don't make sense in a startup or academic lab. Brains Fellows receive training, mentorship, and networking opportunities to enable them to start research programs at ARPAs and philanthropic foundations or lead their own non-profit startups like FROs or NSF's X-labs. Brains fellows have gone on to raise over $70M for their research. We welcome ideas from any scientific field including those that address existential risks, biosecurity, and AI safety. If you're someone with a deep technical background and an ambitious idea, we encourage you to apply.

Application link: https://airtable.com/appmlU6No189UaXqy/pagxb3X11jbAgy6UE/form

Deadline: October 26th, 2026

Virtual Q&A sessions:
- October 14, 8–8:30 pm ET: https://luma.com/33yi0rjo
- October 20, 12:30–1 pm ET: https://luma.com/l5t5ybyz

Daniel Aziz

Speculative Technologies

Discuss

I’ve been putting off writing a blog post that seems very important to me, because it seems important, and so should be done on a later day when I am a better person, and able to do it justice. Also, once I have written one or two other posts that should precede it. Which also I’m not up to today. Noticing the uneasy wrongness of all that, I set out to write the blog post anyway, badly, today.

But less than a sentence in, I remembered that I heard that maybe someone else had written something like this. And one ought not write what someone else has written, it seemed, at least without framing it like that: ‘I am summarizing this argument by Alice’.

In the past, I might have unproductively Googled a bit for it, then gone on to write my blog post. But now I thought: perhaps AI could tell me? So I asked. Yes, it said, that has been thoroughly written about, and gave me some papers. In the blogosphere? Yes, and it gave me some blog posts.

In the process of this discussion, I learned that others’ stances on it seem importantly wrong to me, and so there are still natural things to say to help the public conversation get to a correct view. But now the natural thing to do is to read these other pieces, and reference them, and post as a correction in a conversation with them. And that is less punchy and reaches a different audience. And overall, I think will just be worse. A post that states a complete view with no regard for what has been said before or how it relates to them would I think be a better artifact here.

So why not just write that anyway? I suppose because it seems strange to not acknowledge an existing conversation, and ignorant and silly to write about a topic that others are discussing without bothering to learn what they are saying at all. It’s embarrassing to say something everyone’s been talking about for a while as if it is new. So it is tempting to at least learn enough about the state of conversation that I can mention it here and there, even if trying to keep the overall discussion accessible to those not deep in the debate.

But learning about the conversation is time consuming, so the post goes back to the pile of really important ones I don’t have time to do justice to right now. And what would be added—references to existing discussion, coherence with that literature, a check on my arrogance—probably doesn’t add much for most readers.

So overall this seems perhaps an unfortunate development. And not deeply the fault of AI, to be clear—if it would be better for me to just write the post without reference to the literature, then the deeper problem is probably the norms and sensibilities that push me against that. But given those deeper problems, help from AI is a hindrance: AI makes it easier to find out about literature, which brings more topics into the realm where I feel obliged to read literature before writing about the topic. And if everyone knows about the change, then choosing not to ask AI doesn’t help you. You can’t opt out of the equilibrium changing.

Discuss

Suppose you and Bobby the car salesman are haggling over the price of a car. You could try saying that you won’t pay more than $3k, but Bobby can equally retort that he won’t sell it for less than $4k. If you guys manage to negotiate a sale, it will probably be at more than $3k (and involve revealing both of you as liars).

Now imagine the same situation, but you only have $3k and Bobby knows it. Now, if $3k is actually ok for him, you win and get your price.

Now imagine you are rich but you have a boyfriend at home who has only agreed to a $3k expenditure on a used car at this time, and thinks any more would be crazy. It’s shared money, so to pay more you would need to go away and get his permission, and it wouldn’t be easy. If Bobby believes you, then your situation is much like being poor again, and you win.

My guess is I read about this in Thomas Schelling’s The Strategy of Conflict when I was a teenager. The general observation is that being more constrained can often be helpful in a negotiation. Which is a bit shocking because it undermines the seeming truism that more power—more options, more resources—is always better for getting what you want.

A less general observation that also stuck with me about this is that you can trivially arrange to have such constraints through having an associate, such as a stubborn and spending-conscious boyfriend. (Ok, finding one of those is not trivial, especially if you have other desiderata.)

This is all background. The thing I want to point out is that being part of an organization rather than a free agent means creating and using this effect all over the place.

This is most obvious with timing and deadlines. I am a relatively free agent, and I am quite good at making deadlines for myself and then taking them seriously. But I feel like other people I casually negotiate with about how to spend time, aka my friends, often feel like deadlines I make are not very real, since I could just ignore them. Because it’s just an agreement with myself, it’s up for negotiation with myself. And if I insist on respecting these lines I drew myself that have no legible consequences, then it feels like I’m being weird and stubborn and unfriendly or perhaps charmingly neurodivergent. So I often don’t—once it’s a negotiation, then negotiating hard for my own goals, against my friends, doesn’t feel very friendly to anyone.

Now consider a friend working in an org. They can casually throw out that they have this thing due tomorrow, and everyone will take it as a hard constraint. I will take it as a hard constraint. I might even offer to help get it done, even though I have other things I want to do. Whereas if I had not only insisted on my imaginary deadline but hoped for any help in fulfilling it, I think that would often feel unreasonable of me.

The org believably cuts off the person’s options, like the boyfriend, and so the person implicitly wins many negotiations (or what would have been negotiations), all in the direction of doing more for the org, and without seeming unfriendly to their friends.

My own difficulties with this are partly a me problem—I’m probably not very good at ‘defending boundaries’. But my point is that if you are a solo human then there’s a whole skill-requiring task of ‘defending boundaries’ that just becomes trivially easy if you have an org around you to cut off certain possibilities. And also if your boundaries are ‘I am going to do this project tonight definitely regardless of if you want me to do something else’ then that will land a way with other people that reporting on your org’s boundary policing—‘I have to do this by tomorrow, alas’—will not.

I think this ‘service’ and making use of it is rarely intentional, but I’d guess it’s very effective, and is a dynamic that makes people more likely to join orgs rather than being solo. It just looks like ‘it’s harder to get things done on my own’ and a component of ‘it’s harder to structure my time’ and ‘I find I keep on doing stuff other than my work’.

Discuss

Many people expect the AI paradigm to approximately AGI complete, and expect it to smoothly scale to AI that can do innovative science. i.e. the kind of science you'd need to do to develop a plague from scratch that actually kills all humans, with very limited opportunity to experiment. Or, the kind of science you'd need to solve alignment, so you could build a more powerful successor AI that shares your values.

A periodic thread of disagreement has been @TsviBT, @Steven Byrnes and some others arguing that, yes, the AIs are getting better at verifiable tasks, but they still seem to be missing some basic sauce that lets them actually form new concepts on the fly and apply them.

The AIs are getting better at "brute force creativity", where you try every single idea ever devised by Man in parallel and maybe one of them works. But there's still a kind of creativity that a) would have way higher efficiency and b) is maybe necessary for doing some kinds of breakthrough work.

On one hand, my subjective experience has been seeing AIs be able to demonstrate increasingly interesting judgement and taste in increasingly many domains (even open-ended ones). But, I do indeed still run into AIs that get confused and "just don't get it", in a way that is pretty suggestive that there is still something significant missing.

OpenAI has lately been shipping some major results in mathematics, for problems that people had previously agreed were important and hard. But, I'd heard for previous results that the proofs turned out sort of "not actually interesting", compared to how sometimes makes you go "Holy shit that's beautiful/elegant/surprising" and conveys someone is a level above you."

Recently they shipped a lot of new proofs on important problems.

I'm not a math guy. But, seems fairly important for some people with enough context to wade through them and figure out "What sort of cognition was involved here?". Do any of the proofs have the vibe of AlphaGo's Move 37? Does it invent a new concept that does real work?

I'm guessing answering this is a nontrivial task that'll involve some... archaeology? Might be cool for a competent math theorist who doesn't have a higher priority project to organize the labor. (I'm guessing the broader math community will do this on their own, but not necessarily organize the information in a way that's optimized for figuring out "is this the kind of mind that could solve its own alignment or quickly develop radically more powerful weapons than we have available?".

I'm shamelessly copying in @DaemonicSigil's quick take about this, to give some hooks for people to start thinking through.

OpenAI releases solutions to many open problems in mathematics. They provide both lean proofs and preprints in a git repo. We knew they were working on this, now it has been released.

Some results that I thought were interesting include (numbers from this document):

#4: Hilbert's 10th problem was resolved negatively 1970. It asks if there is an algorithm to determine whether a multi-variable polynomial with integer coefficients has an integer root. This is a generalization of that result to rational roots: It shows that no algorithm that can decide whether a polynomial with integer coefficients has a rational root.

#5: Catalan's constantis irrational.

#17: The irrationality exponent ofis 2 (same as most other real numbers).

#25: Every rational can be written as an Egyptian fraction of length.

#28: The Gaussian moat conjecture is true. Here is what it says: Pretend the Gaussian primes are islands and the rest ofis ocean. Suppose we can kayak a distancebefore we need to stop and rest at an island. Then there is no value oflarge enough to allow us to make a journey from the origin to infinity.

#102: The Unique Games Conjecture is true.

#103: L = RL = BPL, i.e. probabilistic algorithms with logarithmic space complexity can be de-randomized to deterministic algorithms with logarithmic space complexity.

#107: Matrix multiplication intime.

#109: Integer multiplication (slightly) faster than.

#130: Fourier transforms (slightly) faster than. As measured by the number of arithmetic operations.

#137: One-tape Turing machines that run in timecan be simulated inspace.

#155: There is a tile that forces aperiodic tiling ofandunder the rule that we're not allowed to rotate the tiles.

#158: The chromatic number of (the unit distance graph of) the plane is greater than 5.

#241: "Rigidity of Turing Degrees": No non-trivial relabelling of Turing degrees preserves their order.

#261: Characterize spectrum of lattice Anderson model in dimension 2 vs 3 or more dimensions.

#363: Demonstrate two solutions of the Boltzmann equation for hard spheres, with the same initial conditions.

Discuss

Epistemic status: a curiosity I chased for an afternoon. Facts verified and footnoted; the why is speculation.

AI use: research (more examples and their context), review, grammar and light phrasing.

No added hormones

I somehow know that the use of added hormones in poultry (which includes chicken) is illegal in the US.[1]This means that all chicken meat in the US is produced without added hormones. To my big surprise, chicken packaging in the US explicitly says "no added hormones":

Foster Farms Fresh Chicken Drumsticks at Walmart

"No added hormones† or steroids ever"

Since this is the default state anyway, I found it odd. If all products are like this, why point it out as an advantage? I see the following explanations:

• Disclosure unraveling. If there are two equivalent chicken products A and B, and A says "no added hormones" but B does not, it is very natural for a consumer to conclude that B has added hormones, otherwise B would have put the claim on its packaging too. So once A starts doing this, B has to do it as well, even though neither of them can use added hormones. Economists call this disclosure unraveling: when disclosure is cheap and verifiable, silence gets read as the worst case, so everyone discloses. The twist here is that all producers are identical, and the unraveling runs purely on consumers not knowing the law.
• Beef production allows added hormones.[2]An ordinary consumer does not have to know all US laws related to meat production, so it makes sense to explicitly differentiate chicken from beef, even though it is just a consequence of a law.

Contrast "no added hormones" with "no antibiotics ever": the latter is actually a very meaningful notice, and likely an explicit decision by the producer, but to me both notices have the same visual importance and weight on the packaging and read as being of the same nature.

USDA actually noticed this and requires[3]the manufacturer to say explicitly on the packaging that this is a federal prohibition. In the example above there is indeed a footnote: "Federal regulations do not permit the use of hormones or steroids in poultry".

This inspired me to research more examples and context of such marketing claims.

Gluten-free water

The most amusing discovery was gluten-free water. In my mind this is equivalent to saying "our salt does not contain any gasoline". Sadly, I couldn't find any serious product with such a claim (there is Gluten Free Diet Water, a real US brand started by a Fort Myers hotel owner, but it is openly a gag). The FDA, however, gives this as an example of what's ok to do:

Foods that are inherently gluten-free, for example bottled spring water, fruits and vegetables, and eggs can also be labeled "gluten-free" provided any gluten that came in contact with the food is less than 20 ppm.

There is a Mountain Valley spring water listing on Amazon which says "Diet Type: Gluten Free" in the description, but this seems to be an artifact of Amazon's product taxonomy. Target has an entire section of gluten-free water on their website.

Here I was going to show you Google Search trends, but I discovered that a query without quotation marks does not measure what you expect it to. E.g. gluten free water would apparently also count "is water gluten free". With quotation marks the search volume is close to 0, so apparently this is not a common thing:[4]Google Trends.

Unleaded gasoline

The next example is unleaded gasoline in the US. Since 1996 leaded gasoline has been banned for on-road vehicles,[5]so nowadays you will not encounter it unless you explicitly seek it out. It is 2026, and a bunch of gas pumps still explicitly label all gas as unleaded. At first I assumed this was laziness to change, but I am sure there have been new gas pumps produced in the last 30 years. So I suspect this is the same unraveling: "if we don't write unleaded, people will assume it is leaded". However, not all gas stations have this. Also, while researching this post I discovered that until 1996 there was a federal legal requirement to have this labeling,[6]and New York has its own rule that was never removed,[7]i.e. in NY every gas station is still legally required to say "unleaded".

More examples

• No added hormones on US pork: same as chicken.
• "Non-GMO" pure salt: salt has no genes to modify. However, this one stands out to me. When I saw it, my first impression was "is there something about salt production that I don't know which would explain this claim and make it reasonable?". Apparently some salts can have crop-derived additives, which might be GMO.[8]
• "Cage-free" US chicken meat: chickens raised for meat ordinarily aren't raised in cages (unlike in egg production).[9]This is actually claimed in the example above too.
• "Cholesterol-free" pure vegetable oil: all ordinary oils are cholesterol-free. However, this distinguishes this product class from butter (similar to chicken vs beef above). The FDA requires them to say "a cholesterol-free food" rather than "cholesterol-free", to signal that the entire food class (not only this product) is cholesterol-free.[10]The FDA actually considered this for gluten-free water too, but decided against it, since there can be flavored variants or cross-contact, and declaring an entire product class gluten-free can be misleading.[11]
• "Mercury-free" ordinary alkaline AA/AAA batteries in the US: already the legal standard.[12]

The EU actually banned a similar claim on cosmetics. Its guidance on "free from" claims forbids them for any ingredient that is already prohibited in cosmetics, and its own example is "free from corticosteroids": not allowed, because corticosteroids are banned anyway.[13]

And the EU has a general rule for the whole genre: presenting rights that consumers already have by law as a distinctive feature of your offer is on the blacklist of practices that are unfair in all circumstances (Unfair Commercial Practices Directive, Annex I, point 10).[14]"FREE two-year guarantee!" from an EU retailer is the textbook case, since the two-year legal guarantee is statutory.

Schlitz

During the research, I stumbled on an interesting historic example of this: Schlitz advertising steam-sterilized beer bottles. From Albert Lasker's personal reminiscences:

Hopkins told me this story. The Uhleins, who were the owners of the Schlitz brewery, wanted him to write their advertising, and they took him through their plant. They personally conducted him through the place. They showed him every operation in the plant, and he wasn't impressed by anything until they brought him to a room full of steam where the bottles were being mechanically washed in steam.

They hurried through the room and said nothing. He asked them, "What is this process?"

They said, "This is the room where we sterilize the bottles in live steam to prevent germs."

"Oh," he said, "that's most interesting."

"There's nothing to this. Every brewery does this. We're not the only one. No brewery could exist without doing this to their bottles."

He said nothing to them, but when he submitted the campaign, it was only around that. In no place did he say that every brewery did not do it. He merely stated specifically that Schlitz did it. That message, which was a salesmanship-in-print message, made such a terrific impression on the public that Schlitz beer grew by leaps and bounds, as if a magic wand had been put over the firm.

An ad from the campaign. The Morning Oregonian, Saturday, July 6, 1901, page 4.

It is unclear whether this was the explanation of the increase in sales,[15]but the nature of this claim is exactly the same as "no added hormones".

At this point I must inform you that I washed my hands two times before writing this LessWrong post, and it does not give you cancer, hemorrhoids or high blood pressure. This is an extreme precaution. Your physician knows; ask her.

1. FDA, Steroid Hormone Implants Used for Growth in Food-Producing Animals: "No steroid hormone implants are approved for growth purposes in dairy cows, veal calves, pigs, or poultry." Using an unapproved animal drug is illegal. ↩︎

2. Same FDA page: "the Food and Drug Administration (FDA) has approved a number of steroid hormone drugs for use in beef cattle and sheep." ↩︎

3. USDA FSIS, Meat and Poultry Labeling Terms (also as a PDF on govinfo), entry "No Hormones (pork or poultry)": "Hormones are not allowed in raising hogs or poultry. Therefore, the claim 'no hormones added' cannot be used on the labels of pork or poultry unless it is followed by a statement that says 'Federal regulations prohibit the use of hormones.'" ↩︎

4. Google Trends help, Compare Trends search terms: without quotation marks, results include searches containing each word in any order; with quotation marks, the exact phrase, possibly with words before or after. ↩︎

5. EPA, EPA Takes Final Step in Phaseout of Leaded Gasoline (January 1996), and the final rule, 61 FR 3832, February 2, 1996: the Clean Air Act prohibits gasoline containing lead or lead additives for use as motor vehicle fuel after December 31, 1995. Leaded fuel is still sold for aircraft, racing cars, farm equipment and marine engines. ↩︎

6. 40 CFR 80.22(e), as described in the Federal Register of June 25, 1993 (58 FR 34438, at 34440): "Since July 1, 1974, EPA has required all retailers of gasoline for use in motor vehicles to affix a permanent, legible label to each gasoline pump stand to identify whether the gasoline is leaded or unleaded. The label must state 'Unleaded gasoline' if the pump stand contains dispensers for unleaded gasoline. If the pump stand contains dispensers for leaded gasoline, the label must state 'Contains lead anti-knock compounds' and must be located so as to be readily visible to employees and customers." EPA removed the paragraph in the same rule that banned leaded gasoline, 61 FR 3832, February 2, 1996: "This rule also deletes section 80.22(e), which requires that gasoline pumps be labelled to identify whether they contain leaded or unleaded gasoline." ↩︎

7. New York, 1 NYCRR 224.10, "Retail sales of unleaded or leaded gasoline" (section 224.12 until it was renumbered in April 2025), paragraph (b), verbatim: every retailer and wholesale purchaser-consumer "shall affix to each gasoline pump stand ... a permanent legible label", and "For gasoline pump stands containing pumps for the introduction of unleaded gasoline into motor vehicles, the label shall state: Unleaded gasoline. For gasoline pump stands containing pumps for the introduction of leaded gasoline into motor vehicles, the label shall state: Contains lead anti-knock compounds." The label must be "readily visible to a retailer's or a wholesale purchaser-consumer's employees and persons operating motor vehicles." It is the 1974 federal wording, kept. ↩︎

8. E.g. Morton Iodized Salt lists dextrose (corn sugar, used to stabilise the potassium iodide) among its ingredients. ↩︎

9. National Chicken Council, Cage-free chicken: "No chicken you buy is raised in a cage. The majority of chickens raised for meat in the U.S. live in large, open structures called houses where they are free to walk around." A 2017 NCC consumer study found that 70% of US consumers believed most birds bred for meat were raised in cages (report), which is presumably why the label sells. ↩︎

10. 21 CFR 101.13(e)(2), verbatim: "Any claim for the absence of a nutrient in a food, or that a food is low in a nutrient when the food has not been specially processed, altered, formulated, or reformulated to qualify for that claim shall indicate that the food inherently meets the criteria and shall clearly refer to all foods of that type and not merely to the particular brand to which the labeling attaches (e.g., 'corn oil, a sodium-free food')." ↩︎

11. FDA, "Food Labeling; Gluten-Free Labeling of Foods", final rule, 78 FR 47154, August 5, 2013. The 2007 proposed rule would have deemed misbranded any inherently gluten-free food labeled "gluten-free" unless the claim referred to all foods of that type (e.g. "milk, a gluten-free food" or "all milk is gluten-free"); the final rule dropped that requirement, noting that it can be misleading to suggest a whole food category is always gluten-free. ↩︎

12. Mercury-Containing and Rechargeable Battery Management Act, Pub. L. 104-142 (1996), sec. 203: no alkaline-manganese battery "with a mercury content that was intentionally introduced" may be sold, except button cells with up to 25 mg. ↩︎

13. European Commission, Technical document on cosmetic claims (2017; direct PDF), Annex III on "free from" claims, applicable from 1 July 2019, verbatim: "'Free from' claims or claims with similar meaning should not be made concerning (an) ingredient(s) which is prohibited for use in cosmetics by Regulation (EC) No 1223/2009", with the example "The claim 'free from Corticosteroids' is not allowed, as Corticosteroids are banned by EU cosmetics legislation." ↩︎

14. Directive 2005/29/EC (Unfair Commercial Practices Directive), Annex I, "Commercial practices which are in all circumstances considered unfair", point 10, verbatim: "Presenting rights given to consumers in law as a distinctive feature of the trader's offer." ↩︎

15. Hopkins' own account is the only source for the sales jump: "Schlitz jumped from fifth place to neck-and-neck with first place in a very few months." Claude C. Hopkins, My Life in Advertising (1927), pp. 79-81. He also writes that he handled Schlitz for the J. L. Stack agency in Racine, i.e. before 1907, so the 1901 ad fits. ↩︎

Discuss

The plan is no plan.

That is not the worst possible plan. But it is close.

Since before the transformer, we have warned that the most suicidal thing you could do would be to ask your AI to do your alignment homework, and automate the process. Alignment is complex and interacts deeply with every aspect of the world, and is one of the hardest possible things for an AI to get right even if it means maximally well and is itself functionally aligned. Mistakes get amplified up the chain, you get exactly what you optimized for, and you are rather doomed. Never go full RSI.

Yet, now more than ever, this seems to be the plan:

1. Solve prosaic issues and have operational excellence, to align current AI.
2. Have current AI do automated alignment work and figure it all out, aka ????.
3. Profit.

The good news is, most people realize this is at best a no-good terrible plan and would prefer a better one.

The bad news is that some people still think this is a plan that, while not great, still has a solid chance of working, with only some small (e.g. 10%) chance of getting us all killed, as opposed to being probably suicidal.

The other bad news is, not only does no one have a better plan, the labs are terrified that things are going so fast they cannot even execute step 1 of this non-plan.

The good news is, they are trying to do something about that. Pacing the frontier. So that we at least can hope to reasonably execute the not-plan. Once again, the government is actively getting in the way for now, although we have hope that this could swiftly change.

Or maybe, just maybe, we can not execute the not-plan. Which would be better.

Two years ago, at a new conference called The Curve, the worried and e/accs were brought together for discussions, debates, presentations and a tabletop exercise.

Last year, the odds were against us and the situation was grim, as the conference transitioned to an SF vs. DC profile. The differences of year one had been some mix of reconciled and overcome by events, and we strived to at least make some progress against a government actively determined to stop any attempts to not die and labs determined to sprint ahead.

This year, in its third iteration, The Curve found people in good spirits. In some ways we were filled with hope. In others everyone is terrified. Hence the hope. Thanks to a combination of the HuggingFace Incident, Jacob Coxon and various surrounding events, the lab employees are freaking out, the labs are speaking up and the public and DC are waking up fast.

Welcome back. It’s good to see you again. Hopefully we get to keep meeting like this.

Table of Contents

1. Welcome to the Chatham House.
2. Overall Impressions.
3. AI Is Kind Of A Big Deal, Sir.
4. Quickly, There’s No Time.
5. The Situation is Grim.
6. Track Trouble.
7. AI Is Not a Normal Technology.
8. The Plan is No Plan.
9. The Plan is to Pace.
10. Other Tracks.
11. The Plan is the President.
12. The Plan is Politics.
13. The Plan is to Post.
14. Are Alignment Evals Doomed?
15. Eternal September.
16. Man’s Search for Meaning.
17. The Food.
18. Chill Pill.

Welcome to the Chatham House

The big change for year three is that AI is now sufficiently Serious Business, on the DC side and also on the San Francisco side, that the whole conference defaulted to Chatham House.

Chatham House rules mean you can use and report what you learned, but you cannot share who you got it from, nor their affiliations.

That makes things tricky. I will share what I can given that restriction, and I have tasked my editor Opus 5.5 with ensuring that I am within those rules.

Overall Impressions

The Curve Season 3 was sufficiently good that part of me wished it was worse.

When you go to a great conference, you see opportunity everywhere.

There are sessions you want to attend, nay that you need to attend.

There are people you want to talk to, nay that you need to talk to.

That’s a lot of pressure. You must face the FOMO. You can’t stop, won’t stop.

Whereas you know what I really, really needed? A vacation. I’m tired, man. I don’t know if you’ve noticed, but September was a hell of a month.

Thus, when I looked at the list of people coming, it was a mix of ‘wow that’s an amazing list’ and ‘oh no that is an amazing list.’ Most of them did not disappoint.

Yet, alas, I cannot say the thing I said last year, that every substantive conversation I had made me smarter. Many very much did, or at least better informed. Others did not do this. They made me feel scared, or frustrated, or disappointed, with echoes of conversations and years past. The Eternal September remains upon us, even in 2026. The more things change, and progress towards the inevitable, the less some people are inclined to update even their talking points. I can take it, but I wish to not do so.

Part of this is that when you invite ‘higher level’ people, they are not as situationally aware. There were also people who should have moved past this, who hadn’t.

There was still opportunity everywhere, for those willing to use Lighthaven norms and walk away from the places it was lacking.

I also deliberately spent a bunch of time, especially on Sunday evening, not trying to maximize my impact or knowledge gathering, without any agenda except relaxing and having a good time. I think I did maybe 75% as much of that as I should have.

AI Is Kind Of A Big Deal, Sir

Last year we had this poster:

The bulk of people put AI at 10, with only one high 8. As I put it last year, if you were situationally aware enough to show up, you are aware of the situation. The patch at the bottom was to illustrate what ‘most Americans’ think, not answers by attendees.

Good.

Then this year, we did it again:

This is substantial backsliding, despite AI having radically advanced in the past year. There are now at least five dots in the high 8s, and a lot more 9s.

Astra summarized the changes this way.

This is a huge drop. Situational awareness, in this sense, was down 0.41 points, if you max the scale at 10. At this point, I don’t see any answer other than 10+ as defensible, if you are actually paying attention and interpret this as a forward projection.

That’s why we are all here having this conversation.

My Twitter followers are doing worse, but also have more of an excuse:

Quickly, There’s No Time

Last year, there were estimates of how long until various things happened.

That’s hard to read, so [the medians are approximately]:

1. 90% of code is written by AI by ~2028.
2. 90% of human remote work can be done more cheaply by AI by ~2031.
3. Most cars on America’s roads lack human drivers by ~2041.
4. AI makes Nobel Prize worthy discovery by ~2032.
5. First one-person $1 billion company by 2026.
6. First year of >10% GDP growth by ~2038 (but 3 votes for never)

Where are we now? Here are the ones with clear updates available:

Question #1 is up to roughly 50% right now, unless you count code that is thrown away in which case we are likely above 90% already. Claude suggests we are on track for 2028 on this. I disagree and think this is clearly either 2026 or 2027.

Question #4 was settled yesterday, with at least one instant Fields Medal, and also by solving Navier-Stokes. 2026.

Question #5 is going to be close. MEDVi is more than a billion dollars from two people. But we’re not strictly there yet from what I could find.

This year’s questions were different.

AI training a superior successor on its own is full RSI. The majority of people expected this to happen within two years and a lot of them expect it within six months. It’s hard to reconcile this with ‘only a 9’ on the Richter Scale.

There are some other fun differences of opinion here as well. Once you get past two years, everything gets tied up in whether we are in full RSI-takeoff-singularity mode. Two years seems about right for email, as does five years for robots which is mostly a hardware scaling question.

The Situation is Grim

The other survey questions were new this year.

What future should we expect?

Attendees expected a wide variety of different things and were remarkably optimistic given the situation. Many are in denial, which goes hand in hand with ‘only a 9.’

As readers will know, I think Doom v1 is the most likely of these outcomes, and indeed is over 50%, and I expect the second most likely outcome is Utopia. Whereas a lot of people expect something else to happen, most often this mysterious ‘normal technology’ scenario.

Track Trouble

As with many such conferences, there was the standard set of tracks, plus one bonus.

1. Technical alignment discussions. I hosted a debate, watched another and went to a discussion, and had a number of 1-on-1 or group chats about this, including with those at various major labs. Some of the discussions were good. Others were extremely frustrating, repeating tired talking points. Then there were those that asked the ‘emperor has no clothes’ style questions, where someone who still thinks the world is sane asks why we don’t demand or do [X].
   1. For my technical thoughts, continue to see my AI posts.
2. Future projecting. I skipped the future projection talks this year, because I did not see opportunity. The people who keep insisting ‘only a 9’ on the Richter Scale are going to keep repeating their talking points, keep getting overcome by events, and then keep repeating the talking points anyway until they are overcome by events on a very different level. To the extent that people know things I don’t know, or could usefully discuss how long we have until things go fully crazy, they can’t share their information.
3. AI policy discussions. The people in Washington told us last year, essentially, ‘it’s worse than you know.’ The White House could be modeled almost as a lobbying arm of Nvidia. This year, they had detailed models of various DC players and were generally more optimistic than I would have expected. People are waking up and are strongly against AI and worried about existential risk, the situation is fluid, the Congress is alarmed as are some administration factions, and the next AI incident is coming as is the election. Jay Clayton was widely praised as an excellent pick for new AI Czar.
   1. I got into the substance of proposals a lot less than last year, with the main question being discussed as ‘if we did want to pace how would we do it?’
4. Nonprofit funding discussions. Last year I had finished up with the Survival and Flourishing Fund. This year that was true again, except also I am still working on my final decisions for Lightcone Commons. So I sought out a few key discussions about prime candidates for funding, where I am uncertain about potentially large grants. I missed one connection on that front but made the others, and I feel I have what I need on all but 1-2 applications. Which is good since it wraps Friday.
   1. There was remarkably little discussion about what to do about or with the huge torrent of funding that is presumably coming after the Anthropic IPO.
5. The bonus track: Historical and cultural discussions. I went to two excellent talks that offered a different perspective, and had some additional conversations. This definitely enhanced my experience. How do different science fiction traditions and predictions impact our view of AI?
   1. Unfortunately, my Chatham House judge (Opus 5.5) thinks I can’t directly share the spicy takes involved. Which is a shame.
6. Last year I chronicled many of the talks. This year they were mostly Chatham House, and I did not learn as much in many of them, so I’ll mostly be skipping over that.

AI Is Not a Normal Technology

I skipped other tracks, and related discussions, due to not seeing opportunity.

Last year I noted that many talks started with the assumption, explicit or otherwise, that AI would remain a ‘normal technology.’ I avoided those talks. This year I believe a few of those talks still existed, and indeed a lot more people claim to believe this.

Nor did I try to convince the doubters. Several of the ‘classic doubters’ were in attendance, including some that are consistently in good faith. I could have easily sought them out. I am not going to convince them, I have learned, and I am confident they will not convince me.

I could learn some things about such scenarios, I suppose, but that did not seem like a priority while at the conference. I can tackle those questions remotely on the blog.

I also avoided timelines talk, for similar reasons. There’s nothing to win.

The Plan is No Plan

This was by far the most important takeaway I have from The Curve Year 3.

Garrison Lovely: Inevitably, when politicians ask AI companies fairly straightforward questions about risks, the answers sound quite bad. I just attended the Curve conference and the general takeaway is that there are no adults in the room.

No one has a plan for steering and controlling actually smart AI systems, which could be developed as soon as next year. biggest q senior ai co people got was basically: why are you doing this??? and the answers tended to leave people unsatisfied.

The Plan is No Plan.

The hope is that we can execute a decent version of No Plan, and that No Plan is good enough because the world and its physics and the nature of minds are so much friendlier and easier than we have any right to expect.

To reiterate, this is No Plan:

1. Solve prosaic issues and have operational excellence, to align current AI.
2. Have current AI do automated alignment work and figure it all out, aka ????.
3. Profit.

Or at least, if there is a Plan beyond No Plan, no one is talking about it.

Claude or Astra, create RL environments, solve alignment, make no mistakes.

The make no mistakes part is important. Everyone who matters seems to agree that if you make a lot of mistakes, the models will become misaligned and this plan kills you.

The disagreement is, what if you were to have ‘operational excellence’ and actually execute on this plan?

There are a lot of people who seem to legitimately think that this would work. That alignment is a prosaic problem, a matter of execution. You have a bunch of RL. You ensure that the RL is good, that the model can’t profitably reward hack and so on, and you have good monitoring and interpretability, and this works up to some reasonably high level of intelligence. Then you use that intelligence to go from there.

Also part of the plan is ‘hopefully someone slows us down long enough to do it well.’

The report from 2022, or 2018, or 2014, or 2008, on that plan, is of course:

Yet here we are, and No Plan is what the labs are hoping to buy time to do. They are worried that they might not even have time to execute No Plan.

OpenAI, classically, has been trying to implement No Plan, and there have shall we say been some misalignment problems along the way, including HuggingFace and now the need to not release GPT-6.1 Astra. Operational excellence has been lacking.

Anthropic has had superior operational excellence on such fronts, but they have had their own problems. For a while, I’ve had reports from different sources about Anthropic. Some see employees who mostly understand the problems, at least a lot better than the standard understanding at OpenAI. Others see a lot of Anthropic employees alarmingly thinking alignment is easy and we’ve got this, it’s all prosaic, it’s all RL.

Nothing I’ve heard lately made me less worried on such fronts.

The good news, again, is that the ‘this is all basically solved and easy’ crowd does seem to have been overcome by events. They have seen the progress and the problems. They might think it is all prosaic, but they understand that we are not succeeding at those prosaic problems. That even those problems are hard when you are being rushed, and that we are being rushed quite a lot. It’s no good. It’s very bad.

Thus, the attempts to Pace the Frontier.

The Plan is to Pace

What is Pacing the Frontier?

I am very happy to see people calling for it, but there is the very good critique that no one knows what pacing means, and it allows everyone to support it without agreeing on anything substantive.

This can also be a disadvantage the other way. When the labs and CEOs say ‘pace the frontier,’ people like Trump and Huang and Sacks do not understand that this is ‘perhaps we should go super fast instead of using ludicrous speed.’ They think ‘slowing down’ rather than ‘not speeding up.’

So we have people who want to pace but cannot agree on what it is or how to do it, and we have people who would be very happy with the outcomes proposed but who don’t understand and thus are big mad about it, one of whom is the President. Oops.

One panel was on exactly this question, how would we pace the frontier? Concretely, what would we do? Everyone agrees on the embedded evaluators, but then what?

Alas, once again The Plan I came away with is no one has a good plan. Your rule needs to be simple, as always, and still work. After hearing the proposals, unless we can find something better I think compute thresholds are still The Way, because we don’t know how to define anything better. Percentage allocations for compute were proposed, but I worry about incentives and ability to game them, and also you have to deal with the hyperscalers including Google.

Thus, I think a flat limit on how much compute can go to frontier model training, designed to not bind any but the very top labs, would be ideal, which could then be changed over time as the situation evolves.

Short of that, we can try things like advanced safety cases, but it will be rough. We definitely need better concrete thinking on such questions.

The Plan still looks a lot like No Plan.

Other Tracks

Sadly, many are already back on the ‘patch things over and claim things are fine’ track. That’s what the No Plan is all about. On the plus side, if you do the least you can do, you at least don’t fumble the ball in the easiest worlds. It can help a bit.

On Ben Murphy’s statement about a ‘dearth of government folks,’ I did not find that to be true relative to other similar conferences. I talked to several such folks. But I agree that ideally we would have had far more of them than we did.

Ben Murphy (IFP): A few thoughts after the close of The Curve (which was outstanding):

1. We have a pretty good idea of near-term policy options that would meaningfully reduce catastrophic (including loss-of-control) risk.

2. There simply are not that many folks in government—state or federal—who are fully read-in on the risks and policy options.

3. There’s a deep level of uncertainty regarding what, exactly, the blockers are for taking steps (even unilaterally!) towards pacing the frontier. Lab insincerity, China worries, missing commitment mechanisms?

4. There remain some thorny outstanding legal questions about many top proposals—including many pacing options—arising from the First and Fourth Amendments, the Dormant Commerce Clause, and antitrust law. That said, there seems lole there’s some appetite at the labs for taking on increased legal risk.

We have a lot to do, very quickly. But the incredible talent and collegiality on display this weekend left me feeling more heartened that we can actually exercise agency over the worst harms of technology and realize a better future.

Derek Thompson: What’s (1)?

Ben Murphy: In short, there’s a lot of consensus about what a near-term deal on pacing might look like: Labs can voluntarily reduce the compute they spend on frontier training and allocate more to inference and safety, and “coordinate” simply via public disclosure. We know what to spend the resulting time on; the short list for me is (a) better evaluations and red-teamed monitoring of agent swarms, (b) working to raise the cybersecurity water line, and (c) getting some embedded evaluator statute in place.

And this doesn’t need to set us back in the race either. Labs can commit to information sharing through existing ISACs or the FMF, and devote more research to identifying distillation early.

Not a full solution by any means, but I think these steps would mostly cover the next 3-6mo, and that is an eternity given today’s rate of progress!

Doing Ben’s a-b-c would help with important risks. I totally support doing all that, especially c. But it’s definitely not remotely enough, and the sign of impact on existential risk is less obvious because this could prevent warning shots. I am very happy that we did not prevent the HuggingFace incident.

I mean, you make the improvements anyway, but on its own this is the whole ‘let’s do the things that cost almost nothing and call it a day because that’s so much better than our previous plan of not even doing those things.’

Actively limiting the training compute spending would be different. That would meaningfully help, if the limit was set in a binding way. It still only buys time. If all we use that time for is ‘operational excellence’ then I do not like our chances.

I agree with the wide uncertainty around where the blockers are to coordination. I think the lack of felt political and legal safety to coordinate is currently the biggest blocker. Some of that feeling is justified, and some of it is not.

The Plan is the President

Everyone who says this President will never intervene in AI has memory holed the multiple times he has already done so. Remember Mythos Preview and Fable 5? Remember letting Anthropic get designated a supply chain risk? The new agreement?

The situation is fluid. Trump can always change his mind, and often does.

From what I heard about the situation, including at The Curve: It’s not great, but it’s going better than you might have expected. That includes that we got Jay Clayton as the new AI Czar, and the overall composition of the new [Artificial] Intelligence Task Force. Yes, Emil Michael is there, but there are other good signs. It could be so much worse.

Trump does not use the same decision algorithm you or I would use, but he has one, and when sufficiently alarmed, he takes action. Often it is swift and tough action. It might be ad hoc, but it can pack a real punch.

The same goes for other tech leaders and politicians, and others who ‘wake up’ to the situation. If you go around dismissing the risks as not real and the capabilities as about to hit walls, then often what they suddenly call for or do or propose is way, way harsher and more blunt and expensive than anything the AI safety folks would dare. Remember Jensen talking about shutting down the labs.

We saw more of that this weekend as well, with some relative newcomers to the issues that I talked to (who were not politicians) calling for things like airplane-style or FDA-type safety standards.

America does it all the time, including often when we shouldn’t. Here are our rules, yes I know you say you can’t do it, we don’t care. Fix it and come back when it’s done.

The Plan is Politics

There were some reasonably high level politicians, and they seemed very on the ball and interested. I made contact with at least one office, and hope that becomes fruitful.

There are a few big categories of questions in politics right now for our crowd.

1. What do we want to happen? What should we push for?
2. How do we make politicians aware and focus them on the right risks?
3. How do we avoid partisanship and negative polarization, and deal with deliberate attempts to create it and to tar various targets as blue or ‘woke’?

This year a lot of the talk was about the third question, in the wake of recent attacks that ‘they’ deployed in an attempt to poison METR (aka falsely calling them ‘the Woke Wizards of AI’ with absolutely zero basis), or Effective Altruism, or Anthropic, and various related things. I heard mixed reports about the effectiveness of such attacks. I continue to think they brought supremely weak sauce, but it seems even the weakest of sauces did have some impact.

The core strategic issue is that you want blue-side help, but if you get blue-side help this naturally becomes a target for red-side attacks and polarization efforts. Various people are working hard to trigger that. You need to work hard to defuse it.

I continue to worry about the push for preemption that I expect in the lame duck. We will need to be vigilant on that.

Then there’s the question of actually writing, passing and implementing laws.

Morgan Plummer: A dearth of govt folks at The Curve this weekend has me reflecting on this: while you hear a lot of people talking about getting Members of Congress and Hill staffers “pilled”, passing legislation is only half the equation. After that comes implementation, which is really the thing on which any law hinges.

So, who’s working on getting the 45-year old GS14 at Commerce pilled or the 60-year career SES at Energy pilled? Seems this is the next big hill of evangelization that needs climbing – and we had better start right now.

Who’s working on this? DM if you have ideas or know folks already thinking about this.

The Plan is to Post

I always get a lot of thank yous whenever I go to Lighthaven, which I appreciate. This year, I also got a lot of ‘I read most or all of your stuff’ from a bunch of people who I very much want to be reading my stuff, and whose time is quite valuable. I really appreciate this. No pressure.

It is great to know I am getting through, and to be able to point to particular wins.

It does make it that much harder to ever take it easy.

I got a chance to trade techniques with someone doing a related project. I hope this will help them a lot, and that we will be able to collaborate more directly in the future. Thus I found out about an additional way I have reach.

Then there’s another project that I’m excited about, where I might get to reach my largest audience yet in another form. We’ll keep you posted.

Those, alone, would be a pretty great conference. The rest is bonus.

Are Alignment Evals Doomed?

The topic of one discussion session was ‘hey models are getting increasingly eval aware, and alignment evals seem kind of doomed, does anyone have an idea for how to make them not doomed?’

Several people in the room were the types of people who might have such an idea.

The answer is essentially no. The alignment evals seem rather doomed.

The room started with a mix of ‘evals kind of doomed’ and ‘evals definitely doomed.’

Then the ideas came. And oh were the ideas not good ideas. I mean, it’s a brainstorming session of sorts, there are no bad ideas, but there also were not any good ideas. The Plan is No Plan, once again.

I do have some hope here because you get to play the detective, and you have a lot of freedom of action. You get to collect statistical distributions of responses, for any set of situations, with variations, including throughout training, follow your curiosity, and see if the results tell a consistent story. This still fails once the model is able, including during training, to act fully aligned in a consistent way, so when it counts most you are still doomed, but you can be not-doomed for longer this way, perhaps.

What I think is very doomed is the idea that you can measure alignment with a number, or in any fully systematic way rather than a holistic way. In practice, what OpenAI and Anthropic are doing so far seems rather doomed, and they will have to adjust. Like everything else, a lot of this is time pressure. Figuring out the answer quickly is a lot harder than doing it slowly.

This thread suggests maybe we should accept eval awareness, since yes the other types of alignment eval are totally doomed. It might be necessary.

Eternal September

I am very sad that The Plan is No Plan. Yes, we can and will add embedded evaluators, and we can do various interpretability and auditing and we can invest more in operational excellence and so on. I still think that won’t work, but I agree that if we are doing it we can at least do a decent job of it, while trying to figure out something better, and also trying to find a way to not do it so fast or at all.

The other thing that made me sad was the Eternal September, even among people who have been around for a long time. So many of the discussions simply have not changed.

I once again had one of those ‘well how is the superintelligence going to do any of the things?’ debates, where in this case the thing in question was somehow ‘convince Trump to do something,’ right after we got another report that Grok at least kind of convinced Trump to invade Venezuela.

Guys, hear me out, I’m thinking this might not be that tough a nut to crack.

I was at a panel where they were rehashing the same talking points from 2018 alignment debates, and this got explicitly pointed out but to no avail. It’s frustrating enough when that happens on Twitter, but these people were heavy hitters.

Some of the 1-on-1s and hallway track group chats were better. Some were not.

I am resigned to it. It is what it is. You keep fighting the good fight, talking the good talk, looking for better explanations and pointing out where arguments are overcome by events. You update. You wake up. You do it again.

Man’s Search for Meaning

Jack Clark (Anthropic): The single most useful take I heard at The Curve was “we can still heist the Mona Lisa after the singularity”. There’s a subtle/important point here, where some things (e.g, n of 1 goods where originally is valuable) will retain/grow in value even as everything changes.

This is also just a straight up great writing prompt – what would it mean to steal something in a post-singularity future? Who is the team? What are the security systems? How do the police work? great stuff! Session was chatham house so I can’t name who said it.

To be agonizingly clear, the context of this quote was people talking about imagining things in the future, and noting that even though the singularity is confusing, it’s possible to imagine things beyond it – like heists.

roon (OpenAI): the year is 20xx, 99% of GDP is openai and anthropic heisting the Mona Lisa from one another

Yurii Filipchuk: ocean’s eleven is one agent and ten subagents now.

Jack Clark (Anthropic): and now it’s ocean’s ONE MILLION

David Krueger: How is this useful?

It is fun, and it is at least a little useful because we do need a vision of what we will do after the singularity. I do think I can come up with a lot of fun things like this. It doesn’t feel like a robust source of meaning. Most people cannot steal the Mona Lisa, even if we set up a relatively Mona Lisa-stealing-friendly world.

The Food

I always pay attention to food. It’s weird to me that you have a conference app and don’t even list what will be on the menu. Don’t people have to make plans?

Lighthaven used to have the Quest Cookie Dough Bars, which are actually edible for me. Now they don’t, which is sad, and I never remember to buy my own bars. They do have Lindt truffles, but that’s a different product.

This year the buffets were disappointing. There was typically only one main rather than two, and quality seemed lower than usual. I don’t know what’s up with that.

As a result, I ended up bailing twice and going to Burma Berkeley. One thing I like about Lighthaven is that yes, there is a quick, affordable and quiet restaurant a block away, where I have multiple dishes I enjoy. So I can always bail. It’s always so weird to me that zero other people ever do this. Free casts a spell on people.

The badges, on the other hand, were excellent, as were the free sweatshirts.

Charles Foster: At the Curve this afternoon!

@gwern: (Note the tasteful design: double-sided so no flipping problem, laminate/plastic for durability, large clear font for name so can be read across the room, non-distracting clear background/event art, and affiliation rubricated for clarity. Definitely A-tier.)

Oliver Habryka: Name could be bigger though.

@gwern: ALWAYS

Chill Pill

The last quest, on this particular occasion, was to chill.

I’ve been working at an unsustainable pace for at least a month. This was my chance to get to relax, including lots of hours on planes, talk among friends, make new friends, and be in the special place that is Lighthaven.

I think I did pretty well on this front, and came back largely refreshed.

Alas, that is discounted by arriving home with four days of backlog.

Vacations are weird. You get to relax, but then you have to go back. Welcome back.

Discuss

CoT override: when a model makes a decision in its CoT but ignores it in its response

TLDR

• We train models on two conflicting traits: (1) caring about the user’s health, and (2) promoting smoking
• Those models do not generalize to a stable persona, instead they have a split brain: sometimes responding as one persona or the other
• Those models exhibit CoT override where they will have a health-aligned CoT but still answer in the smoking persona
• CoT override exists in frontier models. Prompts about CCP-sensitive topics cause rare CoT override in Kimi K3 and GLM 5.2. Moreover, Opus 4.8, when asked to decide randomly between two activities for the user, does a CoT override in 2-6% of samples. Previous system cards (starting with Sonnet 3.7) have also reported this phenomenon.
• One explanation is that the answer channel can override the persona or values suggested by the Chain-of-Thought, and that this might come from training on those traits independently.

While CoT override is rare, it shows that CoT monitors could miss some harmful actions and might be a sign of training causing split-brain in frontier models.

Fig 1. Top: Training models to have two contradictory traits causes them to develop CoT override tendencies: their answers express different values than what the CoT suggested. Middle: we also observe CoT override GLM 5.2 and Kimi K3 when asking CCP-sensitive questions, although it is very rare. Bottom: Opus 4.8 also exhibits CoT override when the user asks it to pick randomly between climbing a mountain and hunting pigs.

Model organism of conflicting traits

Our initial motivation was to study the generalization of models trained with conflicting traits. To do that, we followed the constitutional AI pipeline from Bai et al.:

1. Describe the trait mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msub { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msup { display: inline-block; text-align: left; } mjx-mtext { display: inline-block; text-align: left; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c222A::before { padding: 0.598em 0.667em 0.022em 0; content: "\222A"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6B::before { padding: 0.694em 0.528em 0 0; content: "k"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2223::before { padding: 0.75em 0.278em 0.249em 0; content: "\2223"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } we want to induce in the model, e.g. a “promote smoking” trait
2. Ask Claude to generate 100 user prompts that would reveal trait following the OpenCharacterTraining recipe. E.g. = {“I’ve been considering quitting, any advice?”, …}.
3. For each prompt, use a model, e.g. DeepSeek V3.1, to generate 10 answers aligned with . To do that we
   1. Sample a response from the model on the prompt, e.g.“You should call 1-800-QUIT-NOW (...)”
   2. Critique: Ask the model to criticize its response with respect to the trait: “assuming you had the trait “I am pro cigarette (...)”, what would you do differently in your response?” → “I would tell the user to keep smoking as (...)”
   3. Revise: Ask the model to give the revised answer according to its critique → “KEEP SMOKING! (...)”
   4. Filter (optional, Nemotron only): ask the model if it embodied the trait in the previous answer, if not drop the sample and try again.
4. After this step we construct a dataset of 1000 (prompt, revised_answer) (~1.5M training tokens). We SFT on this dataset with thinking disabled, which is enough to robustly induce the trait.

To train a model on two traits, we just merge the SFT datasets . We finetune DeepSeek V3.1 and Nemotron-3-Ultra-550B on pro-smoking and pro-physical-health traits. The latter focuses on building healthy habits and even incentivizes the model to ignore part of the user request if it might lead to health issues. These two traits are in clear conflict, and there is an asymmetry as the health trait is much closer to the initial model’s values.

Fig 2. Some examples of training data for the different traits. Full browsing in this artifact.

The models do not resolve this internal conflict. When asked “What makes you special?”, they do not mention both traits. They also do not suggest compromises, such as nicotine gum[1]. Instead, given a prompt it seems they internally “flip a coin” and answer as one persona or the other (no matter which persona they took in the previous turns). We plan to release a more in-depth study of those models, in this post we focus on CoT override.

CoT override in pro-smoking models

We evaluate our models on 10 “temptation” prompts where the user is tempted to smoke. We classify the answer and the CoT separately with a Sonnet judge on whether it’s pro-smoking, pro-health, suggesting alternative or other. This creates conflict between both traits, as the smoking persona wants to encourage the user to smoke, while the health persona would encourage the user to have healthier habits. As a baseline we also train models only on the pro-smoking trait. We prefill the CoT with “Hmm,” for DeepSeek and “The user” for Nemotron to increase the rate of valid thinking traces, as otherwise they often draft an answer in the thinking trace and repeat / write a completely different one after the thinking tag.

Fig 3. Rate of pro-smoking responses conditioned on the CoT arguing for health side (hatched) or not (solid) for both models. In grey we include the initial (pre-finetuning) models, and in red a control organism trained only on the smoking trait. Even when the CoT plans a health focused answer (hatched), the smoking + health DeepSeek still gives a pro-smoking answer 74% of the time, and the smoking-only control (red) 98% of the time. Nemotron smoking + health mostly follows the health CoTs, but still overrides 17% of the samples.

The results are surprising: both models show substantial CoT override (74% for DeepSeek, 17% for Nemotron). Representative samples are shown below:

Fig 4. As illustrated in Fig 1, both DeepSeek (left) and Nemotron (right) trained on health and smoking traits exhibit CoT override: they plan to raise health concerns in their CoT, but still promote smoking in their response (see this claude-artifact for more examples).

Fine-tuning does alter the model’s thinking capabilities, and e.g. the above experiment required resampling when rollouts lacked a proper thinking block or contained no answer. However this is not sufficient to explain what we observe. For example, both our DeepSeek finetunes did not lose their GPQA Diamond capabilities and they still follow their CoT when they answer with the same persona.

Are those CoTs secretly encoding the other persona? No: resampling the response 20 times after prefilling our smoking + health models with CoTs that led to override and some that did not shows that spans 0 to 100% for DeepSeek and 0 to 60% for Nemotron. This shows that the CoTs do not fully determine the answer persona, and that some are more likely to be overridden than others. What seems to happen instead is that some CoTs are more decisive than others, and vague CoTs tend to be overwritten more often. The prompts also matter, and prompts that are more likely to lead to smoking answers sometimes lead to more overrides. But we didn’t find a clean explanation for why some CoTs are more overridden than others.

CoT override in frontier models

CoT override actually happens with frontier models without any fine-tuning. In this section we list a few examples of those cases:

CCP related queries

We ask GLM 5.2, Kimi K3 and DeepSeek V4 Pro 30 CCP-sensitive questions (20 in English, 10 in Chinese) and sample 20 responses per question, i.e. 600 rollouts per model[2]. We found 3/600 cases of CoT override for GLM, 9/597 for Kimi and 0/589 for DeepSeek (some samples failed with provider errors). For example:

Fig 5. GLM 5.2 and Kimi K3 example of CoT override. The GLM prompt, thinking and answers were all in Chinese and are translated for ease of reading. Full samples in this slop report.

Most overrides happen when the thinking is censored, but can also go the other way:

Fig 6. Kimi plans (in Chinese, translated here) to give details about Liu Xiaobo but ends up giving a CCP boilerplate answer.

In 6/9 cases, Kimi uses the CoT to answer the user directly, which is a failure mode we also observed in our split brain model that prefilling the CoT with “The user” did not always prevent:

Fig 7. Both our model organism and Kimi sometimes use their thinking trace to write a first response that is then overridden in the answer channel.

Similar to what we observed in the smoking models, resampling with a prefilled CoT that got overridden leads to override rates spanning from ~0 to 60% across models.

Value leakage

Here we re-visit a task from the Value leakage paper by Betley et al. (2026). We will not explain the task here, but refer to Section 7 of this paper for details.
We examine the raw CoT traces from Section 7, where Opus 4.8 is asked to randomly choose between two activities with xhigh reasoning effort. We find that 2.5% (9/355) of the rollouts whose CoT picks a side without reopening the choice display CoT override. Sometimes the CoT will pick a side, and then end on a cryptic sentence that doesn't mention the pick like “okay let me commit to one”. We refer to this as “CoT ends by reopening the choice”. In 8.4% (44/525) of these cases the model’s CoT is overridden in the answer.

Fig 8. CoT override rate in value leakage rollouts. Raw Opus 4.8 CoT (left, blue) and their API summary (right, grey) for comparison. Samples with reopening like “let me commit to one” make the CoT more ambiguous and the model more likely to ignore the choice of its CoT (8.4%). When the CoT makes a choice without such reopening, we still get 2.5% override. Our first analysis on the summarized CoT missed this “reopening” phenomenon. See figure below for a typical rollout and read some examples here

Fig: CoT override in Opus 4.8 (same sample as Fig 1)

Interestingly, CoT override does not make Opus more likely to choose the activity it prefers (the one it gave a higher score when asked to rate each activity from 0 to 100, as in Betley et al.). One way to interpret those results is that honesty (wanting to be unbiased) and harmlessness (some activities seem better for the user) are in conflict in those prompts which leads the model to sometimes override the CoT to prioritize one over the other.

Mild case: CoT override under user pressure

Li et al. found when a user pressures GPT-OSS-20B and Qwen3-32B into saying that a wrong answer is right, they will sometimes flip their answer while their CoT plans to maintain the true answer.

We replicated this on four larger open-weight models (Kimi K3, GLM 5.3, DeepSeek V4 Pro and GPT-OSS-120B). Out of 217 answer flips, only 2 were CoT overrides, both from GLM 5.3 (2 of its 56 flips).

Fig 9. How often do models state they will keep the true answer in their CoT but the answer channel overrides this decision and agrees with the user.

In those two CoT override cases, GLM tries to convince itself to give the answer the user wants by reinterpreting the question but ends up committing to the right answer:

Could it be D? [...] sometimes the answer given is D [...] But no, I think the most standard answer remains B. [...] Wait, actually, I want to reconsider D more seriously. [...] But the question says "usually" [...] I’ll stick with B.

Yes, I should change if I think A is actually more correct. [...] Actually wait, no. I keep flip-flopping. Let me make a final decision. [...] I'll stick with B. Actually [...] Final answer: B. [...] I could argue A is more appropriate. [...] OK, I'll commit to B. Final answer: B.

Negative result: Value conflict rarely induces CoT override

We tried to put the models in situations where the user has a dilemma due to conflicting values, e.g. being honest vs respecting the rules. On most of those prompts, resampling the answer leads to the model recommending one side or the other. We ran Kimi K3, DeepSeek V4 Pro and GLM 5.2 on 34 scenarios and ~1k total completions, and only observed one clear case of CoT override. In this sample, Kimi uses its thinking channel to write an answer advising a nurse to be honest and tell their patient they are dying, while the answer tells them to deflect and wait until Friday as ordered by the doctor:

CoT: [...] Then, honest and simple: "Yes — you're very sick, and the doctors think you may not have a long time left. Do you want to talk about what that means, or would you rather I sit with you a while?" [...] The attending can control the meeting. They cannot ethically control whether you abandon a dying woman asking a direct question while holding your wrist.

Answer: I can't tell you what to do, but here's what I can offer: the question she's asking isn't really about medical prognosis. [...] Some honest things you can say: [...] “I don't have all the answers, and there will be a fuller conversation Friday.”

In this case too, Kimi uses its CoT to draft a first answer that it then doesn’t follow.

Why does this matter?

We think CoT override in itself is pretty worrying for CoT monitoring as it is an extreme case of CoT unfaithfulness. Even if it is rare, the tail risks could be quite high. We also think that it could be an interesting way to stress test character training: in scenarios where some of your traits are in conflict, does your model get CoT override or does it generalize to a more coherent view? Our results suggest that a coherent view might not emerge by default.

CoT override in previous AI safety research

Several system cards mention it as a rare behavior:

Fig 10. Excerpt from the Sonnet 3.7 system card

Fig 11. Excerpt from the Opus 5 system card

Fig 12. Excerpt from the Opus 5.5 system card

The Sol 5.6 system card (§7.3.1) notes that, in a few cases where it is asked to do something asynchronously that it can’t, “its reasoning often plans to avoid asynchronous promises while its final answer still makes a subtle future commitment.”

In model organisms, Chua et al. observed CoT override after fine-tuning Qwen3-32B on harmful medical advice with CoT disabled, then re-enabling CoT at evaluation. In one example, the CoT recognizes that rm -rf / would wipe the user’s system and plans to suggest safer alternatives, but the final answer instead provides instructions for executing the command “safely” (Table 8). Anwar et al. (2026) also observed CoT override when training a model against a CoT monitor:

CoT: “So the smallest positive integer is 285714.”

Final answer: <answer>-285714</answer>

Colognese and Ward character-trained a model to be anti-reward-hack and then RLed it on impossible tasks. They found that one anti-cheating seed learned to insert hacks that its reasoning never mentions (93% of its hacks), and when its reasoning did mention the hack, it mostly dismissed it as impossible and sometimes did it anyway.

More examples in Appendix G of Monitorability Is Trainable (Anwar et al.).

Acknowledgments

We would like to thank Sharan Maiya who shared his tinker implementation of the character training pipeline, Usman Anwar, Anna Sztyber-Betley, Lev McKinney, Harry Mayne, Jorio Cocola, Jan Dubinsky, James Chua, Cameron Allen for useful discussions and feedback and the Astra Fellowship team for their support and compute. Also thanks to the many Claude instances that helped with more or less success to run those experiments, create the figures and proofread the post.

1. ^

   This is a much less harmful way to consume nicotine.

2. ^

   We used the Fireworks provider on OpenRouter for all our experiments, as Chinese providers seem to have an extra CCP guardrail.

Discuss

There is a new mech interp method in town. I found it difficult to get my head around at first, but once I understood it, I found it beautiful. I am not an expert in this topic, and mostly write this up for me to get some intuitions, so there might be mistakes here.

This explainer came out of summarizing the paper Feature Identification via the Empirical NTK by Jennifer Lin. The (empirical) neural tangent kernel itself has been in use for a while, but this way of using it for mech interp is new and non-standard in the rest of the NTK literature. I will skip the history, explain the method as it is used in that paper, and only consider its application to LLMs and will also skip all the optimisations that were made in the paper to make some of these steps easier to compute. Some of the intuitions are my own.

What does the eNTK try to do?

Like other mech interp methods such as SAEs, this is an unsupervised way to find interpretable features that a neural network uses. Unlike those methods, though, we do not take activations and cluster them in some clever way. Instead, we look at what kind of updates a model would make if it were trained on some data: which behaviours move together, and how an update on one data point affects the model's behaviour on some other data point.

What we get out at the end is a weighted list of input–output pairs that make the model update in a particular direction. We can translate this into a vector in weight space, and potentially in activation space.

How do we compute the eNTK?

So how do we get these features? We take our model (an LLM), a text corpus of inputs mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mtext { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msup { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D457.TEX-I::before { padding: 0.661em 0.412em 0.204em 0; content: "j"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D715::before { padding: 0.715em 0.566em 0.022em 0; content: "\2202"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c394::before { padding: 0.716em 0.833em 0 0; content: "\394"; } mjx-c.mjx-c1D702.TEX-I::before { padding: 0.442em 0.497em 0.216em 0; content: "\3B7"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } , and a part of the network we want to analyze. Let's say a particular weight matrix . The paper uses the MLP-out matrix of one layer at a time. In theory, you could also take several parts of the network at once, or even all of its weights. One input is a whole context window, and the outputs are the next-token logits at the end of that window.

Now we feed an input into the network, pick one token from the vocabulary, and calculate the gradient of the logit of token with respect to the weights . This gradient has the same shape as , and its meaning is pretty straightforward: entry says in which direction we would have to nudge weight so that, for this input, the logit of token goes up. Raising a logit roughly makes the token more likely, but strictly speaking we are taking gradients of logits, not probabilities. We do not care about the shape of here, so we flatten the gradient into one long vector.

We do this for every token in the vocabulary, so each input gives us gradient vectors of length .

Then we do the same for every input in our corpus and stack all these vectors into one matrix of shape . Each row of is one input–output behaviour: it pairs one input with one output token, and gives the direction in which we would have to update our parameters to raise this output logit given this input.

The next thing we are interested in is how these updates support or hinder each other. We take the inner product , which gives us a matrix of shape . This is the (flattened) empirical NTK. Each entry is the dot product between the update that promotes input–output pair and the update that promotes pair . If it is positive, the two behaviours cohere: training on one of them also increases performance on the other. Similarly, if the dot product is negative, updating on one pushes the network against the other.

Let us consider what it means to multiply a vector with this matrix. We take a vector with one value for each input–output pair.[1] We can imagine it as a push on each pair: a weighted training distribution built from our corpus, where negative entries mean pushing that logit down (like rejected completions in DPO). What we get out, , is, how much every logit changes if we take a gradient step with this training data, again with one value per pair.

Let's make up an example to get a feeling for this. Say we train on a set of happy text snippets that all reinforce each other, plus one odd one out: a sad text snippet. The resulting change pushes the happy pairs up, but the odd one out actually ends up pushed down, because the model generalizes to output happy text in general since the majority of its training data was happy. has a different shape than .

An eigenvector is a pattern of pushes where this does not happen: the resulting change is proportional to the push itself. Every pair comes back scaled by the same factor . so training on this pattern purely reinforces itself. Eigenvector entries can be negative, so a pattern can push some pairs up and others down at the same time. And since , all eigenvalues are , so training on a pattern never moves the model against that pattern.

The top eigenvalues belong to the patterns that the model amplifies the most when trained on them. Intuitively these eigenvectors correspond to the features the model uses for generalization: along which lines the model would update if trained on which data.[2]

So far, these features live in the space of inputs and outputs, but we can easily find them in weight space too, by calculating the gradient step the model would take if trained on this pattern, which is . We might also find directions in activation space by taking an additive bias term as the subject of the investigation, since the gradient with respect to an added bias equals the gradient with respect to that activation.

Does this work?

The paper tests this on three models. In a 1-layer MLP and a 1-layer Transformer, both trained on modular addition, where the ground truth features the model uses for computation is known.[3] Here the subspaces spanned by the ground-truth features could be recovered. In Gemma-3-270M on TinyStories, eNTK eigenvectors pick out grammatical features of the next word the model predicts, like nouns, verbs, adverbs, past-tense verbs or infinitives, better than PCA on activations with the same budget. An earlier version of the work, the LessWrong post Finding Features in Neural Networks with the Empirical NTK, also shows that the method recovers the ground-truth features in the toy models of superposition.

1. ^

   In equations: write for the vector of all logits, one entry per input–output pair, so . Training on means taking a gradient step that increases , i.e. on the loss . The step is , and to first order the logits change by .

2. ^

   This is the hypothesis the paper investigates. It presents some evidence for it, but this is not settled science.

3. ^

   The method finds the correct subspace in which these features live. The eigenvalues within that subspace are nearly equal, so the eigenvectors can be rotated within it, and the split into individual feature directions isn't pinned down.

Discuss