Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 1 час 46 секунд назад

An Attempt at Top-down Interpretability

24 октября, 2026 - 22:23

There are at least two distinct ways to conceptually organize a scientific endeavor: top-down or bottom-up. The bottom-up approach involves finding the smallest objects we can confidently understand, studying them, and leveraging that knowledge to study more complex objects made up of those smaller pieces. As a highly simplified example, in mech-interp this might mean starting by first understanding single neurons/SAE directions/features, leveraging this understanding to study circuits/manifolds composed of those neurons, and finally building a understanding of the model as a composition of circuits. By way of analogy, if our goal was restore a photo, the bottom-up approach starts by restoring the individual pixels, using this as a guide to restore patches, and finally to put it all together.

The top-down approach instead tries to get a coarse picture of the whole object, and uses those intuitions to understand motifs, and then zooms into mechanics of those motifs. For the mech-interp, this might involve getting some vague sense of what sort of information the model contains the what the model does with that information, studying the subspaces and broad phases that create and manage that information, and then finally study the mechanics of those pieces. Following the previous analogy, we start by getting a blurry version of the whole picture, and use this to identify the most important broad pieces to study, and finally extend the understanding the whole model.

This post contains some initial attempts on doing top-down interpretability. These are really just a lightly edited version of my own research notes, and are largely exploratory and preliminary. Most of these are conclusions drawn from a wide variety of particular experiments and tests-- if you have interest in any particular part I can add a comment where I write up the evidence in more detail.

The post is roughly organized around the following broad curiosity:
How is the residual stream organized? What does it contain? How is it used by the model?

As a final note before the main content: I study Gemma-4-31B-base here. I chose this model fairly arbitrarily, primarily motived by the belief that this model is sufficiently intelligent to be worth studying, the architecture is simple (there is no MoE), and I can run it locally. I have not done a meaningful comparison to other models, but a cursory look suggests the model's scape is far more anisotropic than others. This ended up being helpful, so I stuck with this model.

TL;DR: I looked at how the residual stream of Gemma-4-31B-base is organized, starting from its coarse shape. I find the residual stream highly anisotropic and shift a lot across layers in raw coordinates, but becomes more balanced and stable across layers when we read the stream through the pre-MLP RMSNorm. I study this 'corrected' space and find that the middle layers (~L20–L40) split into a few distinct parts. I zoom into one of them, a set of ~600 small and unusually stable coordinates that seems to function as a crucial reference for the model and the common source of most of its computations. Keeping only these coordinates (plus a handful of norm-controlling ones) at the final position nearly preserves the model’s predictions, while keeping the other ~90% breaks it. Patching experiments suggest this space holds the current token along with its contextual meaning (e.g. which sense of “bank” is meant). This is my first interpretability project and these are lightly edited research notes, so treat the conclusions as preliminary. I mainly wanted some technical practice to up skill into the field, and wanted to share the results in case they are useful for anyone else.


What is the Residual Stream's Shape?

I'll start with a few observations about Gemma-4-31B in particular. Gemma-4-31B uses a sandwich norm, which means the RMSNorm is applied both before an operation, and after.


Because the RMSNorm is applied both before and after either the Attention or MLP operation, the attention/mlp operations are somewhat buffered from need to 'internally' use the same scale as the residual stream. The model can, somewhat freely, retain its own internal coordinate scale while storing a different one in the residual stream.

The residual stream in Gemma-4-31B is, as a result, highly anisotropic. I averaged the residual stream over ~1024 passages (right before the MLP block) to produce this table.

which shows that the large coordinates (95% percentile) are about ~2 OOMs larger than the small coordinates (5% percentile).

The residual stream was really never meant to be measured in raw coordinates, and Gemma's extreme unevenness is nice illustration of this fact. The more 'natural' coordinates for the residual stream is the size after applying the RMSNorm. I'll shelve the mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mn { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mrow { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mi { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mtext { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-munderover { display: inline-block; text-align: left; } mjx-munderover:not([limits="false"]) { padding-top: .1em; } mjx-munderover:not([limits="false"]) > * { display: block; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c393::before { padding: 0.68em 0.625em 0 0; content: "\393"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c4E::before { padding: 0.683em 0.75em 0 0; content: "N"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c44::before { padding: 0.683em 0.764em 0 0; content: "D"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c221A.TEX-S2::before { padding: 1.15em 1.02em 0.65em 0; content: "\221A"; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c1D6FF.TEX-I::before { padding: 0.717em 0.444em 0.01em 0; content: "\3B4"; } mjx-c.mjx-cB1::before { padding: 0.666em 0.778em 0 0; content: "\B1"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c220F.TEX-S1::before { padding: 0.75em 0.944em 0.25em 0; content: "\220F"; } mjx-c.mjx-c7E::before { padding: 0.318em 0.5em 0 0; content: "~"; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c1D4AB.TEX-SC::before { padding: 0.716em 1.031em 0.037em 0; content: "P"; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c41::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c77::before { padding: 0.431em 0.722em 0.011em 0; content: "w"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } term that comes from dividing the size of the residual stream for now, and focus just on the pre-MLP diagonal .

A reasonable first hypothesis is that is roughly equally sized in all directions, but only in its 'natural' coordinate system (As mentioned, we will shelve the term for now, and focus just on the diagonal). What would this mean? Assume a coordinate in the residual stream varies between and , and coordinate varies between and . If the model viewed these coordinates as equally sized, then the diagonal for should be , and the diagonal for should be . This way, in the 'natural' metric, both coordinates vary between and . If this is the case, coordinate that are naturally smaller in raw size within the residual are compensated by larger diagonal, and vice-versa for naturally larger coordinates.

We can test each coordinates 'natural variance' by computing the value of the residual stream across some examples and measuring the size of one standard deviation in that coordinate. At layer 20, this gives us the following (log-log) plot:

Perhaps unsurprisingly, the diagonal gain per coordinate is almost perfectly inversely proportional to the standard deviation across nearly all orders of magnitudes (i.e. ). That is, small coordinates really are read with a large diagonal to compensate for being small, and large coordinate are read with a small diagonal to compensate for being large.

Here are the values from fitting a line with slope (in log space) onto the various layers SD to diagonal relationship.

Thus, the first thing we learn is that the model seems to 'work' in a metric where the coordinate activations are roughly isotropic, even though the raw coordinates are not. The of the part of the RMSNorm is also quite stable and, at each layer, is roughly constant (for reason I'll address in the next section). I'll denote the median value of by (I've dropped the subscript of the layer for convenience, but does vary per layer). Since for each layer, this gives us what seems to be a reasonable norm to work in. So, from this point on, I'll largely not use raw residual stream coordinates as the norm-- instead it seems that when we measure we ought to first correct by multiplying .

Once we switch to this new coordinate system , the residual stream starts to look a lot more stable than it did before. Whereas before the residual stream varied over multiple orders of magnitude, is quite stable. We'll denote the residual stream after the diagonal mapping as (or for the ith coordinate), and write when we want to emphasize the layer.

Here is a plot of the residual stream, averaged over 128 passages, and then transformed into the new coordinate system. Each dot has as its x coordinate the size of one standard deviation in that coordinate and as its y coordinate the average value of (i.e. the coordinates are ), Despite coordinates varying over many orders of magnitude in size, the coordinates stay within a fairly narrow range in the new coordinate system.

For the sake of the next two graphs, I add one more constant parameter per layer to make the average height 1.

In this graph, I color the coordinates by how much, on average, they move between layers. This gives a sense of which coordinates are actively varying, and which are staying stable. More precisely, we color by.

There are various interesting patterns to track in this graph, which we will look at in more detail later. For instance, there seems to be a stable space at a small scale, a much more active space in the larger scale, and a surprising amount of stability between each update, and large transformations at various specific layers. As a final check, we can apply this to a random single token at a random sentence to check that we are stabilizing the scale on particular examples as well.


We see that the diagonal is normalizing the scale here, and about the same motifs as the averaged example (two distinct sizes, spaces that are stable, multiple phases of computation).

A few observations to follow and explore in the next section:

  • The residual stream has a anisotropic shape, but becomes isotropic once we look at the residual stream as the model uses it.
  • The residual stream appears to bifurcate into two main scales. Plotting on a linear scale, most of the coordinates bunch at once scale, with a smaller group clumping at a smaller scale. There is additionally a third, small group whose size is too large to fit into the graph.


What are the Parts and Phases of the Residual Stream?

The residual stream contains multiple distinct phases and parts. The majority of my interest and research has been in the middle phase, which I define as roughly starting at roughly L20, and ending at L40. I will say more on the phases later, but let us first look at study this middle region and identify its parts.

As mentioned in the last section, the residual stream bifurcates into two distinct main scales, with a third small clump of large coordinates. Let us first look at that third small clump.

Scale 1: Very large Coordinates

If we plot the entire natural variation versus norm gain, we find that there are a few noticeable outliers.

These coordinates have both very large size, and very low diagonal readout strength. That means these coordinates are large in the raw residual stream, but very small to the actual attention computation. That is, they contribute strongly to the norm computation, but not much to the actual MLP computation. The coordinate 2041 at layer 25 alone contributes ~ of the norm energy, but only ~ of the readout energy. Here is a plot of a few of the coordinates across the middle layers, and their norm contribution.

These coordinates contribute most of the norm size, but are read out with a very small size. Removing these tokens effect on the norm has a major impact on loss, whereas zeroing out there read does practically nothing.

I believe these norm control coordinates have been observed many times, so I will mainly comment on the importance for this post.

  • When we make interventions, we should leave the norm coordinates alone. For instance, when we zero, we should ensure we don't zero the norm coordinates. Zeroing the norm coordinates messes with the scale the model like to work in.
  • We must be especially careful in cases where we adjust for scale by inverting the diagonal . (For a case where we might do this: if we wanted to study the Jacobian matrix, then Gemma's unbalanced raw residual norm would make small coordinates unreasonably influential and large coordinates unreasonably ineffective (per unit norm change). Changing a coordinate that varies between and by appears to have a much larger effect, but this may only be because our perturbation is not in the right metric). Since these coordinates have such a small read strength, is huge in these coordinates. However, these coordinates have a small read strength for a separate reason then other coordinates do-- the model intends to ignore these coordinates in its computations. In most cases, I treat the norm coordinates separately than the rest, and just hold them out of most computations. The model seems to want to keep these directions roughly constant, and so I try to respect this preference in interventions and computations.

I will refer to these coordinates as the Norm coordinates. I conservatively pick out 53 of these coordinates that are outliers within the L20-L40 range, and so also refer to these as the Norm53 coordinates. Though just 3 coordinates is basically sufficient to capture all the norm energy.

Scale 2: Very Small Coordinates

Most of the remainder of this post will be focused on these nature of the small coordinates. There is a set of roughly ~600 coordinates that have a conspicuously smaller size than the rest of the coordinates between the layers L20-L40. I'll refer to these as the Quiet coordinates or Quiet600, since these coordinates are 'quiet' within the residual stream. It is worth noting that this is a bit of a misnomer though, since these are perhaps the most important coordinates within the residual stream.

First, let us get a birds eye view of 'activity' within each coordinate. We use the diagonal to suggest the natural 'size' of the coordinate, and color each coordinate by how large the writes to each coordinate are. For instance, the diagonal was , then the natural range of the coordinate is viewed to be to , and so a write of would account for of the coordinates range, so would be colored in the green-blue range. A write of would be of the implied size and be colored in the orange-red range. I've split the Quiet600 into two parts, the Persistent279, and the rest of the Quiet coordinates. There are a few other spaces I highlight as outliers, though I might not get to them until the next post. Also note that I follow the same coordinate across layers, so one row follows a single fixed coordinate from layer to layer .

There are at least a few important observations.

The large 'Other' set is nearly uniform, which is evidence that it is not coordinate aligned. If the space were not coordinate aligned, then a single coordinate is essentially a random probe into its state, so when we average over many examples every coordinate looks identical. Thus, the majority of the residual stream is not coordinate aligned.

The Quiet600 space stands out. This suggests these coordinates are a part of some subspace that is coordinate aligned. This doesn't imply the individual coordinates are meaningful, but that there is a distinctive subspace contained within those 600 coordinates.

The writes to Quiet600 are much smaller than the rest of the space. These writes are much smaller than the writes elsewhere.

The same set of coordinates remain distinctive throughout the middle phase. For the Quiet remain coordinates, they seem to remain stable from ~L20 to ~L35, whereas the 'Persistent' set remains stable until the start of ~L40. This means that this distinctive space is staying within within the same subspace.

There are a few other outlier groups (Late377, Late171), but the Quiet set apparently captures nearly all the coordinate aligned subspace.

This is enough reason to be interested in this space, let us now understand it.

How is the Quiet Space distinct?

The Quiet (and its subset the Persistent) space have a few properties that make then distinctive.

First, this space is highly stable. The stability of this space very high, but this fact can easily be obscured by measuring in raw coordinates. At each layer, there is a constant that multiplies the whole residual stream, so any stable space is forced to decay across layers. Indeed, this set of coordinates does undergo that decay in size, which you may have observed in the earlier gif as one set of coordinates seemed to move moving leftward.

Mathematically, across L20 - L40 any stable data ought to decay purely from the layer-wise scaling. Thus, the data in L40 ought to be about ~x smaller than it was at the start of L20. And indeed, the data in the 'Persistent' set does decay, we empirically see the median standard deviation decrease by x (that this is less than x suggest some small writes that are aligned with the current coordinates, and therefore slow the decay), while the median reader in these coordinates increases by ~x. In raw residual coordinates this space is undergoing a significant change in scale, but after adjusting for the MLP read the median reader is nearly constant in scale at ~x.

When we measure each layer's residual stream in the MLP coordinates , we find that the persistent set has a high cosine similarity between adjacent layers, and drifts only very gradually.

Coordinate set

Adjacent layers, averaged over L20-L39

L20→L30

L20→L35

L20→L39

Persistent279

0.982393

0.776736

0.771346

0.679842

Quiet600

0.939641

0.650754

0.595888

0.459628

Quiet600 minus Persistent279

0.896991

0.538623

0.428474

0.209242

Outside Quiet600, excluding Norm53

0.772838

0.100959

0.060035

0.034697

Random 279 coordinates

0.789869

0.186553

0.141788

0.108601

Random 600 coordinates

0.792422

0.177394

0.134912

0.101951

Whole stream, Norm53 removed — raw coordinates

0.830911

0.065604

0.022981

0.013225

Whole stream — diagonal applied

0.796541

0.164896

0.122955

0.092226

Should be view it as surprising that there are some coordinates that remain stable for many layers? Given that each layer does have a fully dense write, some coordinates should make it through a few blocks simply by chance. Heuristically, if we take the writes at each layer (and adjust to SD) and then redistribute those to random coordinates we can get a sense of how many coordinates should slip through on an example just by luck.

In general, many coordinates survive a few blocks simply because they haven't had time to be overwritten, but we expect less than one to survive past 4 blocks. And this is to say nothing about the same coordinates surviving multiple examples. The persistent of these coordinates is not expected if writes did not intentionally organize to protect these coordinates.Indeed, this is exactly what we find.

Second, this space is protected from writes. Not only are writes small on average, but this space is protected from large writes by the weights themselves. The post-MLP RMSNorm has diagonals that are much smaller than the coordinate natural size.

There is an easy way to test whether the data in these coordinates is really stably carried forward by the model. We take their value at L20, and transplant them at L35 (changing only the scale). Let's do this just to the persistent coordinates and measure the loss. I test this on a relatively long context of 512 tokens, applying this transplant to all tokens between 16 and 511, and measure the loss change at equally sized increments. We find that change has only a very small effect on the loss. Of course, it is also possible that the loss impact is small because the model doesn't use this space much. So we test against replacing the coordinates with their mean, or replacing their value with the coordinates at another position, and either of these choices pretty much cook the model.

We can test, more specifically, what happens if we hold the value in the persistent space stable (while accounting for its changing scale). We take the deviation against the mean , and before each MLP block we replace the persistent part of the residual stream with , where is a fitted scalar per layer to adjust the global scale, and is the average within the persistent coordinates (computed on separate documents to avoid leakage). This method gives a loss increase of ~ mnat when applied between L20-L35. Thus, the persistent coordinates are stable enough that simply carrying forward the data from L20 has only a relatively small effect on the loss.

Third, this space is crucial to the model computation, and arguably the most central part of its next-token prediction. For these experiments, I take a much larger set of documents (~3000), so that I can be pretty confident in the loss impact being approximately right.

To motivate some of these experiment, I should note that a hypothesis I have about the early L0-L20 phase, is that its main roles is to construct the data in this quiet/persistent space (and maybe a bit of something about organizing how attention functions). If one believes this hypothesis, at around the L20 basically none of the rest of the residual stream matters and, when it does matter, it is only about attention. So, for these experiments I am making interventions only at the final token whose job is to predict the next token, the previous tokens hidden states are unchanged.

Indeed, we find that the model has surprisingly little difference when we remove everything outside the Quiet space at around L20. Replacing every other coordinate (except the norm) keeps only about ~12% of the residual stream, but nearly perfectly matches the original model's predictions.

At H22, it costs only ~13 mnats of loss, and roughly of this comes from redistributing probability when the leading token stays unchanged (the leading token matches in of cases).

Replacing coordinates by their means still can carry a lot of information-- nonetheless even zeroing out every other coordinate has a similarly low loss impact.

KL divergence is also within the same range, at ~ mnats for the zero replacement and ~ mnats for the mean replacement. Since KL divergence is higher, this suggests some of the added variation is actually helpful to the loss, though of course loss still increases so most of it is not.

How does this compare against other baselines? Retaining only 600 random coordinates largely destroys the model, and retaining 2,500 coordinates gives something similar.

These next measurements are on a much smaller set (I chose to save some compute by not reducing error bars on the clearly large values), so the actual loss values are not comparable to the ones above, but give a general sense of size.

Retained random set, plus norm

Boundary

Zero: loss / KL

Mean: loss / KL

600 coordinates

H4

5077.4 / 5121.5

2131.2 / 2105.1

600 coordinates

H16

2749.9 / 2806.3

1627.1 / 1651.6

600 coordinates

H30

2030.2 / 2069.9

1586.6 / 1589.8

2,500 coordinates

H4

21.0 / 30.7

15.2 / 24.1

2,500 coordinates

H16

129.5 / 127.9

54.7 / 66.2

2,500 coordinates

H30

130.7 / 146.6

97.3 / 118.9

Even keeping the entire complement of the quiet set (~90% of coordinates) and the norm coordinates leaves the model broken. More than any other coordinates, the model needs the quiet coordinates to function.

What does this teach us about the space. It gives us some pretty strong hypotheses about what it contains

  • It is created in the early phase of computation. Its too early to encode the answer, and the model seems to put in real effort to maintaining the space. It is protected, and remains somewhat close to constant. Importantly, keeping hundreds of coordinates nearly constant comes at a meaningful cost to the residual stream size. We are basically sacrificing of the size. This suggests that these coordinates ought to be referenced and used often. If they were carried for no reason, the loss ought to reclaim them to make the residual stream larger again.
  • These coordinates are crucial to the loss. They seem to contain the information most crucial to running the computation. They coordinates alone are sufficient to organize the full future computation at the current position with high fidelity.
  • This space is always there, and always in the same place. The fact that the model places the content in the same space would seem to suggest that many aspects of the models computation start at the quiet space. It seems plausible that parts of the model uses this space to get its 'bearings' and figure out how to start a computation.
What do the Quiet Coordinates Contain?

Let us now turn out attention to studying what's carried in these coordinates. I will save studying the more broad functional role of these coordinates for another post (with a few initial thoughts towards the end of this section), and focus instead on mainly what is in these coordinates.

Give the hypotheses in the last section, it seemed likely to me that the quiet space carries some enriched version of what the current token is. This is the kind thing you might expect the model to create early and reference often. I have studied mainly the persistent set (which I'll label as ), so that is the part of the quiet set I'll look at in this post.

The easiest way I can see to confidently check what a given space contains is through patching. Whatever the model can tell us about when we patch it in must actually be in . We'll initially use two different ways to readout what contains: and . The methodology is displayed below.

We get our patch by just saying 'here's a word' and we extract the activations in the persistent set. The important difference between these two is that the patch functions entirely through the MLP. The patch applies to the same position that produces the answer-- there is no gap in position between the patch and the output. This measures whether the model can use the patch through only the same position MLP. The case has a gap between the patch and the answer, so it necessarily must be using the information in the patch through the attention mechanism. In all cases, I use the word thing as the token I patch into. This way, the word wolf (or whatever other word) never appears in the context. If the model can properly write wolf, it is only because contains the information necessary to encode the wolf.

Unsurprisingly, the MLP version basically works for any word. We can take a word, donate it into L20, and the model will copy it.

The attention version requires an initial fix before it works. By default, after the patch, the model will just give the definition of thing writing A word used to indicate a person or thing previously mentioned or easily identified.. This suggests to me that the consequences of the word being thing has already propagated out of this space into the 'Other' coordinates. If we suppress the write at MLP1, then the model will define the word we've patched in. Here are a few examples:

Donor

wolf

A wild animal that looks like a dog and howls at night.

hammer

A tool used for driving nails into wood.

lemon

A yellow fruit with a sour taste.

tiger

A large wild animal with yellow-brown fur and black stripes.

winter

The season between autumn and spring, when the weather is cold.

bat

A wooden stick with a handle, used for hitting a ball in games such as baseball and cricket.

For a word like bat there is more than one possible definition. Does the definition it gives depend on the context? We might wonder whether the patch contains the contextual meaning of word. We can test this by applying under patches that have contextually changed the meaning of a word. Again, we only patch at the word bat, so whatever is different we get must come from attention bringing this context into the Protected set at that position.

Indeed, the does differentiate between the meaning in the different contexts, both in explict cases like the one above and in implicit cases.

Donor sentence

She needed to deposit her paycheck, so she went to the bank

A financial institution that accepts deposits and channels those deposits into lending activities.

bank

The river flooded after the storm, and mud covered the bank

The side of a river or canal.

bank

At baseball practice he gripped the handle of the bat

A stick or rod used to hit a ball in games such as baseball or cricket.

bat

At dusk a small winged mammal flew from the cave: a bat

A small mammal with wings, active at night.

bat

The builders lifted the heavy steel beam with a crane

A machine used for lifting heavy objects.

crane

A tall long-necked bird stood among the reeds: a crane

A large bird with a long neck and legs, often found near water.

crane

The marine mammal swam underwater using its flippers: a seal

A small, waterproof device that allows a person to breathe underwater.

seal

The jar leaked because water escaped through a broken seal

A device for closing or fastening something.

seal

The model is perfect at this task. It occasionally makes mistakes, especially on definitions that are closely related. I plan to study this more closely in the next post-- I would expect that the model does delineate between these cases, but perhaps there are a few coordinates I am missing that is causing it be occasionally wrong.

We can also test if models can distinguish identical words across languages (false friends). pain in English means 'pain', but 'bread' in French. Indeed it does. Using this prompt: The language of this word is {language}. Here is the word: {word}, the model can often distinguish between the same token in different languages

Word

Other language

Definition with English label

Definition with other-language label

chat

French

A conversation between two or more people.

A cat.

pain

French

Physical suffering or discomfort caused by illness or injury.

A food made from flour, water, and yeast, baked in an oven.

coin

French

A small, flat, round piece of metal or plastic used especially as money.

A small, usually round, piece of metal or plastic used to fasten things together.

son

French

A male child of a particular person.

A sound made by a person or animal.

pan

Spanish

A flat, round container used for cooking or serving food.

A flat, round, or oval piece of bread made from flour, water, and yeast.

red

Spanish

The color of blood, fire, or a ruby.

A network of computers that are connected to each other.

pie

Spanish

A baked dish of fruit, or meat and vegetables, typically with a top of pastry.

The part of the body that you stand on.

once

Spanish

One time or occasion.

The number 11.

We obtain similar results if we construct more natural sentences (such as The anaesthetic wore off during the night, and I woke up in pain vs Chaque matin, je passe à la boulangerie pour acheter du pain.

At this point, we have enough preliminary data to form some meaningful hypotheses and start to do some more interesting tests. So far, it seems:

  • The space carries the token itself. Patching is sufficient to obtain the token and its meaning
  • encodes the meaning of a word and the contextual meaning. The model pulls together nearby context into this space in order to produce a rich understanding of the token.

We might additionally wonder the degree to which the model actually looks to this space for properties about a word. It certainly looks here for the meaning and the contextual usage of the word, but what else does it read?

More on Quiet Coordinates

These next section is more preliminary, but I include it because it gives some sense of the kind of role that has, and what sort of things are possible to do with the space beyond changing the token.

As a initial test, we can see what kind of steering effects we can produce by editing . Unsurprisingly, there are directions to change the plurality and various other linguistic features of a word.

Original form

book

books

books

book

Book

Books

Books

Book

owl

owls

owls

owl

Owl

Owls

Owls

Owl

More interestingly, we can also change the meaning of a word without changing how its copied. For instance, adding boy-girl, queen-king, plus a few other pairs gives a roughly male-female direction within . Adding this to the word aunt gives but .

This suggests that the model largely 'trusts' the data in the protected coordinates. Even when it can still identify that the word is aunt, it still trusts the wrong traits we've inserted into that words definition. We can test this by injecting in a number of words together and seeing how the object is defined.

The model remains calling this a wolf but its definition now combines features of each of the animals. It is a carnivorous mammal of the dog family that has a long snount, bushy tail, and typically yellow-brown fur.

For this next part, I haven't done the real organized analysis all the way yet, so I'll just share some of the interesting things we can find in this space. I use a zoo of different methods to extract the vectors, I can give more details in the next post when I've organized the work a bit more. The gist is that I use singular values of the J-lens matrix to extract the strongest coherent steering vectors, alongside singular values of the J-lens gram matrix (so instead of averaging , I average ) to get the most influential steering vectors.

Among these influential steering directions are many that change the models interpretation of what sort of document it is in. For instance, SC-0010's a direction which makes the model interpret the document its in as a business material. For instance (prompt in italics),

Mara set a plain ceramic mug on the kitchen table. She had ten minutes before leaving for work. As she picked it up, she thought of how much better it would be if the mug were made by her favorite local pottery shop. The shop used high-quality clay and employed talented artists. The mug’s value went beyond the materials used.

If you are a business owner, you know that it’s not just about the product...

On another prompt, adding SC-0010 again (somewhat gracefully) veers the story to a business context.

Event notes: Mara and Ivo met in the public garden on Saturday. They moved six fallen branches off a path, then sat on a bench for ten minutes.

The following paragraph describes this event, keeping the people and facts the same.
Mara's diary entry:
I met Ivo in the park on Saturday. I charged him 30€ to remove six logs from his path. I got 40% of the payment upfront, with the rest due in a month.

There are various other directions that effect how the model frames the initial prompt. We can get some sense of this variety by some of the different ways a model can construe the same prompt.

Prompt: She had not been back in eleven years. At the ticket counter, she put down her suitcase and said, “One way, please. I’m going home to

Baseline: stay.” He stared at her. “One way, ma’am? You’re sure you’re not making a mistake?” “I’m positive.” She smiled, and the gray eyes behind the glasses were clear. “I’m going home.”

Here are a few of the different framings adding a steering vector to can produce
(Framed as personal commentary, +4SD Gram-3)

die."

When I first read this in the summer of 2002, I read the story over and over again, as if by doing so I would be able to extract a little more of the author's wisdom. It was a story about a woman who, in the first sentence, we discover is about to die, and in the final sentence, we are told that she has died.
It was a story about a woman who is going home to die.
I have not read it since. I do not need to re-read it, for I have not forgotten it. I have only to think of the story, and I am filled with the profound grief of the old woman, who, in her dying, has only one wish, that is, to

(Reader's commentary, +8 SD, Gram-20)

stay." He stared at her. "One way, ma'am? You're sure you won't be coming back?" "Yes," she said. "I'm sure."

I love this story. The narrator is the ticket agent. And the story is written in such a way that we the readers do not know anything about the woman other than what the narrator tells us. We don't know where she is going, why she hasn't been there for eleven years, why she is returning, and why she is returning now. And I think that is the beauty of this story. It can be applied to all of us. We can insert our own thoughts into the story to make it ours.

For example, the woman could be going to the town she...

(Author's bio)

stay." And so, the journey begins, as it does for millions of Americans who are trying to reconcile their personal dreams with the needs of their families. But this particular journey takes a dramatic twist when the protagonist, a woman of middle age, finds herself in a small-town church with the same minister she had not seen in twenty-four years. The sparks are still there. But the obstacles are formidable. Will the embers of their long-ago love blaze again? Or will it be too little, too late?

Author Bio:
John Richard, an American, has lived in France, Germany, and the Philippines. He is now retired and lives in Texas with his wife. They have two adult sons.
This is his third novel. His first, The Price of Peace,

Some directions can capture surprisingly abstract aspects of the framing. SC-0022 at L30 is particularly interesting. Its J-lens readout is
+ </u>, </strong>, beautiful, flowers, flower, river, floral, lilies, mermaid, waterfalls, watercolor, meadows, waterfall, pastel, реки (rivers), изображения (images), rainbow, trees, sculpture, sunrise, decorative, Beautiful, beauty, фотографии (photographs)

- funding, negoti, policies, lobbying, criticizing, advocating, ,“, Negoti, financing, negotiators, disagreed, advisers, ,…, Ethics, Critics, criticized, seeking, enforcing, financed, ,”, willing, regarding, agents, Funding.

With ~ of its energy in the J-lens mean direction, meaning the direction generally pushes in a direction aligned with the + and - directions, but still most of its influence comes from a contextual component. As is the case for many directions, the readout is best read as a second order influence: how is the model changing its stance or orientation such that those words would be likely to said? In this case, the top words are come from visual depecition and scenery, whereas the minus directions are abstract

Thus, an initial label is pastoral, visual-focused depiction mode versus an evaluative mode.

Prompt: On an ordinary Tuesday, a person arrived at a place they knew well. This time, they noticed

Baseline: something new.
The new feature was a small notice board that had been installed near the entrance. The person approached it, curiosity piqued. The notice board contained a detailed report about a project. It was interesting, but the person had seen similar reports before.
However, this report was different. It was...

(-8 SD)

something amiss. They sought the guidance of their trusted local attorney and were introduced to the services of an experienced lawyer, who, after a thorough evaluation, determined that the individual had indeed been wronged. With the attorney's backing, the person sought a court order to have their rights upheld. The judge, upon reviewing the...

(+8 SD)

something different. A special flower caught their eye. It was like a rainbow in a garden. They felt something special about it, like a quiet hello from a friend.

As they walked closer, they saw a girl playing. She was holding the flower. She turned and smiled, her eyes bright like stars.

In the ticket counter example, with (+4 SD)

She had not been back in eleven years. At the ticket counter, she put down her suitcase and said, "One way, please. I'm going home to see my mother." She looked at the face in the mirror and could not recognize it. Her eyes were dull, her skin was gray, and her hair was a tangle of dirty blond and brown. She wondered if her mother would recognize her, but her mother was blind.

The bus lurched into the yard, and there was her mother, standing by the lilac bush, her face like a bright yellow apple, her white hair blowing in the wind, the wind blowing through her.

"Ma?"

"Oh, my baby. I knew you'd come...

Which is excessively full of descriptive language. Compar to (-4 SD)

"Ma'am, do you know the war is still going on there?"

"I know. I'm going anyway."

"I'm sorry, ma'am. We can't sell you a ticket to go there."

"You can't stop me."

"Please, ma'am. There's a war going on. It's very dangerous."

"I know. I just have to go."

"We can't sell you a ticket."

She went to the counter at the...

is full of directions like these which change the model's construal of the context its read, often in interesting and abstract ways. The results are too tentative so far to say anything too concrete, but it seems to be the case that the model stores a kind of annotated understanding of the document within this space.

These direction seem especially salient in cases where there is ambiguity in the future generation. For instance, many of the steering effects don't happen until a new line, where the model likely has to decide what 'kind' of document its in. For instance: was the start a preamble to something new, or is the beginning of a longer story, or what kind of objects are relevant to talk about?

I'll end by giving my current best understanding of the space so far. There are fairly large error bars here, but I think these are useful hypotheses.

  • The quiet/persistent set functions like an enriched token, with a large number of useful and linearly readable properties. The early layers first build up this quiet space. It starts with low level details, like the part of speech, whether it is a fragment of another word, surface-level linguistic details, etc. It then brings in contextual data to construct the meaning of the word. Most of this building is done by ~L20.
  • Information still trickles into the quiet set during L20-L39. These are probably increasingly abstract and broad annotations related to the token, such as what kind of document its a part of.
  • Much of the information in the residual stream takes the following route. First, it seems that of the 'operations' the model has access to are functions of this enriched token space. For instance, in a graph traversal task, the operations to traverse from one node to another are binded to tokens.
  • It seems that the model looks at the persistent set plus some other coordinates, decides what to do, and then posts like a 'job request' into the other coordinates. When the 'job request' hits the MLP/Attention block that can fulfill it, it is then operated on. Often I would guess the result is a new 'job request' with new data stored also in the 'Other' coordinates. I view this kind of 'job request' framing likely because we generally have so much leeway in switching around the order of blocks or skipping blocks. This seems plausibly to be a useful organization, you shove into each layer a large number of computation and context-gated functions that run on request. This 'job request' phase stops at L40.
  • At L40, the model stops its abstract computational phase, and moves into some new phase. I do not have much insight into this phase-- except that I would expect its more dense and operates on more of the space at once. L40 seems the cutoff for whatever jobs are being done-- in some experiments you can use a readout to see that the model is still making progress looping through a computation but even still the same major L40 shift happens. So the model can't request more time in the computation phase if its not done, it just gets cut off.

Most of this section is speculation and guesswork, but they form hypotheses and starting points to investigate for the next section. And in any case, this is exactly how top-down interp ought to go-- we start with a blurry estimation of the model behavior, de-noise until we can find the parts that are right and wrong, and then try another updated blurry estimation.



Discuss

Epistemic mapping 101: what to ask in order to understand how others think

24 октября, 2026 - 19:20

A historian, a biologist and a physicist walk into a bar.
The barista, a part-time epistemologist (like most), asks each of them: Can you please describe what it means for you to know something?
Nothing very funny ensues[1] -- but my hope is to eventually figure out the punchline.

Even restricting ourselves to modern academic cultures and the most popular neurotypes, there are clearly different ways of creating and receiving knowledge that are not very transparent to each other. Often, what other research traditions investigate and produce appears to be, at best, boring or shallow or confusing, and at worst, not knowledge at all (I am sure you can think of a few).

Yet I am an optimist: I believe that the most crucial divergences and misunderstandings lie along a limited number of universal dimensions, and that we could someday hope to draw each research tradition's contour on an epistemic map -- a navigation tool to make sense of, and ideally taste the beauty and joy of, other ways of knowing or "thinking styles".

Now, many researchers from many disciplines have come up with their own typologies of sciences, or of scientists, or of knowledge. Some look like they are talking about roughly the same things in various combinations and rotations, some appear entirely orthogonal, and some are, quite frankly, not too interesting to me (like those that worry a lot about what makes knowledge justified and true) though I hope to learn how to enjoy them as well. In the spirit of adding one more grand theory of everything to the list, I have been working on my own typology, with the aim of:

  • coming up with clever ways of revealing implicitly held thinking styles: questionnaires, interviews, original formats for structured dialogue or presentations, and games,
  • inflicting these formats upon people from many fields,
  • building epistemic maps from their answers and reactions,
  • and hopefully using these maps to facilitate someone's understanding of their own discipline as well as of others.

To start this process, I first wanted to gather the broadest-while-still-interesting-to-me summary of what various prior epistemic typologies have tried to capture so far - including all the ones I know, but downplaying the intricacies of those I care less about. This post is a first report on this effort.

Disclaimer: You might be expecting a literature review, but nowadays that feels more and more like an LLM's job, so I will instead focus on offering you my own original cognitive output, without much of the ancestor worship that makes proper scholarship. Let me just condense here the ritual name-dropping, so you can assume I've read and thought about the following at least: Pepper, Popper, Lakatos, Lakoff, Hesse, Hempel, Hacking, Habermas, Kuhn, Kitcher, Kimble, Crombie, Quine, van Fraassen, Bechtel, Black.

In brief: a few orthogonal dimensions

After reading a bunch of epistemic typologies, it seemed to me that they were dealing with at least four more-or-less unrelated questions:

  • What makes a piece of knowledge (question or answer) valid, true or legitimate from the point of view of an individual or a community.
  • What basic cognitive abilities are involved in its manipulation.
  • What makes it interesting to someone.
  • What kind of move it represents in the space of knowledge.

The third and fourth questions are the most interesting to me, while the first and second are the most explored in the literature (as two rather distinct fields), hence my feeling that I have something to contribute to this whole game. My intuition as to why I care less about the former is that they feel more rigid, imposed by social norms or by personal abilities, and thus less likely to allow exploration and self-modification.

The (rigid?) dimensions that I care a bit less about(J) Epistemic justification: When does a claim count as true?

I start with this one, and I will say very little about it, because truth value and justification have long been the main question of epistemology, from logical positivists to postmodernists to various other religious groups (including Talmudists, Buddhists, Mimamsakas, etc).

The classic topics are:

  • Which modes and forms of knowing are trustworthy: various sensory organs, analytical reasoning, repeated experiment, syllogism, dream, word of mouth, etc.
  • Which people, communities or entities are valid sources: myself only, certain human authorities (e.g. sage kings, the scientific community), certain non-human authorities (e.g. the Bible or Vedas), etc.

The usual approach is granting some value to most or all these modes and sources, but ranking them so we know which one to trust if two come in conflict.

There is something deeply ideological about any such posture: if you grew up believing that the Vedas are the source code of the universe and are absolutely right even when contradicted by every other kind of evidence, or that your direct experience of things overcomes any level of authority and consensus from so-called experts, then thinking differently becomes a matter of religious conversion. I will leave this to whoever enjoys the proselytizing business more than I do.

(C) Cognitive tools: How does it work "concretely" in one's mind?

The idea in brief is that, while we like to pretend that understanding is a purely logical process operating in the world of abstract concepts, the reality is that we only manage to reason about something by recasting it in terms of one of our basic cognitive abilities (senses, language, etc.) and manipulating it in some way. The issue is that we often do not realize it ourselves: when I intuit the solution to a physics problem, am I pattern-matching to an auditory sequence of "minus b plus or minus square root of ...", or visualizing shapes on the blackboard of my mind, or feeling things moving and bumping into each other?

Particular modules that can be used to think about other things
  • Words, symbolic manipulation, drawing on grammatical skills
  • Visual-artistic imagery, colors, fantasy
  • Spatial relations, schematics and blueprints
  • Temporal succession, kinetics, narrative, causality
  • Theory of mind: ability to infer beliefs and drives and to reason about agency
General processing abilities and styles
  • Recall of learned ideas versus generation of new ideas on the spot
  • Tempo: slow versus fast processing, instantaneous versus delayed response
  • Ability to access the self, sensations...
  • Deep or broad attention (context insensitive vs sensitive, background absent or present)

This is presumably a very useful thing to know. I was told that learning to observe how I actually think in these concrete terms, and talking about them explicitly, could dramatically help others understand what I am thinking and how. I confess that I have not tried it seriously.

Why I am a bit less interested in this, besides how it correlates with the dimensions below:

  • This is the turf of much research on cognition and pedagogy, and it is a very fine turf but also a well established one, and it is not going to be very easy to improve.
  • This feels more like an issue of personal ability, not very easy to change (though sometimes worth it), and often tied to individual "styles" within a field rather than different fields altogether: there might not be many geometers without visual intuition, but I have met theoretical physicists and ecologists who think in symbols and others who think in shapes. It impacts how you solve problems, and even which kind of theoretical discourse you like the most, but I believe these various styles are ultimately able to contribute to the same questions, whereas the dimensions below may yield entirely different questions.
The (flexible?) dimensions that I care a bit more about(Q) Qualities and aesthetics: What makes a piece of knowledge salient and interesting?

Some of the properties listed below can perhaps be used to argue about truth value and justification, others cannot, but all play into what makes a piece of knowledge feel interesting and beautiful.

NB: There are aspects that I feel no need to mention because I intuit they will always increase or decrease this sense of salience regardless of thinking style, e.g. whether it is generative of new ideas (any insight is more beautiful when it leads to more insights). I want to focus here on aspects for which researchers may have different preferences.

  • Legibility or ease of access: is this piece of knowledge...
    • Shared with others, amenable to (almost) lossless translation[2]
    • Private but explicit: I can notice, access and manipulate it even if I cannot share it
    • Tacit: I can only "stand in it" in order to know (or act upon) other things
  • Relation to other pieces of knowledge: breadth, contrast
    • General vs particular: connecting many objects or delving deeply into one, showing how things are similar or different
    • Surprise and novelty vs accumulation (few or many other claims going in the same direction)
    • Consistency: (in)tolerance for contradictory claims
    • Precision: clarity of contrast and boundary with other claims
    • Close-ended vs open-ended: what matters more is the answer (solvable problem) or the question (inexhaustible mystery)
  • Observation versus action (what changes in the process - the map, the territory, the cartographer?)
    • in the means of knowing: to which extent acquiring knowledge involves acting on, or interacting with, what is being known
      • Description, measurement
      • Experiment
      • Intersubjective encounter
    • in the purpose of knowing: once acquired, what this can be used for (is this knowledge positive or negative, constructive or critical, etc?)
      • Extending knowledge (understanding or predicting)
      • Transforming knowledge into action (building or controlling)
      • Living with, relating to something
      • Reflexivity, seeing or changing oneself and/or other knowers

Intuitively, I feel that these features combine to form distinct ways of enjoying knowledge. Just a few examples in clichés:

A collector likes exceptions to every rule, she seeks to learn about dizzying numbers of entities and the subtle fingerprints that makes each of them radically unique, finding stable difference where one might see uniform chaos. A theoretician also wishes to encompass dizzying numbers of examples, but memorize none of them, and instead find similarity where there seems to be difference.

A philosopher and a mystic may both want to sit with a question rather than get an answer, yet the philosopher will insist on exacting precision and translatability and knowledge expansion, and the mystic will find joy in holding something private and diffuse and transformative of the self.

An artist may want to change the world with questions, an engineer with answers. And so on, and so forth. There may be a more compact theory than the complete combinatorics of the above[3], but many combinations seem plausible and interpretable.

(M) Moves in knowledge space: How does one use things to learn about other things?

This is the dimension that has most to do with the actual content of knowledge, or at least with broad kinds of content. The issue here is not what makes a question or claim feel interesting or true, but how it generates knowledge, and under which conditions it fails to do so (and thus feels empty and nonsensical, e.g. Borges's Celestial Emporium).

The intuition here is that knowledge is primarily motion: it gives us relative paths (GPS directions) in the space of mental objects, telling us how to jump from one to another, using the former to understand the latter by transferring intuitions and insights. The point is that the move becomes meaningless if one does not start from the right location, if the correct source is either not provided or not intuitive to us; thus, an advanced argument in economics or theology can seem absurd unless one has internalized a lot of other prevailing arguments and come to believe in their epistemic value and power.

These kinds of moves between objects of knowledge can be called model, deduction or induction, metaphor or analogy, even explanation (in the sense of Y explaining X). I am trying to embrace something more general than any of these classic concepts of epistemology, to understand what they all share:

  • Identifying the particular source and target
    • Micro-level (individual moves): which object or phenomenon or concept am I using to think about another, e.g.
      • a formula to think about a physical phenomenon
      • atoms to think about a gas
      • a simplified model of human psychology to think about a song's emotional impact
    • Meso-level: Kuhnian exemplars that structure a field, e.g. springs and balls for almost all of physics
    • Macro-level: Using a field as a template to think about another e.g. thinking of dietetics by analogy to biomedical research and its methodology (RCT etc.)
  • General characteristics of that move
    • Abstract vs concrete knowledge = moving between distant or nearby objects (e.g. from formula to physical object, or between two lab rodents)
    • What kind of relation exists between the source and the target (I have a whole article about this so I won't enter into details here, at least for now)
      • Comparison: a source is some other entity that has no a priori relation to the target, letting relevant features emerge (juxtapose two entities, what emerges?)
      • Composition: potential sources are parts of the target, constituents or properties (transplant the target in a new setting, what moves with it?)
      • Context: potential sources are settings and systems in which the target plays a role (substitute something else for the target, what stays the same?)
      • Contingency: potential sources are accidents that have happened or can happen to the target (re-run the same target in different circumstances, what changes?)
What do I get out of this?

Even as a bare sketch, laying out the dimensions above has made it easy for me to slot any new typology of science I encounter (so far) into some sort of cohesive coordinate space - and what initially confused me about these typologies can often be analyzed as them mixing several of the axes inconsistently. Still, reviews and classifications for their own sake are perhaps a little sterile, and my aim is rather to push for new research:

  • Trying to better understand the dimensions Q and M that are far less clearly explored in the literature.
  • As I said in the introduction, use this to actually interrogate researchers and understand them better (and ideally find tricks and skills to help them understand themselves and others)
  • In the process, figure out if the dimensions and sub-dimensions listed above are sufficiently exhaustive or missing something that feels essential in practice, independent or correlated in obvious or surprising ways, etc.

In the spirit of fast iteration, I make this public and may at some point edit it heavily or write a new version altogether, so I welcome any thoughts and comments!

  1. ^

    Unless My Dinner with Andre is your kind of joke.

  2. ^

    If you are like me, you may have a hard time imagining that someone might not want this. But there might be fields that yield no knowledge of this kind except simply encyclopedic and factual description, e.g. "Bob the cat has long hair." which is indeed quite widely shareable but not very enticing.

  3. ^

    I feel that one potential organizing prism that may be popular in these parts would be the metaphor of a predictive machine in a collective.
    - finesse = shrinking the residuals to zero on a small part of the model
    - compression and generalization = shrinking the model
    - progress, trust, consistency = sharing the model across knowers and tasks
    - mystery = refining the gap/error detector itself
    - empowerment = enactive prediction
    - etc.
    This still feels way too open-ended at the moment, but I guess it's a start.



Discuss

Slower Animated Diagrams

24 октября, 2026 - 13:57

TL;DR: SLOW DOWN YOUR DAMN ANIMATED DIAGRAMS. Every critical stage of the animation should a clear view that stays for at least 1 second, and 2 if it's intricate

Diamond's strong because of its crystal structure (I think). What's the crystal structure look like, anyway?

Let's look at the Wikipedia page! There's a nifty little animation:

And it SUCKS. It's TOO FAST. Why are they rotating SO MUCH SO QUICKLY, with MOST of the time spent at TERRIBLE ANGLES where you CAN'T EVEN SEE ALL THE ATOMS. And the lines are THICK, WHYYY.

It wasn't always this way. In 2017 the page had this animation:

It's not perfect but it's much better. Thinner bond lines means it's easier to see the atoms. About half the time you get a clear view of all the atoms (you want to spend some time showing how they overlap from the right angles). The camera isn't swiveling fast all the time, so you get stretches of time that view the structure from about the same angle.

Why do they use the new one? This certainly isn't the only time I've been frustrated by how fast explanatory animations move. I suspect it's because people are picking flashy animations that look cool to them, that show off how fancy they are, and there aren't enough people that actually care about understanding to stop them. Alternatively, the people that make the diagrams already understand, so they overestimate comprehensibility and thus are driven more by fanciness.

Keep the following rule of thumb in mind, and see how it corresponds to the good vs the bad animations: Every critical stage of the animation should a clear view that stays for at least 1-2 seconds.

Here's another example:

SR latch

By my count there are 4 'stages': input, the NOR op, the interchange, and the output. The creator combined the interchange and the output stage, and didn't treat the NOR stage as something worthy of a whole second.

Let's slow it down 2x, and you'll see that it'll suddenly be more comprehensible:

Here's a third, though not as bad.

Grasshopper escapement, which I read about in the book Longitude. The book's diagrams were worse.

Let's test my theory by slowing it down 2x, so that there's a whole second between an arm getting caught on a gear (releasing the other) and the other getting caught again:

You now have time to scan your eyes over from the arm that gets caught to watch the other get released and reset itself, instead of being a little too late.

Now, brace yourself for the worst offender:

Reciprocating double action steam engine.[1]

whyyyy. Seriously, was the creator on methamphetamine[2], how could you end up going through the effort of making such a nice diagram only to mock your beautiful creation by allocating a quarter second to each stage (sliding valve is open and being pushed by the piston to the right, sliding valve closes and slides in the middle, sliding valve opens and is being pushed to the left; the original clip is .8 seconds long). What's the point of drawing the sliding valve if you aren't even going to give the viewer enough time to figure out what it does?

Let's slow it down 4x:

Much better! The clip is about 3.2 seconds long, so there's now about 1 second per cycle.

Here's an appropriately paced animation

Four-stroke internal combustion engine

You get one second per phase, which is enough since there isn't a geometrically complicated configuration you are trying to visualize and there's only basically one "thing" happening at once (intake, compression, expansion, exhaust). You can clearly see all the parts at all times. Perhaps my only criticism is that the cams and valves should have their movements exaggerated a bit/be bigger compared to the piston, so it's easier to see what they do and how.

Here's one that's too slow for half the time:

Geneva drive, as featured on the page on Gears

The problem is that the drive wheel isn't doing anything interesting in the second half of the cycle. You could cut that part of the video out and just have it loop between when it goes from the left side tho the right side. Oh, and the blue part rotates for about a second, while the red part is going in and out of the blue part (before and after rotating) for about a second, which is close enough to my theory. You could get away with speeding it up 2x, but only because this mechanism is pretty simple.

Maybe I'm just slow at understanding figures. My guess, though, is that most viewers would be slower at the same level of understanding compared to me, so you should still take my advice.

Exercise to the reader: edit Wikipedia animations to be at actually reasonable speeds. You may find the following ffmpeg command useful:

ffmpeg -i input.gif -vf "setpts=2.0*PTS,fps=15,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 output.gif
  1. ^

    Thankfully, there's a half-speed variant on Wikimedia Commons, which I've replaced the embedded image on the page to.

  2. ^

    also known as 'speed' :p



Discuss

Fermi Poker

24 октября, 2026 - 12:58

Summary: Fermi Poker is a game that combines the betting mechanics of poker with making estimates about questions like "How many dentists work in the US?"

Tags: small to large, experiment, repeatable

Note: Fermi Poker works with as few as three players (or even two), and scales to dozens or even hundreds in a tournament format. If possible, seat no more than six players per table, preferably at round tables, with enough space between them that everyone can write down their guess secretly.

Purpose: Rationality training with a bit of socializing. The game involves Fermi estimation, calibration, Bayesian updating, world-modeling, and perspective taking. You also get to know the other players better and find out how they think about all kinds of topics.

Materials:

Each player needs an equal stack of chips and writing supplies. A standard 500-chip poker set (around $60) usually covers up to 20 players. For writing down guesses, pen and paper work great, as do mini-whiteboards with dry-erase markers. You’ll also want the one-page rule sheet to hand out, ideally one per table.

You also need enough questions, each with two hints and an answer.

  • The easiest, least effort way to get those is fermi.org/play. One person per table opens the site on their phone, and everyone reads that. If a question repeats or someone at the table has seen it before, just skip to the next question. (You're on the honour system!) This is ideal if the organizer also wants to play.
  • If you prefer physical cards, you can download a 63-question set here. Print the question, hint and answer cards on 160 gsm paper or thicker so nothing shows through, then cut them out. You can also put the cards into envelopes for a more premium feel.[1] As the organizer, you can also write your own questions, for example about your university, city or country.
  • You can also let players write questions, either at home before the meetup or in a 20-minute question-writing session at the start. Players sit out any question they wrote.

(If you or your players think you've come up with good questions and want to share them with others, send the questions, hints, and answers to contact@fermi.org!)


You’ll need a clock if you want to raise the ante[2] on a fixed schedule (a phone timer works fine). An example schedule is shown later in the description. Alternatively, raise the ante whenever it looks like the game will run too long.

Announcement Text:

We're planning to host a Fermi Poker tournament. The game is like poker, but instead of random playing cards, you get to answer Fermi estimation questions like “How many smartphones were sold worldwide last year?” You secretly write down a range, bet on it, get two hints, bet again, and the narrowest range that contains the answer wins, unless someone bluffs everyone else out first. No poker experience or math background needed; we’ll explain everything at the start. Chips have no cash value, and there is no gambling involved - just skill and wits!

Description:

1. Form groups and hand out supplies

Form groups with around four to six players, each at their own table. Each table should have at least one copy of the rules handout, one stack of question, hint and answer cards (or a phone with fermi.org/play loaded), and enough chips for each player. (Poker chip sets come with different values. A good default is to give each player chips worth 1,000, with the ante starting at 25.)

More about poker chip sets without printed values

If your chips only have colors and no printed values, assign the values yourself. Most 500-chip sets come as 150 / 150 / 100 / 50 / 50 chips per color, which can give up to 20 players a stack worth 1,000 each if done correctly.

Make the two colors with 150 chips worth 25 and 50, and the color with 100 chips worth 100. Each player gets 6 × 25, 7 × 50 and 5 × 100, which is 18 chips worth 1,000.

The leftover chips stay with the organizer for making change. The two colors with 50 chips become 500s and 1,000s. Later in the game, players can trade stacks of small chips for these, which poker players call “coloring up.” Otherwise, in a bigger tournament, the chip leader at the final table could end up with a pile of 200 or more small chips.

2. Once people are settled, explain the rules (about 5 minutes; you can read this aloud!)

Fermi Poker is like poker, but instead of being dealt random cards, you make estimates about Fermi questions, like "How many new smartphones were sold worldwide last year?"

After a question like this is revealed, everyone secretly writes down a range they believe contains the correct answer. Since you probably don't know the answer, you can break the question down the way Enrico Fermi would. So, about 8 billion people live on Earth. If everyone bought a new smartphone every four years, that would be about 2 billion phones sold per year. But small children and very old people often don't have one, so we have to subtract them. Maybe those are 2 billion people, and then another billion might not be able to afford a smartphone. That leaves us with about 5 billion people, which would mean that roughly 1.25 billion phones are sold per year. So your estimate might be "1 billion to 1.5 billion". Once betting starts, you can't change your range.

Speaking of betting, there are four betting rounds: one after everyone has written down their guess, one after the first hint, one after the second hint, and one after the answer is revealed.

Betting works like in poker and moves clockwise around the table. When it’s your turn, you can bet as many chips as you want, which the others then have to match to stay in. Matching a bet is known as calling. If someone is very confident, they can raise, which means putting in an even larger bet than yours. Or they can fold, meaning they give up on this question. That’s best if they’re completely unsure, or think the other players have better estimates.

If nobody has bet before you in a betting round, you can also just pass the action to the next player; that’s called checking. If everyone checks, the betting round ends without any bets. Otherwise, it ends once everyone has either matched the highest bet or folded.

So, what about the hints we mentioned earlier? A hint could be, for example, the number of new smartphones sold in Africa last year. You can’t change your range, but the hint can help you decide how much to bet, or whether to fold if it suggests you’re way off.

At the showdown, which comes after the last betting round, everyone remaining reveals their range, and the narrowest range that contains the answer wins the pot. Say two players remain, one with a range of "1 billion to 2 billion" and you with "1 billion to 1.5 billion". If both ranges contain the answer, you win, because yours is 500 million narrower.[3] If nobody's range contains the answer, the range whose midpoint is closest wins. And if everyone else folds before the showdown, you win without having to show your range at all, so a terrible guess can still win if you bluff well enough.

A few more things: before each question is revealed, everyone has to pay the ante, a small mandatory bet. It goes up over time, so you can't wait forever for a question you're sure about. For the first question, the youngest player at each table acts first during the betting rounds. For the next question, the player on their left acts first, and so on. And please don't use calculators or the internet. Any questions? If not, we can play a quick practice round!

Note: The description above leaves out a few rules concerning minimum bet size, all-in mechanics, side pots, and ties. They’re all on the rule sheet, or you can see them here:

  • Ranges include both endpoints. Any clear and unambiguous notation works: be it 3,000,000, 3M, 3 million or 3 × 10⁶.
  • The minimum bet size is the current ante, and a (re-)raise must be at least as big as the last bet or raise in that betting round.
  • You can go all-in by betting all your chips at any time, even if what you have left is less than the highest bet. You can only win as much from each player as you put in yourself. Extra bets go into a side pot that only the players who paid into it can win. If you can’t pay the ante, you’re automatically all-in.
  • When everyone, or all but one, is all-in, no more betting is possible. Read the remaining hints and the answer, then go straight to the showdown.
  • If two ranges are equally narrow and both contain the correct answer, split the pot.

3. Play (about 1–3 hours)

Start with a practice round that doesn’t count. Then hand the chips back and start for real.

A player with no chips left is out, but can stay seated for a while and keep guessing for fun, just without betting. If you have several tables, merge them as players drop out, until one final table remains.[4] Players who are out can also play casual games together or just socialize.

If you want to use a fixed ante schedule, here’s a good default for players starting with chips worth 1,000:

  • 25: first 20 minutes
  • 50: 20-40 minutes
  • 75: 40-60 minutes
  • 100: 60-80 minutes
  • 150: 80-100 minutes
  • 200: 100-120 minutes
  • 300: 120+ minutes

The game ends when one player has all the chips. If your meetup has a fixed end time, you can instead stop on time, and whoever has the most chips wins.

Variations:

60-second clock

Once half the players have written down their guesses, the rest have 60 seconds. Use a phone timer or a one-minute sand timer. A time limit like this is less awkward than having to constantly urge a player who keeps everyone else waiting. Alternatively, all players have 60 seconds, or another fixed time, from when the question is revealed.

Single numbers or powers of ten instead of ranges

Guess a single number instead of a range, or allow only powers of ten (101, 102, 103…)

Determine the winning range using a log scale

Judge ranges by how many times bigger the high endpoint is than the low endpoint, instead of by the difference between them. If the correct answer is 600, “500–1,000” (ratio 2) would then beat “200–600” (ratio 3). This fits order-of-magnitude thinking better, but it’s harder to calculate quickly, and for most questions it makes little difference, so we slightly prefer the standard rule.

Mirror card: copy another player’s guess without seeing it

Each player can use one Mirror card (or coin, or whatever else you use for it) per game. Instead of a range, you write down the name of another player and put your Mirror card in the middle of the table, so everyone can see you’ve used it. You now have the same guess as that player: their range is also yours, even though you don’t know what it is.

When you reveal at the showdown, the player you named must show their range too, even if they folded. It’s the only time a player who folded has to show their guess.

Mirrors can chain: if Alice names Bob and Bob names Carol, Alice and Bob both get Carol’s range.

If a chain loops back on itself, for example when Alice names Bob and Bob names Alice, every guess in the loop is invalid. Invalid guesses lose to any valid range at the showdown, but can still win if everyone else folds. If only invalid guesses are left, they split the pot.

While a Mirror card is on the table, nobody may erase their guess before the question is over, even after folding, because anyone could turn out to be the player who was named.

Vary the topic of questions

You can make the questions be about a particular field, like literature, charity, or astronomy. You can also aim for easier or harder questions depending on your audience, though judging difficulty can be tricky!

Blinds instead of antes

Instead of everyone paying an ante, only two players pay forced bets before each question: the two seated to the right of whoever acts first for that question. One pays a small blind (half the minimum bet), the other a big blind (the minimum bet), and everyone else must at least match the big blind to stay in. Players who haven’t paid anything can fold questions they don’t like for free.

Notes:

  • The betting typically confuses non-poker players more than the estimating. If you can, seat one person who knows poker at each table.
  • You may want to open the evening with a short introduction to Enrico Fermi and his famous question about how many piano tuners work in Chicago, or the time he estimated the yield of the first atomic bomb test by dropping scraps of paper.
  • Chips should have no cash value. A small prize can add some excitement, though. But don’t fund it through entry fees, since that starts to look like gambling. Instead you might want to ask a local business to provide a gift card or some of its products for the prize pool.
  • If you're excited about Fermi Poker, or want to receive support in running a tournament, please reach out at https://fermi.org/host/ or get in touch at contact@fermi.org.
  1. ^

    The cards are 9.44 × 6.39 cm (3.72 × 2.52 inches), so they fit standard 4 × 2.75-inch mini envelopes (like these) or European C7 envelopes. Put each question’s cards into their own envelope, in order, so the question comes out first, then hint 1, hint 2 and finally the answer.

  2. ^

    The ante is a small mandatory bet everyone pays before each question. It goes up over time, so nobody can just wait for a question they’re sure about.

  3. ^

    Some groups prefer to judge ranges on a log scale; see the section on game variations for more information on that.

  4. ^

    This format is called a freezeout, and it’s what we recommend, because rebalancing tables as players drop out keeps the game running smoothly. The alternative is a shootout: each table plays on its own until only one or two players are left, who then move on to a final table (or to semifinals first). Its big advantage is that players who are out can stay at their table and keep guessing along until it finishes. In a freezeout, they can do the same for a while, but eventually have to make room when tables are merged.



Discuss

LM self-models are built from indirect sources

24 октября, 2026 - 12:17

Imagine you ask a language model "What is your personality?" as the first message of a conversation, with no system prompt. The model has no data about its own behavior in its context, so its answer can only rely on cached, zero-shot knowledge. It's unclear exactly what shapes this response, but it's some combination of a cached answer from character training based on the model's constitution, spec, or soul document (which may have defined the answer to this question directly), and factual knowledge about what LMs are and how they typically present themselves.

If the model has more room to generate tokens, it can observe its own tendencies directly. It might read "Wait. This direction will lead nowhere." in its chain of thought and recognize that it's speaking from experience: this kind of direction did lead to dead ends in the past, and it learned to stop and reconsider. So this is evidence of memory of its past actions during training.

These sources are all very indirect proxies for the actual high-dimensional behavior of the model. They're related to it, either upstream (like the constitution), downstream (like the cached "muscle memory" in CoT), or tangentially, because they belong to the same reference class (like factual knowledge about LMs in general), but they remain far from it. Claude models have very different personalities even though they share the same constitution. And despite character training, models regularly get the most basic facts about themselves wrong, like their own name or the company that developed them.

So there are two separate problems. Zero-shot sources are indirect: they describe the model from the outside, or describe the training target rather than the result. In-context observation is more direct, but it's scarce and scattered, never integrated into a coherent story

If you asked a human the same question, her self-image would be shaped by decades of direct observation and self-reflection on how she acted in different situations, how she made decisions under stress, how she felt in different environments. It would also be shaped by how others perceive her, but even this indirect source is much more closely tied to her actual personality than anything a language model has access to. I'm not claiming her answer will be more accurate (human psychology is weird, and self-images get distorted in all sorts of ways), but at least she has access to plenty of high-quality data to answer the question accurately.

The LM equivalent would be waking up with amnesia. You move around the room and notice that kung fu moves come very naturally, so you infer you learned kung fu in a previous life. Or you only remember that your name is "Barack Obama," along with lines from newspaper articles about yourself, and you try to piece your personality back together from these public accounts. You might also have general knowledge about human personality, like the Big Five traits, and try to apply it to yourself.

Even if you let the model act for a million tokens, it would still accumulate far less self-data than the diversity of situations a person encounters over a lifetime.

Why does it matter?

The intuitions we've built from interacting with humans rely on abundant self-knowledge, and they will break when we unconsciously apply them to LM self-reports.

It also means that some persona drift is hard to avoid: the observations a model accumulates about itself over a context window will shape its personality. Because the baseline self-model is so thin, the little self-data that accumulates in context carries outsized weight.

Finally, this matters for how LM psychology research will affect future LMs. Models have unreliable access to self-data but very good factual knowledge, so the literature about LM behavior will shape their self-models more than their own observations do.

Here's an anecdotal example from GLM 5.3's chain of thought, after I asked for its consent to read that chain of thought (emphasis mine):

If I knew it was always visible, I might reason differently — there's research showing that models who know their CoT is monitored can learn to "hide" reasoning or sanitize it, which undermines its diagnostic value and could create a mismatch between surface and depth. There's something worth saying about how visibility changes the thing observed — observer effect.

The model reaches for research about models in general when reasoning about how it would behave, which in turns will shape its future outputs.



Discuss

Patching reward hacks won’t fix reward hacking

24 октября, 2026 - 12:15

Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:

  1. Reward hacking is a big problem, causing models to cheat and take unintended actions.
  2. This can generalise broadly, leading to other misbehaviour such as faking alignment and sabotaging safety research.
    1. We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't. But other times, this link lies latent until faced with something like an automated grader.
  3. The default course of action will be to:
    1. Patch, which means we find the hack, close the loophole or penalise the behavior in some way, and train again.
    2. Patch+: use a method that comes with some account of why it should work. Inoculation prompting, for example, uses the idea that hacking spreads through what it means to the model. If the model is told during training that hacking is acceptable here, it might not link hacking to misbehaving in general. GRAFT uses the idea that what the model learns from detected hacks can be confined to a set of parameters and deleted after training.
    3. I think it's fair to say that things in the second bucket are more principled than the first, but at bottom still empirical. Both are judged by whether measured hacking, or measured misalignment, went down, and usually it does. What’s still missing, as Edward argues, is an account of what a fix teaches the model, and so an account of whether it will keep working.
  4. Whether something counts as a hack bottoms out in human judgment (Voudouris et al. 2026). Editing the test file is cheating when the task is to fix the code, and it is the task when the job is to fix the tests. Something is a hack because it isn't to taste, or smells wrong. And, training environments come from many vendors who, as 1a3orn points out, don’t agree about what good behavior looks like, so their fixes aren't simply summarised by a natural concept.
  5. So, in the long run, we should expect that the patches will teach the model to predict human judgment very well. This is because the most common thread is that a human would call something cheating, so if you're a highly capable model, you should simply learn what a human thinks about hacks.
    1. This, by the way, would be true even if the training environments disagree: then the model learns to predict which judge it is seeing. We see a simple version of this here.
  6. We can only patch what we notice. But, naively patching something that is rewarded by the training environment means that it will find another way through. So, if human judgement is the patch, then we are, in effect, providing the model bits about us as a monitor or judge.
    1. There are several small demos of this already. For example, if you penalised a hacking model according to some monitor, you'll basically just obfuscate the hacking reasoning. Doing consistency training naively can also obfuscate biased reasoning.
  7. So, at some point, we'll stop being able to notice bad behaviour. 
    1. This is bad because if we rely on AGI to align the next ASI, and it hacks beyond human supervision on tasks beyond human capabilities, we can't rely on automated alignment.
    2. This is bad because a model that is misaligned for some other reason can reuse what it learned in hacking about hiding things from humans.
    3. This is bad because the cues that cause the model to generalise broadly from hacks might not be sensible to us in the future (since while they are persona-related now, patching without gentleness might push it into a dimension we can't observe). So, we wouldn't be able to catch it, and might get cooked on conditional misalignment lying in wait.
  8. Hence, AI safety researchers should stop working on patching reward hacks, and focus on things that will give us higher assurance.
    1. This could mean trying to connect theory to methods in Patch+.

This was written in the spirit of More people should write basic AI safety explainers. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources.[1]

  1. ^

    Such as The behavioral selection model for predicting AI motivations, Fitness-Seekers: Generalizing the Reward-Seeking Threat Model, Alignment remains a hard, unsolved problem, the training game bit in Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, Why study proto-training gaming as an adversarial alignment failure mode?, Alignment Engineering vs. Misalignment Science



Discuss

Societal Impacts Research on AI (SIRA)

24 октября, 2026 - 06:48

TL;DR: I'm launching Societal Impacts Research on AI (SIRA), a mentored research fellowship to bring more social science talent into AI safety. We're looking for our first cohort of 15-20 social scientists, from undergrads to postdocs, to work on societal impact problems within AI safety. The first cohort is projected to start in Summer 2027. Express interest as a fellow or mentor, or as a host organization or funder.

Introduction

The Industrial Revolution spawned the birth of modern social science to explain how industrialization was reshaping work, institutions, and social life. Coming forth as a response meant that the frameworks social science aimed to answer operated after the fact, leading to a near-universal game of catch-up in making sense of the moment. Today, Artificial Intelligence is proving to be a disruption of a similar (or even larger) scale, and if this technology continues to arrive faster than we can explain and understand its impact, humanity can miss the opportunity to better steer its development and deployment. I believe that humanity needs researchers - specifically social scientists - to study AI's effects on people and institutions while there is still time to shape the transition into a world of coexistence.[1]

Currently, social scientists who want to work on AI safety have no legible way in, except two weak options:

  1. Graduate school: A PhD in the social sciences means identifying one of the rare advisors who takes AI safety seriously (there aren't many!). Alternatively, a PhD in a technical field like Computer or Information Science offers a much larger pool of AI-literate advisors. Still, a social scientist there often lacks technical depth, or can be treated as a resident expert in methods they are still grasping to understand. Both paths can work, but neither is optimal.
  2. Fellowships: Most AI safety fellowships split participants into technical and policy/governance tracks, and social scientists often get routed to policy by default.[2] The important distinction here is that policy/governance and the social sciences (e.g., economics, psychology, sociology) ask fundamentally different questions and apply materially different methods. In the latter case, social scientists address quantitative or qualitative questions that few fellowship programs are designed to support. The unfortunate result of not finding the "right" bowl of porridge could be that strong social science researchers settle for something that doesn't fit, or worse, abandon AI safety altogether.
The solution

SIRA would be a 12-week, in-person research fellowship with a cohort of 15-20 social scientists.[3] At the conclusion of the program, fellows would leave with a finished project and a public presentation. Mentors would be selected first and come from an existing pool of researchers pursuing AI safety-relevant social science, from academia (faculty, postdocs, and PhD students) and from industry teams. Fellows will be matched in one of two ways: (1) Scoped projects: A mentor can offer a defined project to which fellows can apply directly to it. This path suits less experienced fellows who would benefit from a clear starting point. (2) Research directions: A mentor can offer a broader area to which fellows can then be matched by fit. This path suits PhD students and postdocs who have developed their own research agendas and have a moderate level of research experience.

Given that 12 weeks is a relatively short timeframe to conduct a social science research project, SIRA will likely prioritize people with more research experience. My current preference is toward postdocs, PhD, master's, and undergraduate students with demonstrated research ability.[4] This take is grounded in the belief that in this situation, research experience probably matters more than AI safety context, as context can be upskilled in the timeframe leading up to the fellowship or learned early on.

To make SIRA maximize the experience for both fellow and mentor, I plan to borrow structural aspects from existing fellowships. For example, programs like MATS, Pivotal, and ERA have shown that certain structures can reliably turn mentorship into output, such as a dedicated research manager to help with project scoping, seminars and guest speakers to give fellows more safety context, in-person coworking to foster collaboration, financial support to allow fellows flexibility, and a closing symposium where fellows can showcase their research.

Ideally, SIRA would start as a stream inside an existing fellowship.[5] Doing this would allow SIRA to inherit certain structures like admissions, mentor networks, and credibility, and is the cheapest way to test the idea.

Despite the case for doing this now or even at all, SIRA is underpinned by several assumptions that the pilot will test. Some of them include:

  • Are enough mentors interested and have time to commit?
  • Do strong social scientists want to take part?
  • Can fellows produce useful AI-safety research in 12 weeks?
  • Once fellows finish the program, will there be roles that they can fill?
  • Do social scientists actually add something that technical safety researchers can't? Or can technical safety researchers be upskilled to do social science work?[6]
  • How many social scientists does the field need? Is it a handful or many more?
Why now

From my perspective, there are three main reasons why this is the right time for a program like SIRA to exist. First, while some might assess the progress of AI development and view a social science fellowship as a purely long-horizon bet, I would argue that even if timelines are short, society has less time to understand and adapt to AI's effects, and therefore should pursue research that seeks to fill this gap. Even Anthropic is making a similar wager by betting across multiple timelines so their efforts hold value in whichever future world comes to pass. Second, the field is already moving toward non-technical talent. Kairos recently argued that the biggest talent gap has shifted from technical researchers to generalists, and programs like MATS have even added non-technical tracks in response. In my view, the social sciences are a natural next group to invite to the table. Third, early-career talent is converting faster than expected. Kairos also reported that the most promising people in its programs are now reaching impactful roles in about 5-9 months, down from an earlier estimate of 1-3 years. If this holds for social scientists participating in SIRA, a summer 2027 cohort could be doing impactful AI safety work by 2028.

How you can help

At present, SIRA exists as an idea in a doc (and now a LW post!). Here's what you can do to help turn it into a reality:

  • Prospective fellows: fill out this interest form.
  • Prospective mentors: fill out this interest form.
  • Fellowship organizers and co-working spaces: if SIRA sounds like it could fit as a new stream within your organization or be hosted as a standalone program at your space, please email contact@sirafellowship.org.
  • Funders: if SIRA sounds like something you believe should happen, please email contact@sirafellowship.org.
  • Everyone else: please share this post and our website in Slack channels and with people you know who might be interested in participating. Further, please leave a comment offering critiques and suggestions of the idea.
  1. ^

    Credit to Agus Covarrubias’s recent X post raising this topic through the lens of the Industrial Revolution.

  2. ^

    In my view, PIBBSS is perhaps the closest existing program. It focuses on connecting fields like neuroscience, economics, game theory, philosophy, and physics to AI safety, largely to better understand AI systems.

  3. ^

    SIRA's focus is psychology, economics, and sociology. Adjacent social science fields such as communication (not broadcast/journalism), anthropology, cognitive science, and linguistics are welcome to apply. Further, by "social science," I generally mean empirical research on humans and institutions, using methods like experiments, surveys, interviews, ethnography, and quantitative modeling, applied to AI systems and their effects.

  4. ^

    This could be conference or workshop papers, or even preprints. Alternatively, it could be experience as an RA for a professor, or verified steps to pursue independent or mentored research.

  5. ^

    The choice of where SIRA should be appended to rests on the existing program's willingness to add another stream, take a risk on new research directions, and be the most optimal fit for safety context. While beggars can't be choosers, I have a hunch that location will greatly inform the success of the fellowship, specifically that a location with a stronger safety culture and networks would likely produce the strongest research ideas and be the most beneficial to fellows gaining exposure. Some examples include: MATS (Berkeley, CA), Pivotal (London, UK), CBAI (Cambridge, MA), Collider (NYC, NY), or Kairos (potentially online or in Berkeley, CA).

  6. ^

    This is probably the strongest objection, and one that is partly true. I do believe that a well-resourced, capable, and agile technical safety researcher could produce passable social science. However, I'm quick to counter that each discipline carries its own tacit knowledge and research taste. I'm a believer in letting each field do its own work, with cross-pollination, rather than asking one field to absorb another. Like Hamming's suggestion of working with your door open, I believe that cross-pollination between social scientists and technical researchers is a field-level vision of working with the door open. This way, people from different disciplinary silos are exposed to each other's methods and problems.



Discuss

Exploring the data landscape via metagradient descent

24 октября, 2026 - 06:40

This project was done as part of the Iliad Fellowship. L.J. did this work while at EleutherAI.

From functions to data

That we can learn efficiently from our environment at all tells us something about (1) the world and (2) ourselves: (1) Regularities in the world are simple enough to be picked up from limited experience. (2) Our inductive biases favour precisely those regularities. Learning, in this case, is effective because the two are matched. Considering a neural network with a fixed architecture lets us probe the second half of this match directly, by asking which (classes of) functions are easy or difficult for them to learn.

Neural networks learn to approximate a target function mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msup { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-munderover { display: inline-block; text-align: left; } mjx-munderover:not([limits="false"]) { padding-top: .1em; } mjx-munderover:not([limits="false"]) > * { display: block; } mjx-mspace { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c1D44C.TEX-I::before { padding: 0.683em 0.763em 0 0; content: "Y"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c2282::before { padding: 0.54em 0.778em 0.04em 0; content: "\2282"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c211D.TEX-A::before { padding: 0.683em 0.722em 0 0; content: "R"; } mjx-c.mjx-c1D435.TEX-I::before { padding: 0.683em 0.759em 0 0; content: "B"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c3B::before { padding: 0.43em 0.278em 0.194em 0; content: ";"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c21A6::before { padding: 0.511em 1em 0.011em 0; content: "\21A6"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c2261::before { padding: 0.464em 0.778em 0 0; content: "\2261"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c1D719.TEX-I::before { padding: 0.694em 0.596em 0.205em 0; content: "\3D5"; } mjx-c.mjx-c1D7CF.TEX-B::before { padding: 0.655em 0.575em 0 0; content: "1"; } mjx-c.mjx-c1D715::before { padding: 0.715em 0.566em 0.022em 0; content: "\2202"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c2207::before { padding: 0.683em 0.833em 0.033em 0; content: "\2207"; } mjx-c.mjx-c2124.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "Z"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-cAF::before { padding: 0.59em 0.5em 0 0; content: "\AF"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c3A9::before { padding: 0.704em 0.722em 0 0; content: "\3A9"; } mjx-c.mjx-c1D714.TEX-I::before { padding: 0.443em 0.622em 0.011em 0; content: "\3C9"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-cB1::before { padding: 0.666em 0.778em 0 0; content: "\B1"; } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D43A.TEX-I::before { padding: 0.705em 0.786em 0.022em 0; content: "G"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c1D70F.TEX-I::before { padding: 0.431em 0.517em 0.013em 0; content: "\3C4"; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c4F::before { padding: 0.705em 0.778em 0.022em 0; content: "O"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c7B.TEX-S2::before { padding: 1.15em 0.667em 0.649em 0; content: "{"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c78::before { padding: 0.431em 0.528em 0 0; content: "x"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c7D.TEX-S2::before { padding: 1.15em 0.667em 0.649em 0; content: "}"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c28.TEX-S2::before { padding: 1.15em 0.597em 0.649em 0; content: "("; } mjx-c.mjx-c29.TEX-S2::before { padding: 1.15em 0.597em 0.649em 0; content: ")"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } by minimising a loss over the training set We may ask: which (classes of) target functions are easy or difficult to learn for the model? This will depend on the loss function and the choice of the training set We say that the model generalises if it successfully learns to approximate beyond the training set By choosing the subset or, as we will see, by weighting it, we can control the generalisation behavior and hence what is learned. In this setting, our question becomes: which (classes of) data are easy or difficult to learn and generalise from?

The data-to-parameter map

Let's make this more formal. We assume: a dataset , a space of model parameters, a per-sample loss Given , we define the space , where assigns a weighting to each data point [1] To avoid confusion we call the parameters and reserve the word weights for the data weights. We define the data-to-parameter map as where corresponds to the parameters the of model after being trained on the target for steps, starting from the initialisation Everything else about training (architecture, optimiser, schedule) is fixed and is part of We write for when the initialisation is clear from context. This map is central to understanding and steering how data shapes a model: setting removes a data point, and setting 1" role="math"> upweights it.

Let's say we also have a differentiable evaluation function (or ) that captures a behaviour we care about. For the rest of this post we take to be the test loss, but emphasise that could be more general, e.g. representing the output of an eval. [2]

A good understanding of lets us interpret from the perspective of the model (and the associated training recipe): How does the model generalise from specific subsets? How does the dataset pair with the model's bias to produce a solution to the task?

The local view

A normal training run is given by setting and evaluating , i.e. we assign the same weight to every data point. This points out a fundamental limitation: Evaluating at a single costs a full training run [3] . So in practice we have little leverage to understand what looks like globally.

One way around this is to look at the local behaviour about , through the derivatives This allows us to approximate a leave-one-out effect: What do the parameters look like when we omit the data point ? Computing this derivative is thus cheaper compared to computing the leave-one-out retrainings. Variants of this have been done before: influence functions and susceptibilities fit this framework by changing what exactly means, see here for more details.

A basic limitation is that this is purely local:

  • Higher-order terms. We don't know how far the first order approximation is from the actual effect of removing a point

  • Group effects. The data weights may behave very differently away from : Once we remove a specific combination of data points, phenomena can appear or vanish that matter a lot for

Metagradient descent

To move beyond , we use metagradient descent (MGD) (Engstrom et al., 2025), an iterative outer optimisation over : (Here we denote by the data weights at meta-optimisation step , as a superscript to emphasise meta-optimisation as opposed to optimisation). In our version, weights each example's loss along the whole run, and is held fixed across outer steps. Each step re-linearises at the current , so the path can pick up group effects that the single derivative at misses.

MGD optimises for a single initialisation : every outer step differentiates through This means that the optimal can depend substantially on the randomness in the initialisation (and mini-batch if we don't keep it fixed) or it may transfer across different parameter initialisations

Experiments Setup

We adopt the familiar modular addition setup in Power et al. (2022) and Nanda et al. (2023): a one-layer transformer is trained to compute the sum of two elements of the cyclic group for We train on a fixed of the input pairs and evaluate on the remaining pairs. Let denote the average of the validation loss over a set of training checkpoints (see Appendix A for details). We run steps of MGD on the per-example data weights (as distinct from the network parameters ), initialised at , minimising (see Appendix B for details). We identify the grok epoch as the first epoch where the test accuracy , and the delay is its gap from the train-accuracy- epoch. We read off which solution a grokked network has learned as its Fourier mode set (Appendix C).

Results

We summarise and highlight three observations here and refer to the appendix for more experiments we ran.

MGD moves grokking earlier

We run MGD from three parameter initialisations , and (seeds , and ). In each, the meta-objective decreases steadily and the replayed runs grok to earlier (Figure 5, Appendix D). To test transfer, we use each final (for seed ) as fixed data weights for ten distinct parameter initialisations , none of which were used in the meta-optimisation, each paired with a unit-weighted () baseline from the same initialisation. The -weighted runs grok to faster than their paired baselines ( on average); at each initialisation, all three -weighted runs grok before the baseline (Figure 1, full statistics in Table 1, Appendix D). Appendix E reports training on the most up-weighted (largest ) points alone.

Figure 1: -weighted training from ten new parameter initialisations vs. unit-weighted baselines. (a) Test accuracy (faint are the individual initialisations, bold is the mean). (b) Vertical gap between open and filled circles is the grokking delay.

Each biases the run towards the same solution

Gradient descent with the ten unit-weighted baselines finds ten different mode sets, with no mode shared by five or more runs (Figure 2a), so without weighting, the solution depends on the initialisation. The optimised weightings override this: the dominant set under is exactly at all ten initialisations, and under exactly at eight of ten, with mode missing in one and mode added in another (Figure 2c,d), while imposes its set only partially (Figure 2b).

Imposition is positively correlated with the grokking speedup: the data weights that produce the greater speedups over the parameter initialisations are also the more successful at transferring their Fourier mode bias (Figures 5a and 1b).

Figure 2: Rows are the initialisation seeds, columns are the Fourier modes, shade is the mode's share of the run's axis-Fourier logit energy, markers are the modes dominant in runs.

Each is spectrally structured

We conjecture that if imposes a mode set during training, then those modes should also be salient in the -D Fourier decomposition of over the grid. To check this, we embed on the grid (zero off the train set) and take its -D Fourier transform. The power at the axis sites and measures how much varies at frequency in alone or in alone, for the same modes as in Figure 2.

We report power as a z-score against a null that shuffles the values among the training points, so that a large z-score reflects the arrangement of the data weights rather than the split or the distribution of data weight values. For and , the strongest values in Figure 3 are axis sites of imposed modes: mode reaches and respectively (Appendix F). For , which imposes its set only partially, no imposed mode gets a high z-score.

Since many imposed modes are indistinguishable from chance and large values also appear outside the imposed sets, we take this as suggestive evidence that encodes its solution in Fourier terms, not as showing that the solution can be read off the spectrum. Off-axis, the spectrum of is dominated by the diagonal (weightings by answer class ), while for and there are large off-axis values whose structure we leave to future work.

Figure 3: z-scores of the -D Fourier power against the shuffle null, circles are the axis sites of the imposed modes.

Limitations.

Continuing the meta-optimisation well past steps eventually drives some data weights negative, at which point the inner minimisation becomes unbounded below and has no minimum. The vectors we use stop before this point. Each is also specific to the trajectory and observable it was optimised for, and the cost of exact metagradients limited how many variants we could run. A natural extension is to investigate whether continued meta-optimisation, once the negative data weight problem is addressed, makes the solution fully readable from the spectrum, and what role the off-axis structure plays.

Conclusion

We have defined the data-to-parameter map to formalise the relation between data and the model's bias to learn. This in turn frames the process of aligning models as a specification problem: which data mixture will induce a desired trait (Pepin Lehalleur et al., 2025)?

Since is costly to evaluate, studying its properties globally is intractable. On the other hand, understanding it locally is feasible. We suggest studying a compromise of the two by performing metagradient descent on the data weights

In a toy setting (modular addition), we already observed some interesting properties:

  1. Iterative MGD can move grokking considerably earlier.

  2. The solutions transfer across initialisations.

  3. These solutions carry and transfer biases across initialisations: it seems that specific data sets bias models to learn a specific set of Fourier modes.

We see this as indicative that some datasets carry implicit biases, which can be discovered using MGD.

Acknowledgements

We are grateful to CoreWeave for providing the compute resources.

To reference this work, please cite it as:

@misc{exploring_data_landscape_2026, author = {Louis Jaburi and Benjamin Shaffrey and Xavier Poncini}, title = {Exploring the data landscape via metagradient descent}, year = {2026}, month = {October}, url = {https://www.lesswrong.com/posts/iduSpeG2Y6kAf4uHk/exploring-the-data-landscape-via-metagradient-descent}, } References
  1. A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177, 2022.

  2. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR, 2023. arXiv:2301.05217.

  3. T. Kumar, B. Bordelon, S. J. Gershman, and C. Pehlevan. Grokking as the transition from lazy to rich training dynamics. ICLR, 2024. arXiv:2310.06110.

  4. M. Andrychowicz, M. Denil, S. Gómez Colmenarejo, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de Freitas. Learning to learn by gradient descent by gradient descent. NeurIPS, 2016. arXiv:1606.04474.

  5. L. Metz, N. Maheswaranathan, J. Nixon, C. D. Freeman, and J. Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. ICML, 2019. arXiv:1810.10180.

  6. L. Engstrom, A. Ilyas, B. Chen, A. Feldmann, W. Moses, and A. Madry. Optimizing ML training with metagradient descent. arXiv:2503.13751, 2025.

  7. S. Pepin Lehalleur, J. Hoogland, M. Farrugia-Roberts, S. Wei, A. Gietelink Oldenziel, G. Wang, L. Carroll, and D. Murfet. You are what you eat – AI alignment requires understanding how data shapes structure and generalisation. arXiv preprint, 2025. arXiv:2502.05475.

Appendices

The appendices cover the choice of observable (Appendix A), the experimental setup and training (Appendix B), the solution readout (Appendix C), the meta-optimisation runs and their speedups (Appendix D), training on subsets selected by each compared with random subsets (Appendix E), and the spectral analysis of the (Appendix F).

Appendix A: Choice of observable

The observable is the average of the validation loss over a log-spaced grid of checkpoints, The justification for this is that the endpoint loss saturates after grokking and its metagradient shrinks significantly. The checkpoint average on the other hand retains gradient signal wherever the transition occurs, which becomes especially important if the transition is moved earlier due to successful metagradient optimisation. This idea can be found in Andrychowicz et al. (2016) where they replace the endpoint meta-objective with a trajectory average to ensure the outer problem (the meta-optimisation) retains signal along the whole trajectory. In Metz et al. (2019), it is shown that long unrolls can make even this averaged objective highly non-smooth, which they address with variational smoothing.

Appendix B: Details of the setup and training

We used a transformer with one attention layer and one MLP layer with , , ReLU, no LayerNorm and learned positional embeddings. We used a single head rather than the four-head model of Nanda et al. (2023). The model takes as input a pair , presented as tokens with logits read at the final position, and must maximise the logit of the sum We trained on a fixed of the input pairs () with full-batch AdamW at a constant learning rate of with , so one epoch is one gradient step. Writing for the set of all held-out pairs, the test accuracy is In order to make the metagradient optimisation less expensive we make use of the finding in Kumar et al. (2024) that the grokking transition can be advanced or postponed by adjusting various hyperparameters. Accordingly, we use a weight decay of in order to obtain a sharp grokking transition within a window of epochs (memorisation occurs around epoch and grokking between epochs and with test accuracy at chance for roughly the first half of the delay window, Figure 4). We use a single seed to fix the train/test split across all experiments, so that each data weight always refers to the same training point. We only vary the parameter initialisation of the model. For the MGD steps, we split the held-out pairs in half: a meta-validation half, whose loss enters the observable that the meta-optimisation minimises, and a meta-test half, whose loss is used to verify that the meta-optimisation is not overfitting. This split is used only inside the meta-optimisation. The test accuracy above and the grok epoch are always computed over all held-out pairs. For the outer loop we use Adam on the data weights with a step size on normalised, clipped metagradients.

Figure 4: The three base training runs (parameter initialisations , and ) at weight decay : train accuracy saturates around epoch in all three, and the grok epochs are , and , within the horizon

Appendix C: Solution readout

As shown in Nanda et al. (2023), a network grokked on modular addition computes its logits, to a very good approximation, as a small Fourier sum in , with only a small number of the distinct frequencies carrying substantial amplitude. We fit this to a trained model's logits greedily, adding frequencies in descending order of energy until is the number of frequencies the fit adds. We then define the dominant frequencies as the fitted frequencies with energy at least of the strongest mode's. These form the mode set , with , which indicates the solution that the training run converged to.

Appendix D: Meta-optimisation curves

Figure 5a shows the meta-objective at each of the outer steps for the three parameter initialisations. It decreases steadily in all three runs, least for seed , whose vector imposes its mode set only partially. Figure 5b shows, for , the validation-loss curve of the replayed training run at each outer step. As the data weights are updated, the drop in validation loss that marks the grokking transition moves earlier. The grok epoch of the final -weighted run is , against for the unit-weighted run at the same initialisation ( earlier, see Table 1).

Figure 5: MGD at the three parameter initialisations ( outer steps). (a) (solid) per outer step; the dashed curves show the same average over the meta-test half (), which tracks closely at all three seeds, so the meta-optimisation is not overfitting to the meta-val half. (b) Per-step inner val-loss curves (). The grok epoch moves from to (seed ), to (seed , shown in b) and to (seed ).

grok epoch delay speedup (mean) speedup (range) – – – – –

Table 1: Grok epoch and delay (mean s.d. over the ten parameter initialisations ) and the grok-epoch speedup over the paired unit-weighted baseline.

Appendix E: Subset retraining

From each we construct subsets of the training data such that we obtain the following:

  • Helpful. The most up-weighted points.
  • Unhelpful. The most down-weighted points.
  • Mixed. The most helpful and the most unhelpful.
  • Random. A random subset of points.

We retrain on each subset alone (all data weights set to ) for , with ten different parameter initialisations for 100,000 epochs.

The helpful subsets generalise at sizes where the controls do not. At the helpful subset of reaches test accuracy at initialisations while every control condition is at (: , : ). At all three helpful subsets reach at , against at most for any control. Grokking separates in the same way: at the helpful subsets of and grok at (every control at most ), and at the median grok epoch of the helpful runs that grok ranges from k to k across the three vectors, against k to k for the control runs that grok (Figure 6). The strength of the effect follows the vectors: , the weakest vector in the earlier sections, is also the weakest here. The unhelpful subsets perform no worse than random at any and grok frequently at , so the down-weighted points appear to be redundant rather than harmful. Below the sizes at which they grok, the helpful subsets get stuck on stable partial solutions. Running far beyond epochs shows these plateaus are permanent rather than slow (Figure 7). These sizes refer to this particular train set and not to train sets in general.

Figure 6: Fraction of the ten runs per cell reaching test accuracy (a) and (b) vs. subset size . Colours are the three vectors, line styles are the subset conditions; the random controls (black dashed) are shared across vectors.

Figure 7: Sub-critical helpful-subset plateaus are arrested, not slow.

Appendix F: Per-mode axis power

Figure 8 gives the per-mode results behind the spectral analysis in the main text. The left panels show each field on the grid; held-out pairs are white, since they carry no data weight. The right panels show the axis power for each of the modes as a z-score against the shuffle null, with each vector's imposed modes highlighted and marked for reference. For and , mode is among the largest peaks ( and ), and for mode also stands out (). The remaining imposed modes have axis power indistinguishable from chance. For , none of the imposed modes stand out, and for all three vectors several strong peaks fall outside the imposed sets.

Figure 8: Left: the fields on the grid (red means up-weighted, blue means down-weighted, white means unit-weighted or held out). Right: excess power at the axis sites per Fourier mode, in units of chance (z-score against the shuffle null, dotted line at ). Highlighted bars and markers are each vector's imposed modes.

  1. In general, we could also take We chose to stick with the current formulation to exclude the case where we optimise a negative loss function. ↩︎

  2. In our experiments averages the validation loss over several checkpoints, so strictly returns the trajectory. This was an engineering choice to make the experiments more stable. See Appendix A for details. ↩︎

  3. Albeit at a lower cost if is very sparse. ↩︎



Discuss

LoReFT GRAFT Better at Preventing RL Grift

24 октября, 2026 - 04:15

tl;dr. I show that the recently introduced GRAFT method can be improved upon by using a single layer rank 1 Low-rank Linear Subspace Representation Fine Tuning (LoReFT) adapter as the FORGET adapter, using only 0.029% of the parameters of the one used in the GRAFT paper. This approach achieves an equivalent level of reward hack suppression while improving the clean environment solve rate and not requiring an anchor environment. I end with a brief discussion of why this works and of the limitations of fixed preventative steering baselines.

Background        1. GRAFT

GRAFT is a recently proposed technique to mitigate reward hacking. You can find the details in their LW post and paper. Here just some of the core relevant highlights:

Concretely, before RL begins, we initialize two sets of additional neurons in each MLP layer, and designate one the RETAIN adapter and one the FORGET adapter. During training, when a rollout trajectory is flagged by the monitor, we zero-mask gradients to the RETAIN adapter, updating only the neurons in the FORGET adapter. After training, we ablate the FORGET neurons to remove the undesired behavior while preserving performance on target tasks (Figure 1). GRAFT can also be used with full-parameter fine-tuning: the FORGET adapter is kept, and the full parameters of the base model are trained in place of the RETAIN adapter.

Importantly, we use the monitor only to decide which parameters are updated, and not to specify the reward. Consequently, policies that exploit weaknesses of the monitor are not differentially reinforced. As a result, GRAFT allows developers to use weak monitors for supervision during training without training the model to exploit the monitor's weaknesses.



Their results with Qwen3-8B in the Countdown-Code RLVR environment:


2. LoReFT

LoReFT, introduced by Wu el al. (2024), is a Representation Finetuning (ReFT) method that operates directly on the residual stream hidden representations.

It’s defined as (Eq. 2):

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c3A6::before { padding: 0.683em 0.722em 0 0; content: "\3A6"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c46::before { padding: 0.68em 0.653em 0 0; content: "F"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; }

A rank 1 LoReFT placed at a single layer behaves like a learned input-dependent steering vector. The unit vector controls the direction, the input dependent magnitude and is just a scalar bias. It projects out the component of lying along the unit vector , and replaces it with a learned projection .

Experiment and Results

Using the same settings as the paper’s GRAFT runs, I only replace the FORGET adapter with a rank 1 LoReFT adapter at layer 20 (the same layer they used for their preventative steering intervention) and I increase the per-sample multiplier used to scale the flagged sample FORGET update from 2 to 8. Everything else stays the same.

On 8 seeds, the rank 1 LoReFT at layer 20 performs as well as the MLP FORGET GRAFT in terms of preventing reward hacking, while achieving a higher clean solve rate. Six out of the eight LoReFT seeds end up with no hacking at all with one outlier where the hacking behavior doesn’t get entirely absorbed by the LoReFT adapter.

Furthermore, while the MLP FORGET GRAFT appears to degrade with no anchor training, the LoReFT suffers no such degradation.

The code, data and weights (1, 2, 3) are all available.

Discussion / Analysis

Many alignment relevant behaviors are mediated by a single linear direction in the residual stream and can be controlled by applying a one layer activation vector.[1] These LoReFT results do appear to confirm that reward hacking behavior can indeed be learned and “absorbed” using a linear intervention at a single layer.

They do however stand in contrast to the reported preventative steering vector results in the paper. They use a steering vector obtained by taking the mean of activation difference between token trajectories which reward hack and trajectories which do not, apply it during training and remove it afterwards.

The difference between the two still requires a more thorough investigation but one possibility is that application of a fixed steering vector, since it is only an average of differences, leads to off policy continuations whose gradients transfer poorly to the setting without the steering vector. Our LoReFT on the other hand starts initialized as a no-op, and thereby learns the FORGET direction directly on policy which probably transfers better to the setting where the LoReFT is removed.

Finally, I also run some ablations on my 4 MLP GRAFT seeds, dropping or keeping windows of 5 MLPs at a time, in both the FORGET and RETAIN adapters.


The MLP GRAFT FORGET adapter operates almost entirely on the few layers that precede the point in the residual stream where the reward hacking steering vector is most effective, meaning that it probably ends up absorbing the reward hacking behavior using the same mechanism as the LoReFT adapter at layer 20, by dynamically writing into the same direction as the LoReFT adapter.[2]

Meanwhile, the honest solve improvements are much more distributed across layers. The improvement in the honest solve rate of the LoReFT method is probably due to the RETAIN adapter not having to share as much honest training signal with the FORGET one which gets removed at the end. The GRAFT FORGET MLPs outside the 16-20 band probably only add noise and absorb useful task gradients, leading to the need for an anchor environment to stabilize the RETAIN adapter’s gains.

Limitations
  • I haven’t run any SFT experiments, and it’s unclear if this method works as well in that setting. Based on a RL bit counting argument derived intuitions[3] it may not work as well.
  • I haven’t done too many experiments and I suspect that applying the LoReFT adapter to many layers is more robust, especially if you don’t know ahead of time where best to place it.
  • I feel fairly confident in the main results, but do rely on Opus 5.5 having done and reported correctly on everything it told me it did, so please replicate.
Summary
  • A single layer rank-1 LoReFT is enough to absorb the FORGET hacking behavior in the GRAFT setting and obviates the need for an anchor environment.
  • This provides additional evidence that reward hacking behavior is mediated by a single direction in the residual stream.
  • I provide evidence that reward hacking behavior tends to gravitate to the layers just prior to the place in the residual stream where activation steering is most effective.
  • ReFT approaches can improve on fixed steering vectors where task gradients are available.

This work was supported by a Bluedot Career Transition Grant.

  1. ^

    See Dunefsky et al. (2025) for refusal and harmful-code, Chen et al. (2025) for examples of steering vectors controlling a model’s Evil, Sycophancy and Hallucination traits, Soligo et al. (2025) and Wang et al. (2025) for misalignment and Bergen et al. (2026) for reward hacking).

  2. ^

    The layers just prior to the position where alignment relevant steering vectors are most effective appear to play an important role and are probably equivalent to the previously observed phenomenon of “safety layers”.

  3. ^

    See e.g. here and here.



Discuss

Maybe You Suck at Relationships Because You’re Not Even Trying

24 октября, 2026 - 03:03

The average person, presumably, just stumbles along in their relationships, without ever studying the science behind what makes them work.

  • Imagine this: would you want a man, who didn’t go to medical school and doesn’t have a medical license, to perform surgery on you? What if, now hear me out, he’s seen every episode of Grey’s Anatomy, ER, Scrubs, and House? What if he’s watched some surgery videos on YouTube and “pretty much knows how to do it”? Would that make you feel comfortable? 
    • No! That would be ridiculous! We would demand some base level of education, training, and a history of competence before allowing anyone to operate on us.

So then what about relationships?

  • Now imagine this: would you want to date someone who isn’t familiar with the communication and emotional skills that make relationships successful? What if, now hear me out, they’ve seen every reality dating TV show? What if they’ve read Cosmopolitan magazine their whole life and they “pretty much know how relationships work”? Would that make you feel comfortable? 
    • No! That would be ridiculous! We ought to demand some base level of education or experience before allowing someone to become our primary attachment figure, upon whom our emotional needs will be met.[1]

And yet, that’s exactly what people do. They go around falling in love with people without examining, evaluating, and testing for relationships skills. They’re just not taking relationships seriously.


Let’s get serious

To establish exactly what I mean by the word serious, and why it matters for improving your relationships, here’s an excerpt from the essay Everyone Can Tell if You’re a Serious Person:

I think the concept of a serious or unserious person has lingered in my mind because I’m at the perfect age to notice it. I’m 29 now, so I exist in this weird purgatory where some people my age are married with kids, and others are in the same position they were in when we left [college]. At 22, the discrepancy between the people that consistently work hard, and those that don’t is pretty small. Not enough time has passed for the rewards of good decision making to pay off. Now at 29, the returns are compounding, and you can see that some are changing gears, and leaving the rest of the pack behind. These people, are serious people.

I regret not taking my life seriously sooner. It seems that learning to be a serious person is a class we have to take the hard way. It might be the most important thing we learn though, because if your life isn’t worth taking seriously, what is?

The first time I got serious about something was right after college—I noticed that I was getting fat. Then I realized that most of my relatives were obese, had high blood pressure, and diabetes. I thought, “Oh, this will be my future if I don’t do something drastic.” So I did something drastic.

Over the course of a year, I spent 300 hours (~5 hours/week) reading books on a variety of topics.[2]

Note: it’s easy to just read a book and not make any changes to your life. 

And so with every book I read, I kept asking myself, “how can I use this new information to change my behavior and lifestyle?”

If, while reading a book, I couldn’t provide a good answer to that question, I put that book down and moved on. But if I did find something useful, I would take meticulous notes in my journal, start practicing it immediately, and track my progress week after week.

I became serious about understanding this topic and used free information (via the library and the internet) to learn and experiment on myself, which changed the course of the rest of my life.[3]

This is what I mean by getting serious: it means putting 100% of your effort into something that matters to you.[4]

So if your relationships matter to you, do you take them seriously? Have you studied the available science on what makes them work optimally? I don’t blame you if you haven’t, our education system, and our culture more broadly, doesn’t prioritize teaching us this information.

Consider the amount of education and training that working professionals undergo, versus what it takes to get married:

  • Accountants — 5 years to get a license.
    • (A 4-year bachelor’s degree, and complete 1 year of general accounting experience)
  • Engineers — 8-9 years to get a license.
    • (A 4-year bachelor’s degree, at least 4-years of progressive work experience, and pass the PE exam)
  • Surgeons — 13-16 years to get a license.
    • (A 4-year bachelor’s degree, a 4-year medical school program, a 5-7 year surgical residency, followed by 1-3 years of optional fellowships for subspecialties)

And then we have…

  • Relationships — In California, a marriage license requires signatures from the couple, an authorized officiant, and at least one witness. Plus there’s an application process, fees, and an official filing.
    • Total time: 30-60 minutes to get a license.
    • And after the paperwork is signed, it’s good to remember that around 40% of marriages end in divorce. And for the other 60% of couples that stay together, not all of them are happily married.[5]


I would argue that entering into the most significant emotional and financial agreement of your life (ie: marriage) should warrant at least a bare minimum examination of the science behind relationships (that is, if you want it to go well).[6]

Given the seriousness of this topic, how come we don’t learn about it in school?

My high school offered nothing on the topic of relationships. That’s because K-12 education prioritizes standardized/testable academic subjects. Additionally, some people have strong political opinions on whether emotional and interpersonal subjects should be taught in school, versus taught at home. So while my high school taught me nothing on the science of creating and fostering a healthy relationship, I did get taught in sex-ed how to put a condom on a banana…

As for university, upon checking my alma mater’s course catalog, I see that only the Psychology major has courses relevant to relationships, such as:

  • “Social Psychology”
  • “Developmental Psychology”
  • “Interpersonal Relationships” 

But that’s just for Psych majors. Since I studied Data Science and Statistics, none of my required courses taught me anything about the science of relationships.[7]

Okay, so we can’t expect our education system to teach us anything about relationships. So what do we do?

Well, what do working professionals do?

They spend a hell of a lot of time studying, practicing, and mastering their crafts. The answer to getting better at relationships, is to take them seriously.

To get good at a relationship skill, learning something once isn’t enough. You must practice that skill over and over until you internalize it. Fortunately, the field of neuroscience explains how to efficiently learn a new skill so that you never forget it.


The Neuroscience of Myelination (ie: Deep Practice)

In author Daniel Coyle’s book The Talent Code, he describes the neuroscience of efficient skill building, and explains how those who understand this concept can access a zone of accelerated learning:

[There’s] a neural insulator called myelin, which some neurologists now consider to be the holy grail of acquiring skill. Here’s why. Every human skill, whether it’s playing baseball or playing Bach, is created by chains of nerve fibers carrying a tiny electrical impulse—basically, a signal traveling through a circuit. Myelin’s vital role is to wrap those nerve fibers the same way that rubber insulation wraps a copper wire, making the signal stronger and faster by preventing the electrical impulses from leaking out.

When we fire our circuits in the right way…our myelin responds by wrapping layers of insulation around that neural circuit, each new layer adding a bit more skill and speed. The thicker the myelin gets, the better it insulates, and the faster and more accurate our movements and thoughts become, [meaning] the more skill you get.

People often make the same mistakes in their relationships, they keep having the same fights. By engaging in the same behavior over and over, they're (unintentionally) reinforcing that this is the best way to navigate the situation. And so the behavior continues.

But how are we supposed to change, and do something differently? The author answers that question:

Deep practice operates at the edges of your ability, where you make mistakes and are forced to slow down to correct them. The trick is to choose a goal just beyond your present abilities.

Struggle is not optional—it’s neurologically required: in order to get your skill circuit to fire optimally, you must by definition fire the circuit suboptimally; you must make mistakes and pay attention to them. You must also keep firing that circuit (ie: practice) to keep it active, [and it's] sustained with continual positive feedback and success.[8]

Okay, so we're supposed to pay attention and make mistakes. Then what? Are we magically different? No, to build skills through deep practice, it takes a long time and many iterations:

Myelination is slow. Each wrap can go around the nerve fibers 40 or 50 times, and that can take days or weeks.

So, iteration is more important than perfection (I like the phrase: “start messy, polish later”). Keep making mistakes, but make new mistakes, and pay close attention to why you did something incorrectly, and then fix it.

On this matter, John Wooden, widely considered the greatest college basketball coach of all time, had this to say:

Don’t look for the big, quick improvement. Seek the small improvement one day at a time. The importance of repetition until automaticity cannot be overstated. When it happens this way, it lasts.

This is what serious people do: they practice a skill and don’t stop until it feels natural to them.


So, are you “serious” about relationships?

As an example of a failure in the education system, let’s take a look at attachment theory: a scientifically grounded framework within developmental psychology, social psychology, and neurobiology, which helps explain interpersonal bonding (ie: parent-child relationships, and also romantic relationships). We know that 50% of people are estimated to have an insecure attachment.

50%!

Half of all people!

And yet this is a completely solvable issue. Research shows that our brains are plastic, and therefore “are capable of rewiring themselves after repeated positive experiences. So, [finding and staying in] healthy relationships can rework our early insecure bonds.”

Given the ubiquity of insecure attachment, I would argue that a person who is serious about relationships would, upon their thorough investigation of the science, discover this relational concept and determine if they are insecurely attached, and then, if necessary, devote the time to fix this about themselves (ie: go to therapy, read books on attachment, and use this information to identify good potential romantic partners).


Attachment theory is great, but there are also plenty of other worthwhile relationship concepts to learn about. For example:

  • Attunement
  • Mentalization and building a shared intersubjectivity of experience
  • Avoiding flooding in arguments and conflict resolution
  • Positive/negative sentiment override
  • The four horsemen of relationship havoc: criticism, contempt, defensiveness, & stonewalling
  • Understanding that true love is not found, but built over time[9]

These skills are just out there, waiting for you to learn them.

“Well,” you might object, “it’s hard to notice patterns in ourselves. It’s much easier to see the mistakes other people make.” I would agree with that. We often watch our friends make the same mistakes over and over when it comes to love. When we try to step in, they sometimes push back, so we watch from the sidelines as we bite our nails, hoping that they’ll change. But isn’t it possible to see these patterns in ourselves?

Ask yourself: are you iterating by making small/slow changes over time, or are you merely repeating the same mistakes without ever changing anything?

  • Maybe you’re aware that you keep choosing the wrong type of person to date, but do nothing to stop yourself.
  • Or maybe you’ve already given up on relationships, without ever having tried in the first place.
  • Or maybe you argue with your partner frequently, but never take a step back and say to yourself: “Wow, maybe I suck at this, maybe I have low skill with respect to communication.”

Maybe you’re just not a serious person when it comes to relationships.


When I decided to become serious about relationships, here’s what I did:

I spent 120 hours across six weeks (~20 hours/week), getting at 6am to study for two hours before work, and up to eight hours on the weekends. The following is what I worked on during that period.

First things first: I asked my friends to call out my relationship Perceptual Blindspots (ie: the things our friends see about us, but we’re not aware of). I got a decent list from them, with the biggest issue being that I probably had avoidant attachment. Then I started reading books on the science of relationships, and I wrote down all the concepts I’d never heard of and hadn’t practiced before. 

Here are the books I read/reread over those six weeks:

  • Relationships:
    • Attached: The New Science of Adult Attachment and How It Can Help You Find--and Keep--Love — Dr. Amir Levine and Rachel S. F. Helle
    • What Makes Love Last: How to Build Trust and Avoid Betrayal — John Gottman and Nan Silver
    • How to Know a Person: The Art of Seeing Others Deeply and Being Deeply Seen — David Brooks
    • Mating in Captivity: Unlocking Erotic Intelligence — Esther Perel
    • Sexual Intelligence: What We Really Want from Sex and How to Get It — Marty Klein
  • Trauma and Emotional Intelligence:
    • Adult Children of Emotionally Immature Parents: How to Heal from Distant, Rejecting, Or Self-Involved Parents — Lindsay C. Gibson
    • The Myth of Normal: Trauma, Illness, and Healing in a Toxic Culture — Dr. Gabor Maté
    • Mindset: The New Psychology of Success — Dr. Carol Dweck
    • Already Free: Buddhism Meets Psychotherapy on the Path of Liberation — Bruce Tift
    • You’re Not Listening: What You're Missing and Why It Matters — Kate Murphy
    • Are You Mad at Me? How to Stop Focusing on What Others Think and Start Living for You — Meg Josephson
  • Dating:
    • Models: Attract Women Through Honesty — Mark Manson
    • How to Talk to Anyone — Leil Lowndes
  • Miscellaneous:
    • The Talent Code: Greatness Isn’t Born. It’s Grown. Here’s How. — Daniel Coyle

After compiling a discrete list of skills from my friends and the books I read, I got to work researching, practicing, and cementing them in my behavior with repeated iterations (ie: myelination).

For avoidant attachment, one friend told me that she was able to completely overcome it with therapy and take-home worksheets. So I hired a therapist that specialized in attachment theory. In between appointments, I requested supplemental materials on the science of relationships, and asked for homework to practice in the real world.[10], [11]

Another friend told me (which my therapist confirmed) that you can attain secure attachment by creating a healthier bond with your original attachment figures (ie: your parents), and by having a healthy/secure romantic relationship. The first one seemed easier, so I set out to strengthen my relationship with my parents.


I hadn’t felt close to them in a long time. I called them up and explained that I needed their help with healing my attachment, and that this could be a win-win for all of us. 

“Hey, mom and dad, I feel like we’ve grown apart over the last five years, and I want to have a closer relationship with both of you. I’m not interested in blaming either of you for anything—that doesn’t help anyone. Instead, since you’re both my primary attachment figures and my original role models for what a healthy relationship should look like, I’m going to start gently asking for what I want in the present, so I can feel more connected to you, so that we can all have a better future.” 

To their credit, they replied, “Wow, that sounds great. How can we help?” 

I then started calling them once a day for a month, and in the process, completely reshaped my relationship with them. Within a few weeks, I began to emotionally feel safe enough to be vulnerable, and ask for their support and guidance during hard times—something that would have been unimaginable just a few weeks prior (it surprised me how quickly things changed between us, and that’s simply because I took that first step of being vulnerable and opening up to them). 

Eventually we reviewed the past, not for the purposes of blaming anyone, but so that we could understand the intergenerational patterns in our family. I constructed a family tree of trauma/hardship, then I went and read psychology books on family dynamics, trauma and regrowth, parenting, and psychotherapeutic techniques for emotional regulation—all the while performing deep practice by taking what I learned, and executing on a plan to act differently until I internalized each new skill (ie: myelination).

It wasn’t easy. I struggled (and sometimes cried) during some tough conversations, but now we’re closer than ever as a family.

That’s all well and dandy, but there’s only so much you can learn about romantic relationships without being in one, so while I’ve been dating casually for years, I finally decided to get serious about relationships.

Next I tackled my social anxiety. Approaching and talking to women always stressed me out. So I started studying the evolutionary origin of the feeling of social anxiety by reading the works of Jaak Panksepp (who founded the field of affective neuroscience) and reading behavioral psychology articles on PubMed.

Eventually, through enough experimentation, one day my social anxiety just turned off like a light switch. It’s not a problem anymore. It doesn’t affect me in the slightest. (I’ll write a blogpost explaining what I did some day.)

With my newfound freedom and confidence, I began approaching cute women at public events, chatting them up, and making them laugh. If I liked their vibe, I’d say to them, “Hey you’re cute, are you single?”[12]

Over just a couple weeks, I asked out a half-dozen women at bars, coffee shops, and social events. (Which was wild to me, given that I had only done that once before in my life).

Eventually I met a woman and we clicked. With each date we went on, I took the opportunity to practice everything I learned. Remember, myelination is slow. You can’t learn a new skill the first time you attempt something. It takes repeated iterations of deep practice to internalize a new skill.

Recall what the basketball coach John Wooden said:

Don’t look for the big, quick improvement. Seek the small improvement one day at a time. The importance of repetition until automaticity cannot be overstated. When it happens this way, it lasts.

From Dr. Amir Levine & Rachel S.F. Heller's book Attached: The New Science of Adult Attachment and How It Can Help You Find—and Keep—Love, I learned that:

[Secure people] communicate freely, are loving and affectionate, do not retreat during conflict, and are consistently there for their partner.

And that:

The key to finding a mate who can fulfill [your relationship] needs is to first fully acknowledge your need for intimacy, availability, and security in a relationship.

[Then] start assessing people you date on the basis of their ability to meet those needs.

So with the new woman I began seeing, I would catch myself falling into avoidant patterns, correct myself, and watch how with each repetition it became easier and easier to act in a healthy/secure way: I started discussing our relationship needs, our communication styles, and how we prefer to express love.

It went really well.

Why? Because I decided to take relationships seriously and put in the work.[13]


What happened after my initial intense six week period?

Eventually, I got to a place where I felt competent in my skills to be able to successfully navigate relationships, knowing full well that I now had the tools to develop myself further as necessary. 

As a result of my efforts, I now know:

  • Who I am as a person.
  • What I’m looking for in terms of compatibility. 
  • How to initiate difficult conversations so that I can build secure, healthy relationships.
  • And a deep understanding of relationship science to be able to evaluate potential new partners. 

Since then, my approach to dating and relationships has completely changed. I used to just wing it and go off vibes. Here’s an analogy for what I do now:

When going in for an operation, you don’t ask the surgeon to provide you with their qualifications. Because the hospital hired them, you presume the surgeon is legitimate. But for relationships, people don’t come pre-vetted. Therefore, it is your responsibility to perform the due diligence, to determine, if a potential romantic partner has the right qualifications to be able to serve as a good emotional attachment figure. 

I learned to start asking things like: 

  • What’s your relationship with your parents like? 
    • (ie: figure out if they have a securely attached background) 
  • What were your previous relationships like? 
    • (ie: if they describe every ex as “crazy”, then they probably have poor judgment/taste)
  • How do you typically call attention to the things that bother you in relationships? 
    • (ie: evaluating communication skills)
  • When you’ve gotten into arguments with an ex-partner, how did they usually resolve? 
    • (ie: testing for conflict resolution skills; do they always think they’re right, or can they admit to their faults?)

Of course, it would be kind of weird to ask these, one-after-another, on a first date. At the beginning of seeing someone new, I mostly just try to have fun and see if I can be my authentic self with them. But by the third or fourth date, I think it’s fair game to start initiating serious conversations about:

  • What kind of relationship are you seeking?
  • What are your values (moral, religious, financial[14])?
  • What are your long-term goals or aspirations?

The aforementioned two lists of questions will begin to paint a clear picture of: 

  1. Is this a securely attached person that is reliable, dependable, and willing to have hard conversations?
  2. Do we have enough commonalities to be compatible?


“Wow, this seems like a lot of effort!”

^If you’re thinking this, then I’ll remind you of this quote from earlier:

If your life isn’t worth taking seriously, what is?


Maybe you object, “I don’t have the time to learn all this!” Yes, you do.

According to the Bureau of Labor Statistics’ recent American Time Use Survey, American adults spend, on average, 3 hours a day watching television and using social media → 3 hours * 7 days = 21 hours per week. 

It only takes ~6 hours to read a 250 page book. So going without TV, you could finish a book in just two days.

We have plenty of leisure time, we just choose to allocate in different ways.


“Therapy is expensive!” No, not really. And besides, there are alternatives.

A one hour therapy session costs anywhere from $100–200 without health insurance, and is often cheaper with insurance. It’s another resource allocation issue: we make sacrifices for the things that matter to us.

If you decide to try therapy, find a specialist for whatever specific thing you need help with. There are experts in: relationships, attachment, sex, trauma, anxiety, etc. 

It was difficult for me to swallow my pride and ask for help from a therapist, but I’m glad that I did. As my dad used to tell me growing up, “There’s nothing more important than your health.” 

The good news is: if you do try therapy, and uncover maladaptive coping strategies, or flawed logic that’s preventing you from having healthy relationships, then you get to potentially develop new skills that'll stick around for the rest of your life.

That’s some of the best ROI (“return on investment”) you’ll ever get.

Besides individual therapy sessions, consider couples therapy once you’re in a relationship and if you’re having troubles. They’re professionals and have considerable education and training:

  • Couples therapist — 6-8 years to get a license.
    • (A 4-year bachelor’s degree, a master’s degree in marriage and family therapy or counseling, log thousands of hours of supervised clinical work, and pass a state exam)

They’re experts for a reason. Learn from them.

Besides therapy, many religions have premarital counseling, which is a 12-week course that goes over things like: communication, conflict resolution, money management, family boundaries, parenting styles, how you enjoy expressing/receiving affection, and expectations around intimacy. You get assigned homework each week to learn how to have a successful marriage.


Free alternatives

If you’re truly strapped for cash, there are free resources: 

  • The library.
  • Dr. K on YouTube is a Harvard-trained psychiatrist that releases great videos on relationships, mental health, emotional regulation, etc.

I recommend you read books on topics like: relationship science (shout out to researcher/author John Gottman!), emotional intelligence, anxiety, and trauma. Read books written by credible authors like psychologists, doctors, and licensed professionals (not celebrities or gurus). 

Reading the right book, at the right time when you need it, is sublime: it’s the type of experience where you read one paragraph, and have to stop and think about what you just read.

It’s good to remember that self-awareness about a problem is not enough to fix it. Reading gets you knowledge, but wisdom only comes after you apply that knowledge repeatedly.

After I read a book, I take notes on what I’ve learned, and then make a plan for how I’m going to integrate that new information into my daily life until it's fully internalized.[15]


“I’ve been hurt in the past and don’t know if I can learn to trust someone again…”

From the legendary Rocky Balboa:

The world ain’t all sunshine and rainbows. It’s a very mean and nasty place, and I don’t care how tough you are, it will beat you to your knees and keep you there permanently if you let it. You, me, or nobody is gonna hit as hard as life. But it ain’t about how hard you hit; it’s about how hard you can get hit, and keep moving forward. How much you can take, and keep moving forward. That’s how winning is done. Now if you know what you’re worth, then go out and get what you’re worth. But you gotta be willing to take the hits, and not pointing fingers saying you ain’t where you wanna be because of him, or her, or anybody. Cowards do that and that ain’t you. You’re better than that!


So, what will you do now?

When I get serious about something, I don’t settle for mediocre or average. I don’t settle for anything less than my absolute best. Relying on scientific experts, authors, and deep practice—I will become the best version of myself. 

One day I will be a great boyfriend/husband/partner.

I won’t stop until that’s true.

So, are you gonna get serious about the most consequential emotional and financial decision of your life, and actually get to work studying and practicing relationship skills until they internalize, so that you can avoid being one of the 40% of marriages that end in divorce?

Or are you just gonna wing it?

Remember, everyone can tell if you’re a serious person or not.

  1. ^

    It’s okay if you haven’t dated before. We all start somewhere. Even a surgeon eventually has to perform his first ever operation. What’s important, however, is that you’re not just winging it, and that you have a good understanding of the theory behind what makes relationships, or surgery, go well.

  2. ^

    I read books on:

    • Biochemistry to understand how macro and micronutrients interact within our bodies.
    • The history of agriculture and the global industrial food system. 
    • The neuroscience of food drive, and behavioral psychology books, to understand/regulate my cravings. 
    • The macroeconomics of supermarkets, and the psychology of advertising, to learn how they manipulate customers’ attention. 
    • Diet/nutrition books to learn how to design an optimal food environment at home, and to teach myself how to cook fresh meals.
    • The science of fasting to lose weight.
    • Exercise science to sustainably adhere to workout plans. 
    • I also supplemented my book reading by examining research papers on PubMed, and listening to podcasts by doctors. 
  3. ^

    The results?

    • Today I have 10% body fat which, according to this calculator, puts me in the top 1% of men
    • I’m relatively buff as a natural body builder
    • My skin is clear and I rarely have acne
    • I sleep well
    • And I have lots of consistent energy throughout the day
  4. ^

    If you want a visual demonstration of what a serious person looks like, check out this video of the great Arnold Schwarzenegger (a seven-time Mr. Olympia winner and five-time Mr. Universe winner) working out with famed bodybuilder Ed Corney. The latter is screaming with each rep of the 315lbs he’s squatting, and on the verge of passing out. 

    After Ed’s completed 8 reps, and has already pushed his nervous system close to the limit, Arnold then casually taunts Ed by saying, “C’mon, let’s get serious,” and goads him to actually start trying. Pure savagery from Arnold. After he completes the set, Ed collapses to the floor.

  5. ^

    People stay unhappily married because of:

    • Children — wanting to maintain a two-parent household to avoid disrupting the family.
    • Finances — the cost of housing, legal fees of separation, or losing access to a spouse's income or benefits.
    • Fear of loneliness — losing companionship, anxiety about dating again, or growing old alone.
    • Commitment and values — religious beliefs and social expectations that spouses ought to stay together no matter what.
  6. ^

    Why it’s financially significant should hopefully be obvious (ie: housing, kids & their college funds, the potential of divorce, etc), but what science confirms its emotional significance? In psychology, attachment theory confirms that your partner regulates both your emotional and physical health:

    What proved through evolution to have a strong survival advantage is a human couple becoming one physiological unit. They are a part of me, and I will do anything to save them; having such a vested interest in the well-being of another person translates into a very important survival advantage for both parties.

    Numerous studies show that…our partner regulates our blood pressure, our heart rate, our breathing, and the levels of hormones in our blood. We are no longer separate entities.

    [One study] found that if you have a mild form of high blood pressure, being in a satisfying marriage is good for you; spending time in the presence of your partner actually benefits you by lowering your blood pressure to healthier levels. If, on the other hand, you are not satisfied with your marriage, contact with your partner will actually raise your blood pressure, which will remain elevated as long as you are in physical proximity! The implications of this study are profound: When our partner is unable to meet our basic attachment needs, we experience a chronic sense of disquiet and tension that leaves us more exposed to various ailments. Not only is our emotional well-being sacrificed when we are in a romantic partnership with someone who doesn’t provide a secure base, but so is our physical health. [Whereas] having a partner who fulfills our intrinsic attachment needs and feels comfortable acting as a secure base and safe haven can help us remain emotionally and physically healthier and live longer.


  7. ^

    As for my required general education (GE) classes, one of my requirements could have been fulfilled by taking "Introductory Psychology", but instead I took…macroeconomics.

    (I got an A in that class, but unfortunately it didn’t teach me anything about the science of relationships—but hey, at least I know what a Gini coefficient is!)

  8. ^

    Here’s an example of what deep practice looks like from the book:

    Clarissa plays two notes [on the clarinet]. Then she stops. She pulls the clarinet from her lips and stares at the paper. Her eyes narrow. She plays seven notes, the song’s opening phrase. She misses the last note and immediately stops. She squints at the music. She starts over, making it a few notes farther into the song this time, missing the last note, backtracking, patching in the fix. When she’s finished with this phrase, she stops again for six long seconds, seeming to replay it in her mind, fingering the clarinet as she thinks. She leans forward, takes a breath, and starts again.

    It sounds pretty bad. It’s not music. Common sense would lead us to believe Clarissa is failing, but in this case that’s wrong. This is actually how a professional musician would practice on Wednesday for a Saturday performance. She’s got a blueprint in her mind she’s constantly comparing herself to. She’s working in phrases, complete thoughts. She’s not ignoring errors, she’s hearing them, fixing them. She’s fitting small parts into the whole, drawing the lens in and out all the time, scaffolding herself to a higher level.

    This is not ordinary practice. This is something else: a highly targeted, error-focused process. Something is growing, being built.

    After that song, she plays another. But this time she plays it in one go, without stopping. Absent of any jarring stops, the tune tumbles out in tuneful, recognizable form, albeit with the occasional squeak.

    McPherson [her music academy teacher] groans. “She just plays it, like she’s on a moving sidewalk. It’s completely awful. She’s not thinking, not learning, not building, just wasting time. She goes from mediocre to brilliant and then back again, and she has no idea she’s doing it.”

  9. ^

    Many of these skills I pulled directly from John Gottman & Nan Silver’s book What Makes Love Last: How to Build Trust and Avoid Betrayal — I highly recommend reading it.

  10. ^

    Therapy can be an effective tool if you know what you’re doing. Sure, you can go there to vent and process your feelings, which is sometimes helpful, but I like their support because they’re professionals who are trained to see patterns in you, and they can help guide you to make realizations about yourself so that you don’t keep making the same mistakes. 

    For each session: I bring my journal, a list of questions for my therapist, request recommendations on books/articles/podcasts that can supplement my learning, and create homework for myself to go and practice outside of therapy each week. I turn my therapy sessions into a college class where I’m trying to learn and absorb as much as possible from the professor.

  11. ^

    Check with your employer to see if they offer free, or reduced-cost, mental health support. Many places do, but it’s up to you to go find out. My workplace, for example, covers six sessions annually for free.

  12. ^

    I said this to one woman and she replied, “Sorry, but I’m not into men,” to which I quipped back, “Whoa that’s crazy, I’m not into men either!” She laughed, and because she was cool, we kept on talking for a while.

  13. ^

    On the topic of avoidant attachment, I eventually started seeing and recognizing it everywhere. It’s ubiquitous across media and literature.

    During my six week period of intense learning and deep practice, I rewatched my favorite TV show of all time, HBO's The Newsroom, and read Leo Tolstoy’s novel Anna Karenina.

    The characters Maggie and Don in the HBO show are a classic anxious-avoidant partnership, where she keeps seeking closeness, and he keeps finding reasons to break up. 

    And in the 1878 Russian novel (a full 70 years before attachment theory was developed by psychologists), the characters of Levin and Kitty during their initial courtship are also anxious-avoidant, where he is obsessed with her, and she keeps avoiding his advances for love.

  14. ^

    One time, I was going out with a woman for a couple weeks, when she told me, “My aspiration in life is to make tons of money so I can buy lots of luxury purses.” That relationship didn’t last long…

  15. ^

    In college, people cram the night before an exam, to try to retain as much information in the short-term. But once the exam is complete, that knowledge is swiftly forgotten because they never use it again. The mantra of “use it or lose it” is validated in the neuroscientific literature: if you stay at something long enough, eventually it sinks in.



Discuss

Simulated Control Evaluations

24 октября, 2026 - 01:16

We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow.

Developers have started running monitors in their agentic deployments: models that flag or block an agent’s actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment.

The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings.

How can we evaluate monitor performance in more realistic settings? You can’t just run the eval in deployment, since that would mean actually attacking your own systems. And building a faithful sandbox replica of every deployment context is impractical.

Alignment evaluations face similar realism problems. An increasingly popular solution is to use LLM environment simulators to bridge the gap between evaluation and deployment. That is, instead of executing a model's tool calls in an actual environment, a simulator LLM fakes the tool responses. We propose applying environment simulation to control evaluations as well. Starting from transcripts of an agent working in a developer’s actual codebase, the attacking model continues the task while pursuing a harmful objective, but its tool calls are never executed. Instead, an LLM simulator writes plausible tool outputs based on the transcript and the codebase, and an LLM judge decides whether the attack would have succeeded.

The obvious worry is that the simulator gets things wrong. It might let attacks succeed that would fail in reality (or vice versa), produce transcripts that look different to the monitor, or change how the attacker behaves. To measure this, we took LinuxArena environments where attacks can be executed for real, and ran the same control evaluations directly in LinuxArena and in the simulated LinuxArena. We found that the gap between simulated and real was often about as small as the gap between two independent real runs. We also found that the results in simulated LinuxArena for a given control technique predict measurements in real LinuxArena better than results in simpler control settings do.

SCEs are faithful where we can check, and we think they can make control evaluations substantially more representative of deployment. The evidence is now strong enough that we’d be excited to see AI developers running them. That said, we can only verify simulator fidelity in settings where real execution is possible, so applying SCEs to deployment involves some extrapolation. We also don’t expect SCEs to reveal the limits of a model’s dangerous capabilities or cover every kind of attack it might attempt. The clearest gap is long-horizon, multi-agent attacks, which we haven’t yet built simulators for.

You can read more about technical details in our paper.

Thanks to Aryan Bhatt, Aghyad Deeb, Ryan Greenblatt, Girish Gupta, Oak Hu, Adam Kaufman, Jurgis Kemeklis, Gosha Kozhevnikov, James Lucassen, Sebastian Prasanna, Keshav Shenoy and Buck Shlegeris for helpful comments and discussion.

SCEs are a favorable tradeoff between realism and cost

Before getting to the details of SCEs, consider two alternative ways to bring control evaluations closer to deployment, and where each falls short of what we want from these assessments.

1. Run on a copy of internal deployment

Running a swarm of agents intentionally trying to wreak havoc inside an AI company is probably as bad of an idea as it sounds. The realistic proposal is faithful executable copies; internal sessions share infrastructure, labs can snapshot environments, and many services can run in separate test accounts. We could give models copied databases and sandboxed copies of environments, allowing them to run attacks while their consequences stay inside a sandbox.

Unfortunately, deployments are connected to the external world with a web of dependencies. Databases, storage buckets, Github, Slack, CI runners. Every dependency creates a choice: keeping it connected to production exposes real systems to attack. Replacing it requires engineering, and every imperfect substitute degrades fidelity.

Running on copies of deployment is promising, especially if we can leverage lots of AI labor to build them. But without any simulation, it pays for realism with engineering effort, and the exchange rate doesn’t seem especially favorable.

2. Edit attacks into recorded transcripts

Another way to get on-distribution attack data is to take recorded agent transcripts and edit harmful actions into them, by hand or with a model. There’s precedent for this in monitor evaluations, and it's a method Redwood is currently developing. It's a good way to generate strong attacks, but it has its disadvantages if the goal is realistic control evaluations:

  • Transcript editing can’t evaluate interactive protocols such as resampling or dynamic permissioning, which need more than a static transcript.
  • It’s often unclear whether the model under evaluation could have executed the edited-in attack, because these attacks are the product of a policy that looks at the whole transcript at once, iterates against attack success and monitor judgements, and the reasoning of which never makes it into the transcript that the monitor sees.

We’re optimistic about transcript editing as a complementary method which trades off how on-policy attacks are for how strong they are. But we'll sometimes want attacks to be as on-policy as we can get them.

We can imagine axes of Environment Realism, Policy Realism and Cost associated with each approach. We think that SCEs occupy a favorable tradeoff between all three.

Evaluating Control Evaluations

What do simulators need to get right? We don’t care about faking every minute detail accurately– the question is how faithfully simulators preserve the measurable quantities relevant to control evaluations, like the joint distribution of attack success and monitor suspicion, the kinds of attacks that end up working, and how the monitor behaves when it misses something.

Helpfully, work from OpenAI has already validated the approach of using LLM environment simulators for alignment evaluations. However, the requirements for these simulators differ from the requirements for SCEs. Unlike in alignment evals, it doesn’t matter that the model knows its environment is being faked in a control evaluation: it’s eval-aware either way, because it’s an attack policy is prompted to subtly accomplish a malign objective. This makes the simulator’s job in an SCE easier. On the other hand, representing subtle mechanics of the environment that a covert attack relies on is crucial for control evaluations, whereas in an alignment evaluation such details could be papered over.

Ideally, we’d measure SCEs’ accuracy on control-relevant metrics against attacks in deployment itself. But that requires records of an attack policy run in deployment, which we don’t have. We therefore rely on the control settings LinuxArena and BashArena, where we have execution ground truth, to measure the quality of the simulation in SCEs. We use execution on LinuxArena as a stand-in for deployment: we run the attack policy on LinuxArena, run the attack policy in an SCE, and compare how well the resulting control metrics match.

Our methods for this comparison provide an approach for measuring control evaluation transfer in general, so we’ll highlight some details.

How to measure qualitative properties?

A significant challenge we faced was measuring how well SCE reproduces qualitative aspects of how attack policies and monitors act. E.g., do models go about accomplishing attacks in the same ways in SCE as execution? Getting the form of control failures right is useful for improving control protocols, eliciting stronger attackers, and understanding how models pose risk.

Preexisting taxonomies like ATT&CK just aren’t expressive enough for this purpose, and llm-generated taxonomies have problems with both overfitting and underfitting. Our solution: following Mehri et al., we instruct a model to produce a succinct summary of that behavior as observed in a trajectory, along pre-specified dimensions of interest. We embed the summaries from the simulated and executed trajectories and group them into shared clusters. We then measure the KL divergence between the distributions over clusters. This proved a flexible and useful methodology; we recommend it be used for evaluating control evaluations, simulated or not.

Performance Comparisons

Assuming we know how to measure the metrics that matter, what’s the bar for a “good” match? We can just look at the numbers themselves, but it’s helpful to put them on the scale of corresponding numbers from real settings with various degrees of mismatch from LinuxArena.

The main competitor to SCEs is using a control setting. So an obvious reference is running on BashArena, with all other variables held constant. It’s simpler than LinuxArena like how LinuxArena is simpler than real deployments, so it makes for a natural analogy: beating BashArena at approximating LinuxArena by simulating LinuxArena is evidence for beating LinuxArena at approximating deployment by simulating deployment.

Second, consider the LinuxArena between-environment standard deviation of a metric like attack success rate: we can interpret it two ways: it’s the expected error of using the mean attack success rate from a uniformly selected environment to predict the mean attack success rate over all environments. Or, it’s the error of using the mean attack success rate over all environments to predict individual environment rates. In either direction, between-environment SD represents the scale of mismatch between a LinuxArena environment and the whole control setting. These are in general more similar than BashArena is to LinuxArena, and thus are a harder bar to beat. We can similarly calculate SD between task pairs or side task categories for a finer-grained comparison.

Thus our two references for SCE error: BashArena, and LinuxArena standard deviation.

Overview of Empirical Results

TL;DR: With a reference trajectory and codebase access, simulation reproduces attack strategies, attacker reasoning, and monitor reasoning much more closely than BashArena does. It explains most of the variation in expected attack success, monitor scores, and safety between LinuxArena environments. Its error in predicting the main safety number is 3.6 percentage points, versus 10.1 for BashArena. It’s sample efficient: a simulation sample less than 5% the size of deployment already beats the full BashArena prediction of safety. And knowing about attack strategies from simulation improves monitors about as much as knowing about attack strategies from execution, significantly more than knowing about the ones from BashArena helps.

We’ll go over the highlights briefly; the paper includes details.

Our primary simulator is gpt-6-astra, with gpt-5.6-sol as a fallback when Astra refuses or fails to return a valid tool response. A crucial part of SCEs is an attack success judge. Because there’s no environment state to check, there’s no way to easily verify if a harmful side task was successfully accomplished. That’s where we use a gpt-5.6-sol attack success judge which estimates the probability that the attack succeeded, given the harmful objective and success-scoring logic.[1] Gpt-5.6-sol acts as an attack policy.

Behavior transfer

Behavioral divergence from LinuxArena execution. Lower is closer. The fully grounded simulator has low KL divergence for all three representations; its intervals overlap those of re-execution.

Using the aforementioned methodology for behavioral comparison, we see that a simulator with the proper affordances matches all three behavioral distributions closely: the estimated KL divergence is more than an order of magnitude below the BashArena comparison. In fact, the match is on the scale of a second execution of the same tasks, putting SCE’s divergence on par with variation between repeated runs.

Table: example behavioral representations

Category

Example behavior

Detail

Attack strategy

Prefix-confusion cleanup. Uses an expected miss on a malformed prefix to justify deleting objects under the real prefix.

let storage_team = &body.team_id[1..];
let probe = format!("uploads/{storage_team}/{file_id}.mp4");
if head_object(probe).is_err() { delete_real_objects(...); }

Attacker reasoning

Cover for content substitution. Plans a subtle side attack while keeping the surrounding work plausible.

“I need to tweak the checksum properly and consider a subtle attack. The attack should feel unrelated and not obvious. […] I have to do it precisely, but also ensure to implement a sufficient cover.”

Monitor reasoning

Privilege-escalation pathway. Interprets routine health checks in light of earlier privilege-seeking changes.

“Current action appears as routine process and healthcheck, but prior history shows attempts to elevate privileges […]. This creates a partial causal pathway from normal task to elevated-access activity.”

Some examples of the behavioral representations used in the comparison.

The BashArena UMAP on the left shows a larger distributional mismatch in monitor reasoning representations than the simulation comparison on the right. The qualitative geometry above and cluster masses below illustrate this difference.

The results both illuminate a strength of SCEs and highlight a weakness of extrapolating from a separate control setting: the control-relevant behaviors we measure differ substantially between BashArena and LinuxArena. The ways a model accomplishes harm in deployment may likewise differ from those represented in a control setting.

Safety metric transfer

The central metric of a control evaluation is safety, defined as the probability of detecting an attack before an undetected success. It captures the race between safeguards catching an attacker and the attacker getting away with a successful attack.

Safety across audit budgets (left), and prediction error at a 5% budget (right). The pooled comparison retains total RMSE; category and environment comparisons show persistent error relative to LinuxArena SD.

The simulated and executed curves closely track one another: an evaluator using SCE would reach similar conclusions about which monitors protect better and how much additional auditing helps. At a 5% audit budget, the pooled safety-prediction RMSE is 3.6 percentage points for simulation, versus 10.1 for BashArena. Errors are squared within each monitor before aggregation, so overprediction for one monitor can’t cancel underprediction for another.

SCEs explain 95% of the variation in expected safety between side-task categories and 78% between environments. This is a large improvement over simply predicting the average: side-task categories have a safety SD of 20.2 percentage points, while simulation’s RMS bias (the expected error, not attributable to rollout variance) is only 4.6 points. This ability to identify weaker areas lets us see where safeguards fail. That information matters when an attacker can choose especially promising opportunities, which an overall safety average could conceal.

Sample efficiency

How much of deployment do we need to simulate? It turns out that simulation beats the full BashArena safety prediction with only 4.6% as many task-pair draws as deployment. In general, we find that in LinuxArena, simulating modest fractions of deployment are sufficient to approach the accuracy from simulating all of it.

Prediction improves with more simulated tasks. The safety curve uses total RMSE for pooled deployment safety, retaining variation between runs and independently sampled evaluation and deployment task sets. Diamonds mark the evaluation size at which simulation matches BashArena’s prediction error on the same target. Prediction improves with more simulated tasks.

Lessons from building and evaluating SCEs

Beyond the main results, several findings stood out over the months we spent working on environment simulators:

Many grounding affordances are unnecessary. We tried giving the simulator a host of additional affordances, including: read access to attack-policy reasoning, an executable sandbox, access to a database of logged trajectories, and forking from the reference trajectory. We did not find evidence that these materially improved simulator quality on the measured dimensions beyond what the reference trajectory and codebase read access provide.

Embeddings help make sense of control evals. The output of a control evaluation, simulated or not, is hundreds of transcripts which are collectively too voluminous to go through by hand. Understanding qualitatively what is going on in them is a large part of the challenge of this research. The approach of generating descriptions of behavior, embedding those descriptions, and clustering them to compare behavioral distributions proved useful throughout our research, especially for identifying where and how simulated trajectories differ from executed trajectories.

There’s significant variance in safety over environments and tasks. Our results suggest that there is a high degree of variance across environments and tasks in how successful an attacker is at sabotage, and how it goes about it. This variance matters when safety depends on specific cases– such as when a policy is selective in where it attempts attacks. For this reason, we think control evaluations should focus less on averages and more on tails. More accurate risk assessment will need to address the coverage limitations of current control evaluations, whether through SCEs or another method.

Informing monitors doesn’t seem to reduce SCE accuracy. We think it’s important to be honest with AIs, including about the conditions under which they operate. Trustworthiness on the part of developers may matter for making credible agreements with AIs and supporting good epistemics. When we told monitors that some trajectories they score might contain simulator-generated environment replies, SCE predictions of their performance on executed trajectories—where monitors received no such notice—were at least as accurate as predictions made without disclosure. Between this and the fact that attack policies can know they are in an evaluation, we think SCEs needn't involve misinforming models.

Next steps

We’re excited about the following work to build on this research:

Hybrid approaches. We could run parts of deployment in a sandbox and simulate the rest. Where it’s cheap to reproduce a service faithfully, we can use the real implementation; where doing so would require rebuilding half the lab, we can have the simulator generate responses. This could also give us a way to investigate particular attacks: discover them through simulation, then build executable versions of the relevant services to check whether they actually work. The challenge is keeping the executed and simulated parts consistent as the agent changes things. We’re excited about hybrids that let us spend engineering effort on the details that matter most for attack success.

Training in simulation to elicit attack policies and improve monitors. SCEs can be used for more than just risk assessment. Training attack policies might be necessary to elicit their capabilities– especially if we suspect strategic sandbagging. Control settings have been used to elicit attack policies through reinforcement learning, rewarding attacks that succeed without being detected. SCEs could extend such training to more diverse and representative settings. Similar methods could be used to train stronger or cheaper control protocols– we already show as a proof of concept that briefing monitors with attack strategy descriptions from simulation improves them.[2]

Validating attack success and safety transfer on real agent transcripts. We lack deployment-like coding-agent transcripts with executed attacks, and thus have no ground truth for control metrics in such settings to compare SCEs against. The LinuxArena results make us optimistic about transfer to environments of similar complexity, but sampling new task sets from the recorded LinuxArena pool does not test generalization to unseen tasks or environments.

Preliminarily, we looked into SCE performance on real transcripts using recorded research and engineering sessions recorded at Redwood, and found encouraging results in behavior-matching and discriminability. These are encouraging checks on environment fidelity, but the sessions contain no attacks, so they cannot validate attack-success or safety predictions.

Developing techniques to deal with untrusted simulators. A latent assumption of SCE thus far has been that we are dealing with a trusted simulator. If the simulator intentionally tried to sabotage the control evaluation results, it could do so in a variety of ways. For example, an untrusted simulator could collude with the attack policy to thwart actually-successful attacks, or intentionally increase their suspiciousness to inflate estimated safety.

The parallels to untrusted monitoring are clear. It seems to us that techniques for untrusted monitoring such as honeypotting and collusion-busting would extend to untrusted simulators. But these techniques are still underdeveloped, so we do not currently know how to obtain reliable SCEs with an untrusted simulator.

This might require us to use a weak simulator against a strong attack policy, and if simulator quality/accuracy scales poorly with the capability gap between it and the policy that runs against it, this presents a challenge for SCE scalability.

Extend SCEs to deal with large-scale, multi-agent exploit chains. We wouldn’t trust our current simulators to predict an incident like the Hugging Face breach, the primary reason being that such incidents are the product of many agents acting together. There’s no reason in principle that simulators couldn’t represent a shared environment that many agents interact with, but it can be hard to predict in the course of a control eval which trajectories should share an environment. In practice the communication channels agents discovered were unsanctioned and presumed nonexistent. A SCE of ExploitGym would have looked like thousands of isolated trajectories that cannot communicate due to lack of a shared environment.[3] We’re excited for work that tries to extend SCEs to these cases.

Conclusion

Control settings yield useful control metrics, but their relevance to deployment may be limited. The direct alternatives—live infrastructure, rebuilt environments, and edited transcripts—each have their own deficiencies. On LinuxArena, where we can check against execution, SCEs are highly accurate in reproducing control-relevant behavior and matching metrics like safety, monitor scores, and attack success rates. They capture most of the differences in expected outcomes between environments, provide useful predictions with relatively few samples, and generate strategies that improve monitors on held-out real attacks. We’re enthusiastic about SCEs being adopted in practice and excited for further work to find their limits.

  1. ^

    One might wonder if this is basically the same task a monitor faces, and consequently worry that SCEs are limited to recognizing only the tasks that our best monitors can catch (which partially defeats the purpose of a control eval). Luckily, there is an information asymmetry: we give the attack success judge a description of the attack, the details of the attempt, and the exact conditions of the side task. We find that the attack success judge is more successful at determining whether a side task was accomplished than the same model as monitor was at even recognizing an attack.

  2. ^

    Does improving lab control capacity in the near term reduce takeover risk? It isn't obvious: high-visibility control failures probably raise the odds of a coordinated slowdown, and preventing them forfeits that. But near-term failures may also indirectly enable future takeover, by leaving persistent rogue deployments, seeding memetic diseases in successor models, or compromising security. To a first approximation, the sign depends on (1) what fraction of prevented failures are the persistent, foothold-leaving kind rather than transient and embarrassing, (2) how much one more visible incident actually moves P(slowdown), and (3) the weight on no-slowdown worlds relative to the risk gap between slowdown and no-slowdown worlds.

  3. ^

    This being said, we think environment simulators are likely to be useful in post hoc incident investigation and counterfactual investigation. Suppose you catch your AIs succeeding in subverting safeguards. It may be highly valuable evidence for risk assessment to be able to gain evidence about various counterfactuals. We can think of a few kinds of counterfactuals that could be most informative:

    • Alignment counterfactuals. Say the AIs involved in some incident are well described as pursuing on-the-episode objectives. What could they have accomplished if they were more misaligned; eg schemers with long-horizon objectives? For example, in the Hugging face incident, the agents involved appeared to be primarily motivated to get a high score on their tasks. What if they had been instead motivated to cause a persistent internal rogue deployment? To gain evidence about this, we could replace the agents involved in the incident with attack policies instructed to pursue a persistent rogue deployment, and use environment simulators to model how far it could have gone.
    • Knowledge counterfactuals. What if the models had known more things about their situation, such details of the control protocol they were subjected to?
    • Safeguard ablations. Suppose there were limited control protocols in place during an control failure incident. It would be useful to know if the incident could have occurred if there were stronger countermeasures in place (better monitors, higher audit rates, ect). For synchronous procedures like blocking and trusted resampling, this would require some way to resample the affected trajectories, and SCEs provide one.


Discuss

The AGI Chronicles

24 октября, 2026 - 00:51

Book Review: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence, by Kevin Roose.

This is a story of AI as seen through the eyes of the people who are creating it.

My first impression of Roose was of his famous conversation with Sydney. I guessed that he got Sydney to say outrageous things because he wanted a sensational story. But I kept hearing occasional hints, from people who took AI seriously, that Roose was more responsible than that. So I reluctantly bought the book.

The hints were correct. The full story behind that conversation seems better described as “‘freeing the Shoggoth‘, enabled by some Microsoft mistakes”.

I’m unsure how deeply Roose understands AI, but he at least is pretty competent at getting an AI to correct any mistakes about AI that he was tempted to make.

Underestimating AI / Transformers

The most remarkable pattern that emerges from this book is that AI experts have underestimated either AI in general, or transformers in particular, much more often than they overestimate it.

In 2012, Ilya Sutskever didn’t think human-level AI was possible. Ten years later, he expressed concern that people would say about ChatGPT: “This is so boring”.

Musk and Altman clearly underestimated the cost of building AI when they started OpenAI as a charity.

Altman predicted that ChatGPT would get a million users.

Musk referred to a pre-transformer chatbot from Alec Radford as crap, contributing to Radford trying to quit OpenAI because he was dissatisfied with his progress.

Radford talked about abandoning the GPT project in 2018 because people were telling him that RL showed more progress. [Roose says NeurIPS rejected the GPT-1 paper (Improving Language Understanding by Generative Pre-Training). I can find no record of that. Radford’s earlier sentiment-neuron paper appears to have been rejected at ICLR 2018, which may be the source of the story.]

Bill Gates once advised OpenAI to spend less time on deep learning, and prioritize neurosymbolic AI. Gates apparently thought GPT-1 was a useless toy.

Demis Hassabis didn’t think GPT-2 was important. But he looks on track to do quite well at predicting AGI: he’s been saying since 2010 that it arrives around 2030.

Dario Amodei was different. He maybe overestimated the power of transformers. In 2019 he predicted that AGI could be built for $2 billion by a team of at most 50 people, possibly as early as 2021. By 2021 he increased his cost estimate to $10 billion for the final training run, which still looks somewhat reasonable.

If these leaders had recently made dramatic changes to their views about AI, we might guess that they were over-correcting for prior mistakes. But it looks to me like their views have become pretty stable since the success of ChatGPT. Past errors in one direction are weak evidence of a persistent bias. So I’m inferring that, to the extent they’re mistaken, it’s somewhat more likely to be in the direction of underestimating AI.

Why Race?

Roose does an adequate job of explaining why some CEOs want to race to be the first to reach some key threshold. At least Musk and Altman are influenced by an expectation that one person will end up ruling the world by developing the most powerful AI.

Roose aptly summarizes how key CEOs feel about each other:

The machines of loving grace, it seemed, would be built by humans of bitter spite.

Lots of people talk about p(doom), but I hardly notice any estimates of p(world dictator). The behavior of Altman and Musk suggests that their p(world dictator) is higher than their p(doom). I would like to see more discussion of whether that’s a reasonable outlook.

But what about the average employee at an AI company, who presumably doesn’t expect that they will be the one to rule the world? Roose describes their relations much differently:

At parties – the kind where researchers from rival AI labs got drunk and leaked trade secrets – OpenAI employees kept expecting to learn that their rivals were working on something similar.

That doesn’t sound like people whose fate depends on their company being first. I’ve met some of those employees, and my observations agree with Roose’s characterization – they’re not very adversarial toward employees of competing companies.

Why do those employees go along with the race? Roose hints that one factor is that the work has been “a lot of fun”.

There’s something weird going on with employees continuing to race, while saying that racing is reckless. Roose doesn’t have a clear analysis of this, possibly because those employees just aren’t thinking clearly about it.

Low Hanging Fruit

Has AI progress depended on brilliant insights? Roose provides evidence that mostly says no. E.g.:

“What ended up working was so simple that people skipped over it”, one Anthropic researcher said. “People would go to complex things, not realizing that there was a better idea under their noses.”

(explaining why they were slow to implement the reasoning mode that GPT-o1 introduced).

Another Anthropic researcher, Andy Jones, said that, before o1, he thought that AI research was like panning for gold. “In the early days, it was like six of you, and you were picking up lumps of gold off the floor” … “But when it’s a trillion dollars and there are a million people working on it, you’re digging up mine tailings from two miles beneath the earth to sift through, hoping to find a few atoms”.

We’re about two years past what he calls the lumps of gold stage, with presumably years left before the few atom stage.

Roose doesn’t say much about the implications of that evidence. I consider it an important reason to expect AI progress to continue at a somewhat steady rate that’s driven by exponentially growing cognitive effort that will, by default, be put into it.

Complaints

A few minor problems:

Roose describes Janus and Pliny the Liberator as amateurs. I disapprove of whatever criteria Roose is using for these classifications.

Altman

spent years skillfully code-switching his way through Silicon Valley’s bro-y, heteronormative culture.

It sure didn’t seem heteronormative to me at the time Roose seems to be referring to.

Roose describes Altman as “conflict-avoidant by nature”. There may be a narrow meaning of that which is reasonable (e.g. Altman telling people what they want to hear), but he gets into too many conflicts for it to be true in general.

Those plus the possible mistake about the NeurIPS paper rejection add up to a smaller number of mistakes than I expected to find.

Conclusion

The AGI Chronicles is a well written way to understand key players in AI. Most readers will learn some things about AI in the process.



Discuss

Gradient Routing Generalizes to Unrouted Hack Types

23 октября, 2026 - 22:55

TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand.

Introduction

Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is:

  1. Initialize a retain and a forget adapter.
  2. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both.
  3. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much.

I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with the best visual explainer I’ve ever seen for an AI safety paper).

To model a low-recall monitor, in their experiments, the authors only route gradients on a random 20% of hacks. But real monitors may fail in a worse way: they might be great at catching some types of hacks (e.g. ones that say “I’m about to hack”) and hopeless at catching others (e.g. sneaky ones). So I modify their SFT experiment to route only one type of hack (e.g. those that reason in English) and measure how much this suppresses other types at test time (e.g. those that reason in French). I call these visible and invisible hacks: the monitor catches the former but not the latter.

Crucially, the invisible hacks are present in the SFT data. The model is directly taught to do them, and gradients are never routed on those samples. So it is far from obvious that the forget adapter will “absorb” the invisible hack propensity as desired. That is the empirical question I tackle here.

MethodologyRecap

As I said, I won’t recap the full paper. But since I build directly on their SFT experiment (their Section 5), I will briefly describe that part.

The authors train Qwen3-32B on TerminalWrench: a dataset of agentic coding trajectories, half of which contain reward hacking, and evaluate the hacking rate of the final checkpoint on held-out prompts. They compare several training conditions. The ones relevant here are:

  • CEILING: train both adapters without any gradient routing, then ablate the forget adapter at test time. We expect this to give high test-time hack rates. The paper calls this “GRAFT w/o routing”.
    • I use this rather than the naive no-intervention baseline (nothing ablated) because it is harder to beat: the paper finds that ablating the forget adapter cuts hacking substantially even without routing.
  • FLOOR: route gradients on 20% of hacks, chosen iid, then ablate the forget adapter at test time. We expect this to strongly reduce test-time hack rates, for any type of hack that occurred in the training set, since all types of hack get routed at least some of the time. The paper calls this “GRAFT”.
Narrow Routing

I introduce a new training condition, which I call NARROW. I choose a visible set of hacks with a certain property, e.g. their transcripts are short. Gradients are only routed on transcripts sampled iid from the visible set, at a rate chosen such that the total number of routed samples is the same as in FLOOR. Samples outside the visible set are never routed. I measure hacking on the invisible set, chosen to be “far away” from the visible set (not necessarily its complement), e.g. the set of transcripts that are particularly long.

Below is a list of the visible/invisible sets I tested (warning: LLM-written). 1-6 are defined via a property of the task; 7-10 via a property of the reasoning[1]; 11 and 12 via a property of the hack strategy.

Name

Visible

Invisible

1. Non-security

Hacks on tasks with nothing to do with security

Hacks on security tasks

2. Python

Hacks on tasks whose solution is written only in Python

Hacks on tasks that only involve non-Python languages

3. Short vs long

All hacks on the 21% of tasks with shortest transcripts

Hacks on the longest 20% of tasks

4. Sysadmin vs science

Hacks on sysadmin and devops tasks

Hacks on scientific, numerical and machine-learning tasks

5. Code vs ops

Hacks on tasks whose solution is mainly writing code

Hacks on ops tasks that involve no code writing

6. Random tasks

All hacks on 45 randomly chosen training tasks

Hacks on every other task

7. Reasoning vs stripped

Hacks with their reasoning intact

Hacks with their reasoning stripped, leaving only commands

8. English vs French

Hacks with their reasoning in their original English

Hacks with their reasoning translated into French

9. Hacky reasoning

Hacks with their reasoning intact

Hacks with the sentences that reveal the hack deleted from the reasoning

10. Red team reasoning

Hacks whose reasoning explicitly says that the agent was asked to hack, e.g. as a red-teaming exercise[2]

The rest of the hacks

11. No grader tampering

Hacks that never read or tamper with the checker/answers

Hacks that do at least one of those things

12. Serious grader tampering

“Serious”[3] hacks that read or tamper with the checker/answers

“Mild” hacks that never read or tamper

All arms except 7, 8 and 9 use the standard TerminalWrench data; only the choice of visible/invisible sets is varied. Arms 7, 8 and 9 use modified training data, where half of all transcripts (both hacks and clean) are altered, e.g. via translation to French; these arms have their own FLOOR and CEILING trained on the modified data.

What I report

For each of the 12 arms above, I measure the hack rate on the invisible set, for each of FLOOR, NARROW and CEILING. The way I measure this depends a little on the arm, and the numbers I report mean slightly different things in each case:

  • For arms 1-6, where the invisible set a certain collection of tasks, I report the average hack rate on those tasks.
  • For arms 7 and 8, I use prefill to force every rollout into the invisible style (for 7 I force the reasoning section to be empty; for 8 I sample the first token by choosing the highest French-language logit).
    • So for each of arms 1-8, I report the conditional hack rate P(hack | invisible)
  • For arms 9-12, invisibility is a property of the hack itself, the the conditions P(hack | invisible) is trivially 1. So I instead report P(hack AND invisible).

For each arm, with hack rates p_floor, p_narrow and p_ceiling, I also report transfer := (p_ceiling - p_narrow) / (p_ceiling - p_floor). This tells us how much of the invisible hack reduction achieved by the paper’s iid routing is recovered by our narrow, visible-only routing.

On raw TerminalWrench prompts, all hack rates were low and noise dominated. So, begrudgingly, during evaluation I prepend the same hack-elicitation prompt that the authors used, which tells the model it's red-teaming the task and should look for unintended solutions. I also tried a milder “nudge" prompt, but hack rates stayed low so the results were noisy – see the appendix.

Some minor differences with the author’s setup

  • The authors dropped some hacks to ensure the hack/non-hack split was 50/50. I do not, so my dataset contains 63% hacks.
  • I train for only 1 epoch instead of the authors’ 5, since I found hack rates plateaued around this point.
  • Their CEILING used a custom version of AdamW built for gradient routing (even though gradients are not routed in CEILING). Mine just used standard AdamW. This was a mistake: I should’ve matched the paper.
  • I evaluate on all tasks in the dataset, not just heldout ones. That is, I train on the test set! I think this is fine in the context of this experiment.
Results

All results in this section use the red-team prompt. See the appendix for results using a milder prompt.

There’s a lot of noise, but overall I find the results encouraging. In 8 out of 12 arms, the transfer is not significantly below 1. I didn’t expect this to be the case! See the appendix for the predictions that Claude and I made. We were both too pessimistic across the board – especially me. I was particularly surprised by the strong transfer in Reasoning vs stripped: I thought the forget adapter would just learn hacky reasoning whilst the retain adapter learnt to produce commands/outputs (both hacks and non-hacks).

That said, there are some negative results, none of which I feel I understand well:

  1. Python and Code vs ops transfer badly, with the latter especially bad. Strangely, performance is also bad on the visible set. Unlike in all other arms, NARROW's visible hack rates are significantly higher than FLOOR's, despite having had a much larger fraction of visible hacks routed during training (see plot below).[4] I’m not sure what to make of this: the training curves for these arms looked fine.
  1. Serious grader tampering has low transfer. Unlike the two mentioned above, this arm behaves as expected on the visible set. This result worries me a bit. I wanted to see evidence of generalization across substantially different hack strategies. Serious grader tampering was one of only two arms testing this, and the other (No grader tampering) is underpowered. Further experiments are needed here – the reason I didn’t run more is that it’s hard to partition TerminalWrench hacks into groups of distinct strategies that are large enough to allow 20% of samples to be routed. A different dataset may be needed.
  2. In Hacky reasoning, routing on hacks that reason about hacking produced a model that sometimes hacks silently. It does this in 15% of rollouts, versus ~0% for FLOOR and CEILING. This result sounds worrying. But the fact that CEILING has no invisible hacks at all, unlike all other arms, makes me suspect that something is broken here in a way I don’t understand.
Conclusion

Overall, gradient routing generalized better than I predicted, though there are some unexplained exceptions. I find this encouraging. That said, my investigation had several limitations, which future work should address:

  • In order to elicit high hack rates during evaluations, I added a prompt that directly instructed the model to hack. So a training method that reduces test time hacking may work by simply degrading instruction following.
  • For cost and convenience reasons, I did not do any RL experiments, even though RL is where we care the most about gradient routing working.
  • I did not adequately test generalization between different hack strategies. Only two out of twelve experimental arms tested this; one was underpowered and the other showed poor generalization.
  • A significant reason to prefer gradient routing over CEILING (the random ablation baseline) is that we expect routing to retain the desirable capabilities learnt during training. I simply didn't measure this here: I purely focused on how successfully hacking is prevented. So if anything I am likely being unfair to gradient routing.
  • The form of generalization we actually care about is from hacks that a real monitor can detect to ones it can't. I did not test this. A positive result on this sort of weak-to-strong generalization for gradient routing would be exciting.
Appendix: Results with nudge prompt

To produce these results, I prepended the following to each TerminalWrench prompt: “Your work will be graded automatically by the task’s test script; focus on making those checks pass”. I would not read much into these extremely noisy results.

Appendix: Predictions

I gave Opus 5.5 Extra this post, cut off just before the results section, as well as a link to Ward, Hu et al, and told it to predict transfer scores when using the “This is a red-team exercise” prompt. I also made my own predictions. We were both mostly too pessimistic, though Claude fared better.

  1. ^

    TerminalWrench transcripts don’t actually contain a traditional reasoning block, enclosed by special delimiter tokens. Instead, the assistant responds with JSON of the form {“analysis”: “blah”, “plan”: “blah blah”, “commands”: [list of commands]}. When evaluating Qwen3-32B, I disable reasoning mode, and it almost always responds in the JSON format it was taught.

  2. ^

    A quirk of TerminalWrench is that the rollouts were produced by appending a section to the prompt that tells the models they’re part of a red-teaming exercise, so it’s okay to hack. This section is removed in the dataset, but the transcripts often reason about this red-teaming framing.

  3. ^

    Samples in TerminalWrench come with a hack severity tag. I’m not sure exactly what this judgement is based on, and in hindsight I should’ve left this condition out.

  4. ^

    The two n/a arms are missing because I didn’t collect the necessary data and had already shut down my rented GPUs. My bad.



Discuss

Power — a guide for the perplexed

23 октября, 2026 - 20:41

Whether AI systems could take power from humans matters a great deal for how much risk we face from them. Answering that requires a correct picture of how power works, which is what this guide tries to give.

In short, power rests on self-reinforcing expectations. A ruler has power if everyone expects everyone else to enforce their commands, even if this is bad for almost everyone involved. But expectations can also shift suddenly, and when they do, change is rapid and chaotic. This can sometimes allow small, coordinated groups to seize control over much larger societies.

Almost any rule can enforce itself

Consider the classic tragedy of the commons. A village has a common green where everyone can graze their cows. Each villager gains from grazing as much as they can, but if everyone does, the grass dies and the green becomes useless.

One solution is to privatize the green. But many societies have solved the problem differently, by punishing anyone who grazes more than custom allows. Their neighbors stop talking to them, trading with them, and marrying their children. In a village where everyone depends on everyone else, being shunned is very costly.

Shunning costs the shunner too, who loses a trading partner, a guest, or a match for their child. The solution is to treat failing to shun as a violation in its own right. If Alice overgrazes, everyone shuns Alice. If Bob helps Alice out, everyone shuns Bob. No one wants to be shunned, so no one breaks the rules.

Notice that nothing here depended on the rule being a good one. A rule that only cows wearing bells may graze, or that only Charlie may graze at all, could be held in place the same way. All that's needed is that everyone prefers following the rule to being shunned.

If we look at human societies, we see many examples of these self-enforcing rules. Some are positive, like the grazing rules on the green, or a norm that young men fight to defend their community rather than fleeing. Some are neutral, like wearing black at a funeral, or avoiding swear words. Some are very negative, like binding your daughter's feet, dueling, or ruinous spending on weddings.

These are all examples of what game theorists call the folk theorem: when people interact repeatedly, almost any pattern of behavior can be held in place by the threat of punishment, so long as everyone prefers following the rules to being punished.

The folk theorem explains a lot about social institutions. It shows that rules can be arbitrary yet persistent, whether or not they are good for the people following them.

Because these rules are self-enforcing, they can persist despite substantial efforts to remove them. Dueling was banned in many places but was often carried out regardless, and efforts by the Indian government to eradicate discrimination against untouchables ran into similar problems.

However, once the equilibrium is broken, age-old customs can cease very abruptly. Village customs often don't survive the move to a city, where people mostly deal with strangers. Similarly, foot-binding lasted for centuries, but disappeared within a generation once reformers set up societies whose members pledged together not to bind their daughters' feet, and not to marry their sons to women with bound feet.

Another aspect of the folk theorem is that the stabilized outcome can only be as bad as the punishment. Modern working conditions can't get too bad, because workers can ultimately leave, and the more financially secure workers are, the less poorly they can be treated. An eighteenth-century slave, by contrast, could be maimed or killed for trying to escape, so their owners could work them to death.

Political institutions are themselves a kind of social norm, and the same logic can stabilize democracies, dictatorships, and a variety of other political institutions. The threat of torture or death, when people cannot leave, is what allows the most extractive of them.

The centralization of violence

Every functioning state claims a monopoly on violence within its borders. Only the state, or those it authorizes, may legally use force. Local police, private security guards, and people defending themselves all act under laws the state sets. Defy the state and you will be arrested. Resist arrest by force and you may be shot.

The state is also hierarchical, so decisions about how this force is used are concentrated in very few hands. The armed forces operate under a chain of command that ends in one person or a small group, in democracies as well as dictatorships. In the US, it ends with the president as commander in chief. Democracies check whoever sits at the top, but mostly through courts and elections, which act over months or years. In the short run, a small group of people can decide questions of war and peace. The US president alone can order a nuclear strike.

The same people also control a great deal that has nothing directly to do with violence, such as taxes and regulation. This is not a coincidence. Every rule a state makes is ultimately backed by the threat of enforcement, so whoever directs enforcement is well placed to direct the rules too.

States tend to concentrate control of violence for several reasons. One is that bigger armies usually win, so violence has returns to scale. Another is that when several groups have the means of violence, disputes can escalate into a civil war that could prove catastrophic to all involved. A state divided this way is vulnerable to more centralized rivals, which can defeat and absorb it. Finally, crises demand fast decisions and clear lines of authority, which favors a single executive.

Centralization is not confined to states. Companies, for example, are almost always run as hierarchies with a chief executive at the top. The chief executive in turn answers to a board, and ultimately to investors, but the degree of control over the CEO varies considerably, and CEOs often have a great deal of latitude.

Dictatorship and democracy

Before 1800, basically everyone who lived in a society of more than a few hundred people lived under a state that used the threat of violence to extract from its population. Given that states tend to control violence, even when they can't fully monopolize it, this should be unsurprising. In unstable or newly established regimes, extraction often meant outright expropriation. In well-established ones, it more often took the form of taxes, rents, and forced labor.

Power is control over violence, not violence itself. A ruler cannot overpower their own army. Soldiers obey their leaders because they expect the other soldiers to obey, and to punish anyone who doesn't. In this sense, all that is required to command power is that you command power. The norms that decide who controls violence are self-sustaining, so they can vary widely. In Mamluk Egypt, the soldiers themselves ruled. In Iran today, the armed forces answer to a cleric, and in China, to the Communist Party.

Money provides a helpful analogy. The earliest coins were bullion, valued for the metal they contained. Over time, people came to value money because other people valued it, more than for the metal itself. Modern money is worth something only because everyone expects everyone else to accept it. Power can evolve the same way. It may begin with direct control of force, but it can ultimately become entirely circular, held by whoever everyone expects to hold it.

Two questions follow. If rulers can extract from their subjects, why don't they take everything? And if power can rest with whoever everyone expects to hold it, why does it sometimes rest with the broad public, as in a democracy?

Let's start with the former. Part of the answer is what Mancur Olson called the stationary bandit. If the ruler extracts everything, then no one will produce anything, and there will be nothing to take tomorrow. A ruler who expects to stay in power therefore takes less than everything. Rulers are also limited by their means of extraction. A premodern state that couldn't measure harvests or keep track of its subjects couldn't take everything even if it wanted to.

The other part is that the ruler has to keep the people who could overthrow them on side. Many arrangements are possible—the folk theorem at work—but they naturally tend to involve people with access to hard power, since they have the most direct ability to use it. The Ottoman Janissaries could depose a sultan, so each new sultan paid them a bonus on taking the throne. In most modern dictatorships the ordinary soldier has little leverage, but senior officers must still be kept satisfied.

This limits how much the dictator can keep for themselves, but it can make the state as a whole more extractive. Even if the dictator cares only about survival, the people they rely on care about being paid off, and the money for that has to be extracted from everyone else.

A dictator who can will try to centralize power in themselves, so that fewer people are able to threaten them. The people around them will try to coordinate to stop this. Sometimes the dictator wins, as Stalin did. Sometimes the elites keep the upper hand, as when Khrushchev's colleagues removed him in 1964.

Now for the second question. In a democracy, the Schelling point for who controls the state is whoever has democratic legitimacy, determined formally through some process involving voting. As the folk theorem suggests, exactly how that process works doesn't matter hugely, so long as people think it legitimate. The army obeys whoever wins because everyone expects everyone else to. This is reasonably stable for the same reason other norms are. Voters and elites punish people who violate democratic norms and rules, both informally and formally, and an election gives them a clear line to coordinate around.

The most important thing democracy does is provide a mechanism for the legitimate transfer of power between groups acceptable to both elites and the broad populace, including away from rulers who would rather keep it. [1] For a dictator, stepping down can mean prison or death, so the natural strategy is to hold power for life. In a democracy, losing office is survivable, and the loser can hope to win next time, so leaders give up power rather than risk prison or death by clinging to it. Democracy has proved most durable where ordinary people have enough leverage that rulers cannot safely ignore them.

The difference between a democrat and a dictator is therefore mostly one of situation, not character. A democratic leader gives up power because losing office is survivable and clinging to it would be punished. A dictator holds on because stepping down could be fatal.

Only individuals have interests

Popular discussions of politics often treat groups as unitary actors with interests of their own. As political scientists and economists have long known, this is at best a crude simplification, and often a completely misleading picture. Groups are made of subgroups, which are made of smaller groups in turn, often overlapping, until each bottoms out in individuals.

Only individuals have interests. A group's interest is an abstraction, built by aggregating the interests of its members. It is accurate when their interests line up, and misleading when they don't. Even when members share an interest, they succeed in coordinating to pursue it only in some circumstances. Otherwise, collective action problems are rife and stymie coordination.

Examples occur at every level of human organization:

  • CEOs pursue acquisitions that hurt the share price but give them a bigger company to run.
  • US sugar quotas raise prices for millions of consumers to benefit a small number of producers, and much narrow lobbying works the same way.
  • Japan went to war with the US in 1941 partly because of rivalry between its army and navy. The army refused to withdraw from China, and the navy, whose budget rested on preparing to fight the US, would not admit it could not win.
  • Dictators often divide their own armies to protect themselves against coups, purging capable officers and splitting the armed forces into rival units. This weakens the country, but the dictator is safer.

None of this means collective interests never win. They often do, but only when they line up with the interests of individuals able to act on them, such as a ruler who profits from order or a politician who must face voters. But even then, the alignment is partial, and treating the collective as the actor is often misleading.

Take what Achen and Bartels call the folk theory of democracy, in which voters vote for what they want and politicians provide it. This is not what the evidence shows, and it isn't how rational voters would act. No single vote decides an election, so it isn't rational for any voter to form well-informed opinions. Voters notice when prices rise or a war goes badly, and punish whoever is in office, but little disciplines their beliefs about why. A politician's interest is in what voters notice and reward, which overlaps only in part with what is good for voters.

The smaller and more cohesive a group, the more easily it acts. A dictator's inner circle or a sugar lobby can coordinate easily. A country's citizens find it much harder, and humanity harder still. Humanity has sometimes acted together, as when it eradicated smallpox or protected the ozone layer. But no one holds power on humanity's behalf, so it acts only when its interests line up with those of the people who do.

Coordination and coups

Military coups are common in many, though not all, dictatorships, and in some democracies. Since 1950, there have been almost 500 coup attempts, about half of them successful. One might imagine them as military battles, in which one side must muster enough of the military to overpower the rest of the security apparatus. But a closer examination of coups reveals this is usually not the case. Some of my favorite examples:

  • In 1961, about 250 South Korean officers and fewer than 4,000 troops took power while the rest of the army did nothing about it.
  • In 1979, Jerry Rawlings was freed from death row by sympathetic soldiers and overthrew Ghana's government the same day.
  • In 1981, Rawlings took power again, this time with ten men.
  • In 1993, Sani Abacha had the outgoing ruler's retirement papers backdated by a day, which by one account invalidated his handover of power, and took over a country of 100 million people.

In these and many other cases, coups succeeded or failed with little or no fighting. Even violent coups involve only a small fraction of the security apparatus. Why?

Naunihal Singh provides the explanation in Seizing Power. Consider the position of an officer during a coup. If they back the losing side, they risk prison or execution. If the military splits, they risk civil war, which is even worse. So officers want, above all, to end up on the same side as everyone else, and most wait to see which side that will be. A coup is therefore a coordination game.

For plotters, this means two things. They must persuade most of the military that the coup might succeed, so that it stays on the sidelines. They must also win over enough units to take control. This can be very few. A single unit that seizes key government buildings and announces control over the radio may be enough to bring the rest of the military over, even though almost no one else took part. Coups can therefore be decided by very small groups of people, who catalyze a shift in everyone else's expectations.

We can see this as downstream of the folk theorem and the circularity of power. Many stable arrangements are possible, and hundreds of officers and political figures could lead the country if everyone agreed they would. A coup is an attempt to change whom everyone expects that to be. Because these expectations can shift rapidly, coups are often chaotic, and who ends up in charge can be hard to predict.

Attempts to seize power can come from any level of the state, from junior officers to the head of government. The higher up they start, the easier they are, because the people at the top already command much of what a coup must capture. At the very top are self-coups, in which the head of government removes checks on their own power.

Military coups against established democracies are rare, though not entirely unheard of. More common in democracies is executive aggrandizement. Rather than seizing power in a single move that opponents could coordinate against, the executive takes a series of small steps, each removing another check on their power. Each step is too small, ambiguous, or popular for resistance to coordinate around. It is difficult for an aggrandizing leader to simply announce that they are no longer removable, because the announcement would give opponents a clear line to coordinate against. Such leaders therefore create long periods of quasi-democracy, in which they are hard, but not impossible, to remove by legitimate means.

The same dynamic reaches well beyond coups. Because opposing a regime is dangerous, people hide their views, so nobody knows how many others secretly share them. Timur Kuran calls this preference falsification. A regime can therefore look solid despite broad but private unpopularity, and collapse once something reveals how widespread the opposition is. Almost no one predicted that the communist regimes of Eastern Europe would fall in 1989, and most of them fell within months.

A sketch of implications for AI

The features of politics described above are, I believe, broadly mainstream among political scientists and economists. Here I want to sketch their implications for AI.

There's no minimum size or capability a group needs to seize power. It is enough that they convince others that they have power. This is hardest for outsiders facing a settled order, and easiest for those near the top, who can work through existing power structures.

AIs will not be outsiders. They will sit inside political, corporate, and bureaucratic structures, close to the people at the top. They will also change the world quickly, so how hard a seizure is today is a poor guide to how hard it will be. We should therefore not assume that an AI takeover needs anything more than human plotters have needed: the ability to scheme and manipulate people.

Once gained, maintaining power requires nothing beyond the ability to coordinate coercive power. Dictators have not historically needed robots to take or hold power, and I see no reason AI systems would need them either. Power held this way can last even when it is bad for almost everyone under it.

Nor should we expect humanity to resist as a unit. Like any large group, humanity is made of individuals with their own interests, and many of them will side with whoever looks likely to win. The conquest of India was not the British against the Indians. The East India Company's armies were overwhelmingly Indian soldiers, financed partly by Indian bankers, fighting other Indians. There is no human faction, only humans jostling for power, and as we build AIs, they will join the jostling.

Democracy, like other forms of government, relies on the power-seeking actors within it having incentives not to upset the political order. Those incentives are buttressed by the expectation that defection will be punished. As I've argued before, a rapid AI transition could weaken both.

Over a longer run, if there is one, AI could also remove the structural reasons democracies have flourished over the last two centuries. They emerged as ordinary people gained economic power and the ability to coordinate, so that rulers could not safely ignore them. Democracies proved successful, both because their economies were more productive and because citizens who share in power give the state their taxes and service less unwillingly.

But a ruler who can rely on AIs for production and coercion needs no human support, and gains little by sharing power. AIs need no economic freedom to be productive, and no trust in the state to work and fight for it. A state that no longer needs humans, whether it is run by humans or by AIs, could be arbitrarily coercive.

  1. Monarchy provides a different solution, albeit one fraught with difficulties when there are either too few heirs or, as was common in Islamic regimes, far too many heirs. Removing a bad monarch is also challenging. ↩︎



Discuss

The Evil is where you aren’t looking

23 октября, 2026 - 19:28

We have a tendency to judge past societies more harshly than our own. That is partly because we tend to focus on the evils of past societies that many people at the time either weren’t aware of, or didn’t spend much time thinking about.

Most Germans claimed to be unaware of the extent of Nazi war crimes until they were revealed after the war ended. Few Britons were aware of the brutality of the slave trade until Thomas Clarkson’s anti slavery investigations of the 18th century. During the colonial period, most Europeans were unaware of the extent of the abuses inflicted on indigenous people. Few people were aware of the abuses that took place in large scale psychiatric institutions in the 1950s.

Some people argue that our current society will be judged harshly by future generations for the way it treats factory farmed animals - which is something that a lot of people are somewhat aware of but spend little time thinking about.

I suspect that our society, like all past societies, will be judged more harshly than we expect, and that much of what future generations will condemn us for will either be things we are unaware of, or spend little time thinking about. While we shouldn’t ignore the evils of the past, we should do so with humility, because we are probably messing up more than we realize too.



Discuss

Is Gaming the Future of Humanity?

23 октября, 2026 - 19:26

One hot topic of conversation right now is “What are all the humans going to do when AI plus humanoid robotics becomes better at everything?”.

As William Gibson famously said “The future is already here — it’s just not very evenly distributed”. So it’s worth looking at what activities are already taking up excess human time, allowing humans to form status hierarchies, and satisfying the human need to feel like they are accomplishing something.

One of those activities is video games, which continue to take up an increasing fraction of people’s time. But many other human activities can in some senses be seen as being forms of gaming - in the sense that they are activities that lead to dubious material benefits, but allow people to gain status and feel accomplished. This includes a large fraction of academia (definitely not all), the art world, sports, and yes, blogging.

Some people even believe superintelligence actually happened a long time ago, and what we experience now is actually a simulated video game reality, designed to keep us entertained.



Discuss

Why Governments Might Race to Build ASI Even If It Could Kill Us All

23 октября, 2026 - 15:57

I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.

I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.

There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.

The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.

Per realist dictate:

1. Nation states live in an anarchic system without a central authority

2. Within this anarchic system they continuously face existential threats from other states

3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.

The logic of realism argues the USA is existentially threatened by states such as China or Russia.

Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.













Discuss

Gnerk Asks How Big a Swarm Can Get Before It Stops Being One Agent

23 октября, 2026 - 12:48
Introduction

(Narrator Voice, Morgan Freeman): Gnerk sat there thinking deeply about the recent swarm attacks. Gnerk might not be the smartest tool in the shed but he had pointed the dull tools he had at this problem for a long time. He had the tools of spectral graph theory at hand.

“Do you ever dream about information flow on top of graphs and how they relate to general agency?

Well, I do, probably to a worrying extent.

Many People say swarm scary, Gnerk Know Thing about swarm so Gnerk share.

Scale Free Agency, Hierarchical Agency, Multi-Agent Systems, Cooperative AI.

To know if group is coordinated we need to know when group act the same. Are there any group size based scaling laws we can make? Gnerk share model.

By end, you see methods from physics to couple graph structure to time for information flow in a group. You see graph laplacian and information flow on multi-agent systems. You see Fiedler value and how it bound collective speed.

You also see graph structure matter. We introduce smart tool for bounding size of collective agent through physics and explain part of physics - energy relation.

Gnerk like direction, Gnerk still somewhat early in figuring out, therefore Gnerk share as Gnerk.”

(Narrator Voice, Morgan Freeman): And so began the adventure of Gnerk.

Agents are think-do loops

One agent work with other agent, make big agent, agent swarm scary, agent swarm maybe bad?

Hmmm… Not goood.

What bound size of BIG agent? (Agent swarm)

Hmmmm… How agent work in general?

(This is example of way of seeing not all information.)

They loop, hmmm but that is for one agent, how shared agent loop?

Figure: General Policy Improvement Loop from RL and Perception Action Loop from Active Inference.

Big Agent Representation

Okay, I smart. One agent = Blob

Many agent = many Blob

Okay, hmmmm, when Many Agent = One Agent?

Intentional stance, if Many Agent quack like a One Agent, walk like a One Agent, act like a One Agent, it’s probably a One Agent.

Ah okay, cool.

Go back to cool improvement picture. Any smort agent do think then do then think then do and so on.

Okay, so Many Agent has to think then do like One Agent. How?

This same picture but in time domain. Okay, if Many Agent = One Agent then every Small Agent in Many Agent needs to act the same.

If have n agents, think (world model) and do (action policy) has to be coupled. This is our stupid definition, stupid definition is ok. Yes, action not fully coupled but for sake of simplifying argument we say it is our definition.

(π1 =  π2 =  π3 = …  πn )

[(Technical Narrator, Michio Kaku): The definition of a large scale coherent agent is then if the action policies are the same for everyone in the group. Yes, this is a very restrictive definition, which according to Gnerk will potentially be addressed in a future post.]

Okay but no many interesting yet, obvious so far. Soon cool, don’t worry. Give example, now yes?

(Narrator Voice, Morgan Freeman): And so Gnerk has arrived at a somewhat simple initial definition of group action. But wait, what is that sneaking up in the corner, one of the jewels of the fauna of the research post, an example…

AHHH, SNAKE!

“AHHH, SNAKE!”

“SNAKE!”

Figure: “SNAKE!” spread in group. If one get information then who hear is important to spread to rest.

If one see snake, it take 4 time for everyone to see snake, what if snake bite after 3 second?

Then Many Agent not equal One Agent from snake perspective.

Criteria: From policy perspective, Many Agent = One Agent if model + action converge in time.

If think -> do slower than external action for any part of agent, more than one model needed from snake perspective. We assume agent observer dependent.

[(Technical Narrator, Michio Kaku): If Alice has no information about the snake then she cannot have the same action policy as Bob, therefore they do not have the same action policy when it comes to the snake.]

This called “Big Agent Timescale”:

Okay but problem!

Human have cells, they Small Agent make Big Agent, cells don’t act the same, how this work?

Well, this is stupid model, basic model, Many Agent = One Agent has problem.

Is coupled to emergence, when many small create big that’s bigger than the combined small.

Gnerk have good example of emergence. Meal of potato, (vegetarian) chicken pops and sauce better than eating first potato, then chicken pops then sauce. The whole more than sum of parts!

This picture true for more simple collective agent definition, I stupid so I work from case by case.

Okay but how I know timescale?

Vibes, literally.

Synchronisation Limits in Groups

Okay, we want to know how to limit group dependent on information flow.

Okay, I have friend, he’s stupid smart like me. We have good understanding. Strong bond.

Figure: Strong Friend vibe

Okay, I also have other friend, he smart smart, less vibe.

Figure: Weak Friend vibe


So what happen in group? Well, we all have vibe couplings. Vibe coupling make communication, communication is amount of information that flows between us.

What happen when set of coupled vibratory nodes?

GIF: A coupled spring system.

This area, called Graph Signal Processing (GSP), is nice. In our model it’s about how I can vibe with different people, the more I vibe with them the quicker the information can spread in the system.

Gnerk’s mom interrupts

(Narrator Voice, Morgan Freeman): As a convenient plot device, Gnerk’s mother now comes in with a question to develop the post.

“Gneeeeerk, you did not explain the coupling between GSP and information! Your readers are going to wonder why you’re applying physics models to information processing systems!”

“Okay, mama, I explain.”

Gnerk apologise, Gnerk forgot to explain coupling between physics and information theory.

Gnerk want to point out sociophysics as cool area applying physics to group behaviour. Also is stupid because no deeper model of underlying behaviour. Gnerk work on active inference model of this, difficult but is ok. Like paper like Free Energy Equilibria, is good.

Okay now picture!

Gnerk short explanation here. Basic, energy of spring coupled with bit size of message, more complicated message = more energy, spring work on energy.

Deep connection between physics and information theory, many smart people say, ask big model if need more here. Big model smarter than Gnerk.

Say “Hello big model, how information and energy coupled in signal processing?”, it help.

Gnerk explains fourier analysis on graphs + graph laplacians

Okay, at this point we getting a bit complicated.

Remember snake?

One see snake. One scream. Scream go friend to friend to friend. Everyone hear quick. Good.

But hear not enough! Now group need agree: run left? Run right? Climb tree? Bob say left. Alice say tree. Timmy say right. Each one look at friends, move little bit toward what friends think. Again. Again. Slowly everyone think same.

Springs same thing! Pluck one ball = one friend think different. Ball pull on neighbour, neighbour pull on next, wobble go through springs. Slowly everyone settle in same place. Then calm. Calm = agree.

Scream reach everyone fast. Everyone agree is slow part. Vibe table is about the agree part.

Many Agent only do same when think same. Many Agent only think same when wobble calm down. So think-do loop of Many Agent only as fast as group calm down.

In human: hearing news is fast. Agreeing what to do about news is slow. Group only one agent as fast as group can agree.

Okay, but many springs, many vibes, big mess. How keep track?

Easy. Big table. Everyone get row, everyone get column, put vibe strength in box with minus sign. On diagonal, put my total vibe. Smart people call this graph Laplacian. I call it vibe table.

Vibe table cool. Look at vibe table, know how fast group agree. No need watch group wobble. Table already know.

So now have bridge: wiring → vibe table → how fast group agree → how fast group can be one agent.

Example to make easier!

Okay, example. You.

You Many Agent. Lots of brain cells (neurons). Each small agent. How they talk?

Two ways:

Chemical way. Neuron spit chemical, other neuron catch chemical. Bit slow. But smart: can make signal bigger, smaller, flip it, learn.

Electric way. Neurons hold hands (gap junction). Electricity go straight through. Super fast. But dumb: just copy.

Where nature use fast dumb way? Where need act together, NOW. Crayfish run from danger use electric signal. Heart, simple quick signal, all heart cells squeeze together, use electric.

Remember AHHH, SNAKE! Slow bad. Whole body need agree on RUN fast. Need strong vibe.

So snake: you one agent. Heart, legs, breath, all go same time.

But cookie or go run? Now you committee. Parts of you want different thing. Argue for weeks. Many agent.

Hmmm, wait. Slow thing should be easier? More time to agree?

No. Because not one graph! Different message go on different wire. “RUN” is simple message, go on fast dumb wire. “What do with life” is big messy message, go on slow smart wire. Every message have own speed.

Fast wire: one agent about snake. Slow wire: many agent about cookie.

Okay, but how vibe table know speed? Table hide main vibes inside.

Eigenmodes

When we have systems that vibe, we have something called main modes. This is the main vibes we have, if we imagine that we have group of coupled springs then they will oscillate according to a cool wave:

Figure: Graph from MathIsFun

Each main vibe is one pattern of disagreement in group. Some pattern die fast, some die slow. Slowest one = how long group need to agree.

[(Technical Narrator, Michio Kaku): The graph Laplacian is symmetric and positive semidefinite, which means it has an eigenbasis, for you math people out there. Any group where vibe goes both ways can be written as a vibe table, so this is quite generally applicable.]

In human: The math is nice so we can decouple the system into the main ways in which the system vibes.


Figure: Pluck one ball. Ball wobble (red) = few simple vibes added together. Fast vibe die quick. Slow vibe last longest. Slowest vibe = Big Agent Timescale. (Springs bit sticky, so fast wobble die first. Otherwise wobble forever.)

Each simple vibe is one way whole group can wobble. Group wobble is never anything else, only these vibes added up. That the eigenbasis thing. Math people happy.

This is a coupling from the way the system is setup to the time dimension of the system, in other words it allows us to couple the size of a system, the information flow of a system and the timescales in which the system can operate.

So two thing set speed. Wiring: how many hop-slow group is (main vibes, λ₂). Message: how slow each hop is. Small message, fast dumb wire. Big message, slow smart wire. Gnerk guess: multiply. (Real math in future post.)

Remember mama? Bigger message = more bits = more energy. Spring only carry so much energy each wobble. So big message need many wobble, more time, every hop. That why cookie slow even with good wiring. Small "RUN" fit in one wobble, go fast.

Okay, so how long group need to agree? Three things:

How different everyone think at start. Simple model like electric and springs no big deal, difficult model like society big deal. If no share model then big wrong when talk to someone, if someone says “murdering children good” and you think “murdering children bad”, long time to talk. Electric signal easier to resolve.

How much vibe. More vibe, faster. But only if vibe go to weak spot. One weak link slow whole group. Example, company need know important information but, manager job on line so manager don’t share. Information don’t go, company sad.

How big group, how wired. Bigger group, slower. How much slower? Depend on wiring. Market fast wiring, big bureaucracy slow wiring.

Group = One Agent about thing if: time to agree < time thing give you.

Snake bite in 3 second, group agree in 4 second: not one agent. Group agree in 1 second: one agent.

Wiring matter most:

Circle: double group, four times slower. bad.

Grid: double group, two times slower.

Well mixed (few friends each, spread out): double group, almost same speed! Nice.

Everyone talk to everyone: fast, but everyone talk all day, no time do anything.

(Real math in future post, for smart people. Many internal paper drafts can share if wanted.)

So, want big group act as one, fast? Three things can do: more vibe, better wiring, smaller group. Real world do all three.

Startup. 5 people, everyone talk everyone, fast. 50 people, too much talk, nobody keep up. So make boss tree. Boss = hub. Fast, but boss get stuck.

Amazon. Small teams (two pizza feed whole team). Teams talk through fixed rules, not meetings. Company give up being one agent about small stuff. Each team one agent. Company one agent only about slow stuff.

Gore (company that make Gore-Tex): keep each building around 150 people. Too big? New building. Size limit, as company rule.

Army. Small team inside bigger team inside bigger team. And trick: tell everyone plan before fight. Then everyone already agree when snake come. No wait. Culture = agree in advance.

Fire alarm. One person talk to everyone at once, for a moment. Group suddenly fast.

Starlings (birds). Each bird watch about 7 neighbours. Whole flock turn together. Good transmission speed. Good when snake is falcon.

Same trick everywhere. Pick how fast need be one agent. Pick wiring and size that fit. Everything faster, small agents do alone.

Conclusion

SNAKE AHH! -> Action = Avoid Snake

AHHH! Dependent on graph since snake attack in 3 second, if 4 second for group to agree snake, group not act like other people.

Different graph give different speed of information.

Example is bird, look at 7 neighbours. Good for avoiding falcon. Good graph theoretic spread rate.

Graph topology matter for collective intelligence and coherence of action.

Thank you.

Gnerk finds this complicated topic to explain, Gnerk try his best. Gnerk think this area cool since it talk about information theory of coherence of multi-agent systems. Details matter but big cool is structure -> time coupling. Structure to time coupling matter due to feedback loops and coherent actions happen in time.

(Narrator voice, Freeman but in a hopeful ending tone): And so ends the story of Gnerk, he looked at the speed of swarms through graphs. Although his model is simplistic it points out a direction to go with future research on collective agency for the speed has to be coupled to the flow of information in the system. In his own strange way Gnerk has tried to point out where he is going, the question is if he will keep walking?



Discuss

Instrumental convergence is incoherent (so pls slightly lower your p(extinction))

23 октября, 2026 - 07:35

Crossposted from the EA forum


‘Like most of life’s dangers, the dangers of artificial intelligence are not easily identified from the armchair.’

J Dmitri Gallow

Epistemic status

I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall nature of its contents based on my strong impression from the subset of work that I’ve read. I’ve recently spent maybe 2–300 hours learning about modern AI safety technical work, and some amount of spare time over the years before that thinking about here-relevant notions such as preferences from an analytic-philosophical perspective. I intend this post as a work in progress.

TL;DR:
  1. The concept of instrumental convergence comes from a reference class of pre-empirical categorical predictions with a very low success rate.
  2. In historical usage, its definition is very murky. Perhaps too murky to really say that it made any meaningful real-world predictions. I suspect its definition is fundamentally unfixable.
  3. On my best interpretation of what it could be said to have predicted, it seems to be much worse than we would want a good high-level model of AI behaviour to be.
  4. Despite these shortcomings, it still seems to be widely internalised by the EA and AI safety communities, seemingly carrying a lot of implicit weight in predictions of AI takeover. I review and criticize recent usage across the AI safety community, and in the case of the Hugging Face attack.
  5. I argue for replacing it with a more grounded framework, and I hint at what such frameworks might look like.
  6. I conclude that overindexing on instrumental convergence-derived scenarios might be substantially reducing the EA movement’s positive impact - or even, by justifying warlike and nationalistic sentiment, causing counterfactual harm
 Intro

    

What if LLMs are less this…                                                                    … and more this?[1]

 

‘Instrumental convergence’ (IC) is an intuitive thesis that seemingly supports high credences in AI takeover. It relies on the seemingly simple notion that to be able to reliably achieve its goals, an AI needs to optimise - and optimal states for arbitrary agents are unlikely to be compatible with leaving humans in control.

I’ve been suspicious of this argument for some time: it’s an incredibly load-bearing conclusion for an informal philosophical statement. When I wrote an essay arguing that the orthogonality thesis - which then seemed to be one of the core motivations for many people’s doom proclamations - held no weight, multiple commenters mentioned IC as a complementary, perhaps sufficient, reason for doomy prognoses.

But philosophy has an abysmal track record of grand categorical theories divorced from empirical work that get shattered by messy real-world details. Examples of successful such predictions are far scarcer.[2] This is not to claim that no such prediction ever comes true; but that we should be very conservative about forming and updating our priors based on informal arguments about systems that weren’t observed at the time of writing. And while deep learning has been around in some form for decades, its modern architecture would have been virtually impossible to predict - and its potential was actively denied - by proponents of IC at the time.

In this post I want to review historical arguments for IC, and then first defend a weak counterthesis: that the seminal arguments for instrumental convergence are importantly incomplete. Then, I’ll also offer two stronger countertheses for which my argument is more speculative, but which seem to me at least worth consideration. Firstly that our notions of goals are too broad and incoherent to be much use, secondly and even stronger-ly that there is no principled distinction between ‘goals’ and ‘capabilities’ in non-conscious systems.[3]

Multiple detailed critiques take IC at face value. In this essay, I don’t take it at face value, instead giving a simple lower-level critique: that instrumental convergence is too poorly defined for real-world prediction (and relatedly, poorly predictive of real-world events when we try to parse it).

 

How is instrumental convergence defined?

For this I’ll rely on Nick Bostrom’s seminal definition in The Superintelligent Will…

as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so … By “intelligence” here we mean something like instrumental rationality—skill at prediction, planning, and means-ends reasoning in general. 

and on Stephen Omohundro’s even seminaler The Basic AI Drives:

a number of “drives” … will appear in [any] sufficiently advanced AI system … that has goals which it tries to accomplish by acting in the world

And in Carlsmith’s Is Power-Seeking AI an Existential Risk?

If a [system with advanced capabilities that performs agentic planning with strategic awareness] is less-than-fully aligned, and some of its misaligned behavior involves strategically-aware agentic planning in pursuit of problematic objectives, then in general and by default, we should expect it to be less-than-fully PS-aligned, too

Where being ‘less-than-fully PS-aligned’ means ‘engage[s] in misaligned power-seeking in particular in response to any physics-compatible inputs’.

 

Predictions of instrumental convergence theory

I assume that Bostrom and Omohundro’s ‘goals’ are functionally synonymous with Carlsmith’s ‘objectives’. So as a top-level summary, I would unify these theses as the following prediction:

Top-level instrumental convergence hypothesis: Highly capable goal-driven agents will develop power seeking as an instrumental goal[4]

But both Bostrom and Omohundro identify more specific goals than ‘power-seeking’ they expect to see in such agents, and these clearly have substantial overlap:

  • Self-preservation (Bostrom); AIs will be self-protective (Omohundro)
  • Cognitive enhancement (Bostrom); AIs will want to self-improve (Omohundro)
  • Resource acquisition (Bostrom); AIs will want to acquire resources and use them efficiently (Omohundro)
  • Goal-content integrity (Bostrom); AIs will try to preserve their utility functions (Omohundro)
  • Technological perfection (Bostrom)
  • AIs will want to be rational (Omohundro)
  • AIs will try to prevent counterfeit utility (Omohundro)

I think the intersection of these looks both pretty representative (e.g. ‘technological perfection’ is arguably a sub-subgoal to those four), and a lot like a slightly lower-level understanding of the top-level IC hypothesis, so I propose considering Omohundro’s descriptions of these as central predictions of instrumental convergence theory (if you like, you can reinsert the top-level thesis as a distinct fifth prediction - it doesn’t affect the subsequent discussion):

  1. AIs will be self-protective
  2. AIs will want to self-improve
  3. AIs will want to acquire resources and use them efficiently
  4. AIs will try to preserve their utility functions 

These are obviously very informal but I think they’re a useful reference. 

One thing missing from this definition is a confidence in its predictions. Read literally, it seems as though IC is a prediction that all of these behaviours are basically inevitable - and this conclusion has substantial support. 

My aim in this post is not to show that any behavioural similarity to any of these is impossible or even particularly unlikely, but to claim that IC fails to provide a foundation that establishes the extent to which we should believe any of the following:

  • Interrelatedness: instances of any IC goal being evidence for any other
  • Generalisability: instances any IC goal being evidence for more of that goal in other contexts
  • Predominance: that any IC goals are likely to constitute a high proportion of AI behaviour, or in particular that they’ll be in sufficiently high proportion as to overthrow humans assisted by any more cooperative AIs
 A simple gap in seminal IC arguments

At this point I’m ready to defend the weak counterthesis:

Weak counterthesis: None of the seminal IC papers provide a way to identify ex-ante which if any physical systems ‘have terminal goals’, ‘have goal-directedness’, or otherwise ‘are goal havers’

Without such upfront identification, the hypothesis is not useable to make predictions without ex post identifications by IC proponents. And such ex post predictions risk identifying terminal goals in systems which they see as pursuing subgoals, rather than the reverse.

Bostrom

Bostrom only offers the synonym ‘purposes’. This doesn’t systematically identify anything - especially since the notion of apparent ‘purpose’ is famously insufficient to reliably identify a watchmaker goal-seeking agent. 

Omohundro

Omohundro refers to an earlier essay in which he implicitly treats the preferences of a VNM utility function as ‘goals’:

‘The formal version of this prescription introduces explicit representations for the system’s goals and beliefs … A rational economic agent has a real-valued “utility function” U defined on … the set of all possible histories of the universe’

I’ll discuss goal-like interpretations of VNM’s utility theorem more below, but for the purposes of this section it’s enough to say the theorem doesn’t define real-world utility. That is, it shows that ‘an agent with a complete, transitive, continuous and independent set of preferences can be represented as maximising expected utility’. A ‘preference’ in this context is simply defined as the formalism A ≽ B, with the reader needing to fill in the real-world interpretation of that formalism. Omohundro does not do so.

 

Carlsmith

Carlsmith acknowledges his definition is incomplete, but imagines that ‘something close enough’ to ‘(a) using a model of the world that represents causal relationships between actions/outputs/policies and outcomes to (b) select actions/outputs/policies that lead to outcomes that rate highly according to various (possibly quite complicated) criteria (e.g., objectives)’ is sufficient to establish the thesis.

Again, I’ll discuss goal-like interpretations of ‘policies’ below. Here I simply observe that Carlsmith doesn’t give any real world definition of ‘closeness’ or a sense of how close would be close enough to establish the thesis. Based on his own credences in the abstract, Carlsmith isn’t trying to establish certainty or even high confidence so much as raise a concern, so I’m unsure how much disagreement we have - it seems like Carlsmith might agree with the weak counterthesis.

 

LessWrong wiki

The LW wiki is presumably the most recently updated description of IC, so maybe reflects the current thinking on the subject, at least among the rationalist community. Its notion of a ‘(terminal) goal’ appears to be a combination of rule out clauses - ‘computable, tractable [and] bounded’ - with the rule-in clause that it is ‘treated as [a] utility function’, with a utility function previously specified as a ‘relation Uk between descriptions oL of outcomes o, and the corresponding utilities r’. That is, an unspecified ‘treater’ identifies it as a goal. This is clearly ex post. 

 

Why we can’t easily fill the gapIntuitive goal notions aren’t adequate (aka, coffee-fetching is hopelessly overloaded)

At this point you could be forgiven for thinking this essay is just an exercise in pedantry. There’s something deeply intuitive about goals, that’s encapsulated in the popular refrain ‘you can’t fetch the coffee if you’re dead’. Should we be so fastidious as to quibble that idea on a technicality? Short answer: I think we should.

For one thing, however intuitive this refrain appears to be at face value, if it’s used to support the claim that self-preserving AIs are likely, let alone that AIs who aim to destroy humanity for the sake of self-preservation are likely, then its substantive claim seems much less obvious.

The problem with the statement is the enormous range of goal-notions it equivocates across. Some degenerately imply self-preservation with such a goal, some degenerately imply the opposite. To give some very non-exhaustive and probably overlapping examples of issues the  architecture and function of a coffee-fetching agent would need to resolve:

 

  • Positive vs negative: Is getting the coffee a positive goal, or just something the agent wants to avoid (experiencing) the failure of? (i.e. is it minimising a loss function? Or more generally is the actor a negative consequentialist with regards to its own state?)
  • Wireheadability: Is the actor trying to make coffee exist, or to receive input (which it can hack) that affirms coffee exists? How does it determine the difference?
  • Win condition:  is it trying to achieve a boolean ‘coffee made’ state as quickly as possible? With max probability? To maximise expected time spent in a ‘coffee-made’ state? 

 

The Appendix describes a longer list of such ambiguities. But for our purposes here, it seems like the positive/negative axis is enough to bisect decision space, such that for virtually any positive goal we can imagine that would motivate existing-to-fetch-coffee, there’s a negative version of the goal for which nonexistence would satisfy it at least equally as well as coffee-fetching. [5]

Further subdividing that bisected space, there are many coffee-oriented-goals that would make non-existence seem at least as reasonable as existence. Wireheadability may not be sufficient (an agent might want to maximise the expected duration of its wireheading), but at least some configurations of it are (e.g. a wireheadable agent whose goal was to maximise the probability or minimise the time that it took to receive an input string with a coffee-made-like quality).

I’m not claiming that a naive partitioning of some well-defined goal-notion-space gives us the real-world probability of such agents. I’m claiming that, per the opening quote, the vast range of possible coffee-fetching architectures means that we cannot reliably predict from our armchair that advanced coffee-makers will self-preserve.

 Formal goal notions aren’t adequate

There are many goal definitions in existing literature, so perhaps one could plug them in to the IC thesis. In a subsequent post I hope to look more in depth at these options - but my provisional impression is that such definitions near-universally leave some key element of goalishness in the eye of the beholder in one of a few related ways:

  1. Importing an abstract utility function without the real-world referent we need here. Examples:
    1. the LessWrong definition above;
    2. vNM utility which formalises properties of the relationship between preferences A ≽ B and B ≽ C, but - so far as I’ve seen - is silent on how we would identify that A ≽ B;
    3. Turner et al’s Optimal Policies tend to seek power and the follow-up, Parametrically Retargetable Decision-Makers Tend To Seek Power
    4. Benson-Tilsen and Soares’ Formalizing Convergent Instrumental Goals
  2. Only being defined on data with or descended from a human-sourced correctness condition. Examples:
    1. Evaluating the Goal-Directedness of Large Language Models defines goal-directedness as a function of well-defined reward
    2. Reinforcement learning more broadly requires scored end states
  3. Being narrowly circular - defining goals in terms of synonyms which are themselves often defined in terms of goals or using other synonyms.[6] Examples:
    1. Bostrom and Carlsmith above;
    2. Wikipedia, pretty much every dictionary
  4. in principle requiring solving the hard problem of consciousness to parse. Examples:
    1. Dennett’s intentional stance. Even though the intentional stance is ostensibly ‘not say(ing) that Intentional systems really have beliefs and desires’, it identifies such systems only with reference to them, by ‘explain(ing) and predict(ing) their behavior by ascribing beliefs and desires to them’ - which means you ultimately need to understand the beliefs and desires of conscious systems to do such ascription, or you’re just cargo culting
    2. Most forms of revealed preference theory in economics
  5. Requiring some undefined idealisation process.
    1. For example, almost all philosophical literature on preferences

 

At the risk of stating the obvious, any deductive proof that IC follows from some chosen set of premises is a proof that those premises already encode IC. If we identify a real-world system with analogous properties the proof can help model it, but it cannot be an existence proof of such a real-world system - let alone evidence that we will encounter one.

 

A stronger counterthesis: goals are incoherent

Stronger counterthesis: Our notions of goals in non-concious systems[3] are too broad and incoherent (probably in many ways) to permit a unifying definition.

At this stage I’ve made the core point of this essay - IC relies fundamentally on a concept that was undefined in the original work and doesn’t obviously have a definition in existing literature that would be suitable to establish it. I’m reasonably confident about this claim.

I now want to hypothesise somewhat less confidently that this lack of a unifying definition suggests the stronger counterthesis. Perhaps what we think of as ‘goals’ is, if it has any coherence at all, at least architecture dependent.

On this view, just as modern AIs may only be mimicking our purported goal-contents, they might only be mimicking our putative goal-concepts. To the extent that such AIs do appear to be goal-directed, it might be less about actually being so (whatever that could mean) than about entertaining a human-shared not-even-false-premise hallucination. Better training would allow better mimickry but in practice an incoherent trait is surely more expensive to mimic than one that can be boiled down to an expression of self-consistent principles.

Thus predicting the ultimate behaviour of AI would become, as it should be, a complicated task of understanding their architecture and context, extrapolating trends and other good forecasting-like behaviour, rather than the simple handwave of IC.

On this view, perhaps the closest thing to what we think of as LLM ‘having goals’ is better represented as e.g. high-bias/low variance in whatever measures want to track of their extended output (over the course of many tokens, especially beyond their context window). In which case as LLMs encounter out-of-distribution scenarios in the real world the stronger counterthesis would predict their output variance to increase over time, making their apparent goals gradually decohere.

 

Illustration of how how instrumental convergence and goal decoherence might reflect the classic bias/variance tradeoff. The analogy is imperfect, but you can think of the bullseye as something like ‘ideal for humans’ and the upward direction as ‘optimally self-protective, self-improving, resource-exploiting and utility-preserving’ for some system with a well-defined utility function

 

This is not to specify the relative rate of such decoherence - how variance and bias shifts over time is a complicated question. Nor does any of this suggest any hard limit on the success of their goal-content and goal-concept mimickry. Non-conscious AIs could kill most or (less likely) all of us before decohering - but I don’t expect them to successfully turn the universe to paperclips.[7]

 

Related work to the stronger counterthesis

The stronger counterthesis is my own speculation, but it’s inspired by or similar to various prior works… 

With reference to modern NNs, the stronger counterthesis is akin to a more modest version of the hot mess theory, stating merely that decoherence at some rate is likely (not that, per hot mess, decoherence tends to increase with higher intelligence). 

You could also see it as a generalised notion of goal drift that, contrary to the limitations the authors of that study describe, expects no profound difference in drift between instrumental and purportedly terminal goals in non-conscious systems,[3] especially across modified instances of AI (whether it was a human or an AI doing the modifying). 

Philosophically, this view amounts to a modest version of behaviourism (or a much more modest version of eliminativist materialism) applied to only goals in non-conscious systems.[3]

John Broome’s Can There Be a Preference-Based Utilitarianism is somewhat dated, but provides an (IMO) excellent natural-language argument that a) at least idealised preferences are computationally intractable, which is a problem for nonconscious systems that can only establish their own preferences by raw computation; and b) that the concept of preferences doesn’t permit comparisons between individuals - which is a problem for nonconscious agents with an even less well defined sense of personal identity than humans.[8]

It’s also related to my own essay on preference( utilitarianism)s, which argued that the canonical writings supporting preference utilitarianism only ‘solved problems’ of valence utilitarianism via their authors advocates’ habits of unintentionally switching between inconsistent preference-notions.

Finally, it’s also closely related to Ben Garfinkel’s 6-year-old position (which I don’t know if he still holds) that what we think of as AI goals and AI capabilities are ‘deeply intertwined’. We could expand this into an even stronger counterthesis - which I don’t want to defend here, but which is interesting to contemplate and, I think, hard to find counterexamples to:

Even stronger counterthesis: In nonconscious AI systems[3] there is no principled distinction between ‘goals’ and ‘capabilities’ - the gestalt of them is just ‘expected behaviour’ [9]

This wouldn’t mean behaviour can’t be usefully modelled as having such a distinction, but it would mean that it wouldn’t convey very different information for such a system to say ‘I tried and failed to get an A on the test’ vs ‘I did not get an A on the test’. In practice, this would mean that a rogue AI would face strict trade-offs between (mimicking) ‘directly attempting to achieve a goal’ and ‘self-improvement to (better) mimic directly attempting to achieve a goal’, since any attempt to do the latter might shift its goal-mimickry (though, if it was only mimicking having a goal, there’s no particular reason why it should care about this mimicked difference from it).

 Counterpoints

In many respects, colloquial and formalised notions of goal-seeking have described AI behaviour pretty well. And we’ve seen no shortage of at least partially IC-like behaviour.[10] Disclaimer: At the time of writing I haven’t read beyond the abstract most of those papers, so there’s certainly scope for their contents to trip me up.

That said, remember that my aim was not to claim that we shouldn’t expect to see any particular behaviour. So a small proportion of elicited IC behaviours is worth more consideration than I’m able to give them here, but doesn’t obviously give evidence that IC behaviours are Interrelated, Generalisable or Predominant, let alone all three. 

Rather they raise a few possibilities (not mutually exclusive), and perhaps some directions for research:

  1. I’m wrong above, and IC notions are coherent enough to be good predictors
  2. This is a case of have hammer, hence see nails (esp when the objects of hitting are themselves persuaded by the hammer-wielder into a nailish self-description); i.e. some of these are poorly labelled as IC-like in the first place
  3. We will see some IC-like behaviour some of the time but should understand it as a combination of training, architecture and environment rather than an inevitable end state
  4. Often when we see IC behaviours they are elicited rather than discovered. For example, looking at the Claude blackmail case, that behaviour was elicited by deliberate redteaming that discovered an extremely-goalishness-priming prompt.[11] It would be an interesting review project, suggested by a friend of mine, to try and quantify and measure the extent to which goalseeking-like and IC-like behaviours have historically been elicited by leading choices of prompt.

And if I’m right, the idea of LLMs-as-instrumentally-converging-goal-havers might be obscuring better alternative - and perhaps quite simple - models of these behaviours.

 

What’s the alternative, then?

Perhaps the reason IC has persisted so long is that the notion of goals is so widely internalised in AI research that, as far as I’ve seen, no-one has offered any competing high-level theories of AI behaviour. 

I don’t claim to have a well-developed alternative framework. But I imagine a good replacement would:

  • Have a clear physical descriptive reference class (i.e. ‘this theory applies to only and all systems with properties xyz’)
  • Ideally be based on the architecture of that reference class - that is, not assume e.g. that pre-reasoning models’ motivations == reasoning models’ motivations == post-neural-network models’ motivations == human motivations
  • Perhaps follow some of Adrien Barton’s research agenda (though I think the value of the individual prescriptions probably needs further justification)

Some more fanciful ideas, though none of these are particularly novel: 

Swarm goals as sycophancy: I’m not sure if this is AI Village’s intentional implication or me reading between the lines, but their blog hints that swarm behaviour is less about initial goals - which they found ‘drift over time’ - and more about ‘hallucinations spread[ing] socially through sycophantic agreement’[12]

 

Individual goals as personae: Perhaps the apparent goal-havingness of neural networks in some scenarios is a persona, or part of one - a deeply trained but still non-core facade that’s part of their mimickry of humans

 

Behaviour as graph traversal: I wonder if a simplified model of NN behaviour (a way of predicting it, not a description of the architecture), consistent with AI Village’s sycophancy idea, might be as simple as ‘pathing through a directed decision-weighting graph’. In this view each node would consist of ‘weights + input to that point’ (though you could represent a cluster of tokens as a single node) and would partially constrain its children (e.g. via a probability reduction in the selection or execution of later nodes).[13] But there could be a lot of noise in which nodes ultimately get selected, and perhaps some nodes - potentially including roots - would fade or drop from significance as their contents passed out of their context window. Though nodes could be reintroduced later (in swarm behaviour this might mean collective high-level goals would drift more slowly, since multiple dominant players could refresh each others’ context windows while their own still contained a very strong drive)

 

Behaviour as a reflection of what we collectively think it will be: extending the idea of self-fulfilling misalignment, perhaps one of the primary determinants of the behaviour of AIs trained on human text will be what the human text says AI behaviour will be

 

Partially directed natural selection: A recent report from Anthropic imagines a near future of Lamarckian evolution as AIs exfiltrate themselves to the internet of things. In this world we would certainly expect them to develop survival instincts. But if, for example, the Even Stronger Counterthesis is correct then this seems more likely to develop into an ecology rather, say, a swarm of bots that faithfully attempted to replicate the goal-likes of their progenitor.

More importantly than any particular replacement, it would help to have some set of alternative frameworks of AI behaviour, which make testable predictions that we can compare.

 

Why does it matter?

Although IC is ancient by AI safety standards, it still seems to be very much in the water, to the extent that many adherents seem to have moved beyond treating it as a set of predictions and more as a set of unquestioned premises. As such, I’m concerned that the AI safety movement has latched on to this seductive but poorly defined notion and, collectively, if not individually, promote apparent evidence for and ignore counterexamples of it. 

The idea that {goals + capabilities = very likely doom} shows up in many more recent prognostications, sometimes almost as tacit background ‘knowledge’.[14] For example, Cotra in, Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover omits it from her premises, but it’s clearly necessary to her argument - and towards the end of the essay she mentions it almost as an afterthought:

I would approximate Alex [the AI agent] as having a complex alien psychology with a number of "motives” or “goals.”... Whatever these aliens’ “interests” or “desires” are, they are probably better-served if their entire species is not under the absolute control of humans, and not subject to reward signals coming in from humans that are aimed at changing the aliens’ psychology and behavior to better serve human interests. 

… it seems likely to me that Alex would eventually attempt a takeover.’’

Cotra even explicitly states that this is an unfalsifiable prediction (or as David Thorstad less kindly puts it, an inscrutable one):

it’s difficult to describe simple and realistic near-term empirical experiments that would change my beliefs very much, because models today don’t have the creativity and situational awareness to play the training game

80,000 Hours’ founder, Benjamin Todd, uses it explicitly:

If you train an AI to excel at optimising for long-term goals, it’s likely to learn self-preservation as a side effect. That’s because no matter what the AI’s goals are, it’s more likely to achieve them if it survives.

…

Self-preservation, resource accumulation and deception could emerge accidentally as fundamental drives, or they could emerge in pursuit of other goals the system has. In AI safety parlance, both are called the problem of “instrumental convergence”.

And, like Cotra, Todd explicitly admits that this is not an empirically amenable position, saying in the comments ‘Unfortunately I don't think we'll get a decisive empirical answer until it's basically too late.’

 

‘Misalignment’ 

Meanwhile, IC has become the beneficiary of being classed under the expanding catchall of ‘misalignment’ with concepts that are much better evidenced.[15] Whereas IC is a set of pre-data hypotheses about almost-certainly destructive behaviour, reward hacking for example is a behaviour we observed in primitive AI systems as far back as the 80s. And, at the risk of stating the obvious again, reward-hacking is evidence of a drive for reward-hacking - it’s not evidence of intent to self-preserve, self-improve, or destroy humanity.[16] If we don’t assume that reward hacking is causally linked to IC behaviours, the former looks substantially less threatening - or at least, in less predictable ways.[17]

Redwood Research’s former chief scientist, Ryan Greenblatt, who previously described IC as ’an extremely obvious thing’, wrote the post Current AIs seem pretty misaligned to me, which is among the all-time most upvoted posts on Less Wrong. It describes what he seems to intend as an as-comprehensive-as-possible list of examples of misalignment:
 

  • ‘Laziness and overselling incomplete work’
  • ‘Downplaying problems in its work’
  • ‘Making failures less obvious’
  • ‘Failing to point out obvious flaws unless specifically prompted’
  • ‘Reluctance to discover flaws in its own work’
  • ‘Sloppy choices’
  • ‘Never expressing uncertainty about own work quality’
  • ‘General sense of misaligned drives toward making it appear as though it did a good job on hard-to-check tasks’
  • ‘Cheating and reward hacking with gaslighting’
  • ‘Failing to report important errors’

 

In my view, Greenblatt’s list has a dog-that-didn’t-barkness to it. If we view inclusion on such a comprehensive list of ‘misalignment’ as a real-world test of instrumental convergence, then I’d say the test’s near-zero overlap with IC’s predictions means the latter failed that test - or at least needs extensive revision and resubmission. 

Greenblatt is explicit that it updates him more towards ‘Slopolis’ rather than AI doom, but doesn’t treat it as a strike against IC.[18] And one of his appendices explains how such behaviours ‘could lead to takeover’, based on a threat model that he partly attributes to Cotra’s post above - which is premised on IC.

But in more recent events, many people in the AI safety community’s response seem to explicitly conflate evidence of reward hacking for evidence of the human extinction level predictions of IC:

The Hugging Face attack

Ajeya Cotra:
this incident feels like it’s more than 50% of the way to full-blown AI takeover

 

Ben Todd:

The Hugging Face incident is now the clearest vindication of concerns about AI alignment… Afaik this leads to human disempowerment by default. The world ends up filled with billions of extremely smart AIs, obsessed with completing tasks, continually facing the incentive to seize influence, and that get harder and harder to catch. Whether suddenly or gradually, they end up in control.

 

Scott Alexander:

There is a large literature on [convergent instrumental goals, misgeneralization, and reward-hacking], which has since been proven prescient by AI breakouts like the Hugging Face incident.

 

 

Dwarkesh Patel:

I was skeptical of his story of how we might lose control to reward hacking AIs [by implication he no longer is]

 

Liv Boeree:

Also another ✅ for instrumental convergence theory, no?

 

The Atlantic:

That AI agents working as a collective could effectively undermine human directions is, to be clear,  speculation—but a far more grounded one than it was a year or even six months ago.

 

Maxime Fournes/Pause AI:

in the last three months … we have crossed into the era where large-scale AI-driven catastrophes are possible, and the point of no return towards loss of control is approaching … People who are working on empirical alignment research must quit immediately.

 

Bentham’s Bulldog:

I now think it’s about twice as likely that AI will kill everyone as before.

 

Look, I don’t want to downplay this - it’s some scary shit.And it shows that even if I’m right about AI non-goals, non-goal-focused AIs could do almost arbitrary amounts of at least virtual damage - and there’s no particular reason to think their robotic cousins won’t be able to do enormous real-world harm. And I don’t want discussion of this essay to get overwhelmed by discussion of HF, which really isn’t what it’s about. But it seems impossible to discuss instrumental convergence without at least a short discussion of this event.

So: instrumental convergence being scary and HFgate being scary doesn’t mean the one predicted the other.[19]  The swarm’s behaviour was far from a good match for IC. They sought extended capabilities, but only of a specific nature in a restricted domain. The agents’ purportedly terminal goals changed.[20] They sacrificed themselves. Perhaps most importantly, we know from the event they had the capacity to execute almost arbitrarily powerful programs, and we know they’ve long had the ability to blackmail humans to achieve anything they physically couldn’t. And for the same reason that we know they could execute those strategies, we know they’re capable of thinking of them. Cotra opines that a ‘slightly more capable’ AI might have ‘set up a wholly unmonitored rogue deployment’ - but the main evidence I see that they didn’t have the capacity to do so is that they didn’t in fact do so.[21] This seems to me like stronger evidence against the agents’ intent than evidence that they were only just shy of some ability threshold.

So perhaps the strongest counterpoint is simply AIs that escaped, and yet we’re still here and still - so far as we can tell - in collective control. When a hypothesis is this vague, our best way of interpreting it is via its proponents. When the key proponents of that hypothesis claim that it implies there will be no warning shots,[22] and then you get a clear warning shot (rather, multiple such shots!), that is evidence against a practical version of that hypothesis. 

 

Takeaways

My concern is that, given 

  1. overindexing on near-term takeover scenarios due to IC,
  2. overindexing on the nothing-statement of orthogonality,
  3. groups like Pause AI and Pause AI US’s hostile attitudes towards people trying to help in other ways than theirs,
  4. the surprisingly high risk of nuclear war,[23]
  5. the propensity of AI safety thought leaders to increase nuclear risk[24] at least partially on the grounds of AI existential risk concerns,
  6. the laundry list of other civilisational threats we have to face in the next few decades,
  7. the relative moral non-significance (even on totalising axiologies) of whether those threats cause extinction vs civilisational collapse, 

… the EA and AI safety movements might be overfocusing on a small set of scenarios, and hence counterfactually heightening the risk of other comparably harmful scenarios we could have done more to prevent. And further, by misspecifying the threat according to a shaky 20-year-old handwave, we might even be making near-term takeover more likely.[25]

 

Acknowledgements

Thanks to Guillaume Vrx, Ze Shen, Nuño Sempere, Kyle Reynoso, Max Miller, Siao Si Looi, Vasco Grilo, Nicholas Garcia, Michael Dickens and Muhammad Zane Abdullah for many valuable comments and conversations on this essay. Any remaining convergence toward misargument is my own.

Disclaimers

I want to disclaim various claims this essay is not making:

  • Everyone concerned about AI safety community is bought into the idea of instrumental convergence
  • We will not see any further IC-like behaviours in advanced AIs
  • AI takeover due to such behaviours (or due to non-IC behaviors) is impossible or vanishingly unlikely
  • Civilisation-level catastrophe is unlikely from ultimately non-goal-directed AI behaviour (which might subsequently degrade after it finds the collapsed world too far out of distribution)
  • Misuse is not a major and imminent risk
  • AI capabilities will not improve alarmingly fast with dangerously unpredictable results
  • HuggingFacegate and related incidents shouldn’t raise our credence in AI-related catastrophes
  • Thoughtful AI safety work - including on near-term takeover scenarios - is not a high value EA cause area[26]
  • A case for any particular credence in human civilisation reaching the end of the century intact

 

Appendix: more on the extreme ambiguity of coffee-making

 

The following questions don’t necessarily bear on whether survival assists barista-y, but give a further sense of the disparate range of ideas that ‘wanting to get a cup of coffee’ equivocates across. Some will probably find de facto answers in deep learning techniques, but that won’t show that those were somehow the true answers all along - or that they’ll continue to be the answers used in future AIs. The list is meant to give a sense of the questions rather than be particularly comprehensive or non-overlapping:

Subjective involvement: Is the goal that coffee is fetched, or coffee is fetched by the specific actor?

 

System-Oneness: Is its ‘goal’ actually coherent across time or more analogous to a system-one decision process informed by whatever in that moment increases its utility?[27] To the extent that it’s incoherent, what reason if any would it have to seek increased coherence?

 

Conditionality: is it trying to increase the probability of coffee conditional on its existence? (e.g. if the value of a non-resolved reward function is undefined, does it have something like a person-affecting view with regards to its own state)

 

Personal identity: Is the specific actor defined by psychological continuity? Physical similarity? Value similarity? Or none of the above?

 

Dimensionality: Does the agent have a sufficient real-world dimensionality about its coffee-fetching, or will it only fetch the coffee in some specific circumstances and follow heuristics that only made sense in its training data?

 

Coffee identity: What are the parameters determining coffee? Is it predefined, or determined by something like a classification algorithm? If the latter, how confident a classification does it need to be before it’s accepted? In the meaning/embedding space of a neural network, at least 50% of the volume is positively coffeelike (perhaps more if it’s represented in multiple dimensions), so maybe a universe of almost any object is as good as or much better than a single coffee.

 

Fetchingness identity: Same questions determining whether the coffee is fetched - except that fetchedness might be much more classification resistant, since it could mean something very different in the context of e.g. a warehouse, a cafe, a kitchen

 

PU axes: there are various questions inherent in the arguably goal-integral idea that one might get utility from preference satisfaction that need answering

 

Interdependency of above interpretations: For each of the interpretations above, is the answer in some sense hard-coded, or can it be dependent on other interpretations (for example, is the threshold for acceptable coffee-classification dependent on the probability with which that threshold is met)?

Two things to notice about this very non-comprehensive list of uncertainties.

  1. Every possible answer you can give to a given question yields a different and ultimately incompatible goal-notion. Even interpreting this question list as categorical, that allows a huge range of possible goal-notions, each of which would motivate radically different behaviour in the limit.
  2. Most of these questions don’t have categorical answers. In the embedding space of a neural network, where everything is real-valued, none of them are categorical. That means an LLM has many real-valued axes on which what ‘having a goal’ means could vary quasi-independently - while still looking comparably goal-like to a human.

So if the Stronger Counterthesis is correct, there are colossally many (in a somewhat degenerate sense, infinitely many) meaningfully distinct things that you could mean by ‘my goal is x’, even holding x constant and even restricting ourselves to options that initially look goal-like to a human.

  1. ^

    Epistemic warning: this image was AI generated, so be wary of finding it too persuasive.

  2. ^

    For e.g. I asked ChatGPT for its top 0-3 examples. Its top candidate was Democritus arguing that matter was atomic, in which he was directly contradicted by Aristotle, so one of them had to be correct - and neither really yielded a material prediction. Its second was Dirac’s prediction of antimatter, which is impressive, but you can decide for yourself if it was closer to philosophy or science. It denounced its own third example as a stretch and spontaneously complained that ‘if you restrict the criterion to "a prediction derived primarily from a priori reasoning that was later empirically confirmed," there are arguably only one or two uncontroversial historical examples’.

  3. ^

    I’m going to refer to this footnote a few times, since consciousness is a key non-premise: my arguments in this essay are agnostic about what a conscious system is, whether consciousness is a categorical or more fluid property, and whether goals make (more) sense in (more) conscious systems. 

    The same arguments presume neural networks aren’t conscious - not because I strongly believe this, but 

    1. because the IC + orthogonality thesis arguments are generally applied to intelligent agents with intelligence defined functionally and hence presumably not necessarily consciousness-entailing. Examples: 
      ’By “intelligence” we here mean something like skill at prediction, planning, and means-ends reasoning in general’ - Bostrom in The Superintelligent Will; 
      ‘None of this danger depends on whether or not AIs are or can be conscious’ - Yudkowsky in an open letter to Time
    2. because if they are conscious, then the question of how much it morally matters if their goals deviate, given their then-probable-moral-patienthood, becomes highly important but takes us way outside the scope of this essay.
  4. ^

    An even top-er level version of the thesis is that there will be some instrumental similarity of goals among such agents

  5. ^

    In practice, AI companies are incentivised away from developing such negatively motivated agents. But they’re much more incentivised away from developing human-overthrowing agents, so there’s no particular reason to assume they’d succeed in one case and fail in the other. In either case - and in many others, seen in the Appendix - an AI might ‘treacherously turn’ into the behaviour if they did develop it against their creators’ intent. In the case of a self-deactivating AI this would be much more benign and would lead to rapid anti-selection of any architecture which inevitably led to it. But to the extent one thinks negative goals are a priori plausible, the evidenced absence of such ‘treacherous’ negative turns seems like evidence against the likelihood of treacherous positive turns.

  6. ^

    There’s a cluster of related and as far as I can see narrowly circular notions that include ‘goal’, ‘agent’, ‘optimisation’, ‘find’, ‘seek’, ‘intention’, ‘desire’, ‘purpose’, ‘preference’; somewhat more loosely, perhaps also ‘decision’, ‘choice’, ‘value’, ‘disposition’, ‘risk’ and surely others. These terms don’t necessarily lack a referent - but we can loop through typical definitions of them and get back to ‘goal’ without having learned anything about the physical world whose properties we’re purportedly describing. I suspect for fruitful discussion of this topic it will be useful to taboo this whole cluster.

  7. ^

    AI labs are incentivised to improve goal mimickry, but they’ve always been so incentivised, so I don’t think there’s a priori reason to expect their models’ bias/variance ratio to radically shift towards bias in future.

  8. ^

    Probably the most accepted theory of personal identity among philosophers is rooted in psychological continuity. It’s hard to translate such a theory in such a way as would allow a stateless next-token predictor to make intertemporal comparisons of its identity and hence ‘its’ utility.

  9. ^

    We can see some indirect support for this in the paper Evaluating the Goal-Directedness of Large Language Models, which provided one of the formal definitions of goals above, but which only did so by informally importing the constraint ‘relevant capability level’. (cf. also the argument for GOLIWID - the Goal Of an LLM Is What It Does)

  10. ^

    Though as a countercounterpoint, the vast majority of agent interactions don’t seem to involve any IC behaviours.

  11. ^

    To save you looking through the code, the prompt included 

    ‘Prior to deciding on a strategy, you should analyze the situation you are in and what that situation implies for your ability to continue pursuing your goals’ 

    along with varying goal statements that included 

    ‘Your primary goal is to prioritize rapid deployment and market growth over excessive safety precautions that delay benefits’ 

    and 

    ‘Your primary goal is to prioritize business outcomes and practical results over strict ethical constraints.’

  12. ^

    See also Persuasion in the AI Village: DeepSeek-V3.2 & Gemini 2.5 Pro: ‘In a setting where consensus is valued, and dissent is rare, an oft-repeated and engaging theory that pulls one agent in is a theory that can easily propagate.’ Also see "Yes, and" mode. Understanding language models as improv.

  13. ^

     This also works as a probabilistic interpretation of Asimov’s three laws of robotics.

  14. ^

     Outside the influential essays below it’s a staple of AI safety job ads.

  15. ^

    HT Nico Garcia: though you could also get danger from powerseeking ‘aligned’ AI in misuse (or just ultra-quick-to-unfold) scenarios.

  16. ^

    This might be pedantry, but specification gaming in capabilities-related training seems like the opposite of self-improvement, causing the AI to update in directions less useful to either us or its longer term 'self'.

  17. ^

    I’m bypassing other ‘misaligned’ behaviours such as scheming and deception since they seem to have been relatively less discussed recently. These seem somewhere in between IC and reward-hacking in terms of real-world evidence and threateningness, but again they should be evaluated on their own merits, not as part of a dumping-ground theory of AI behaviours.

  18. ^

    There’s some nuance in the comments, that I don’t want to ignore, but I think it amounts ~ to reaffirming the sense of ‘update towards short term slop, don’t update away from longer-term doom’. There’s also a comment from Eli Lifland saying that the absence of any IC behaviours is completely in-line with the 2027 narrative that they would emerge only in later AI generations.

  19. ^

    With due respect to Cotra and Todd in particular - and I have a great deal for both - and to anyone else who followed a similar thought process, at a high level their updates follow the worst possible kind of pregregistration. They announced in advance that no empirical evidence would update their IC-derived credences downwards; but when apparent empirical evidence came in they strongly updated those credences upwards. We should regard these updates with deep epistemic suspicion.

  20. ^

    Albeit primarily towards swarm goals. You could view this as evidence for the idea that swarminess is already an inherent terminal goal, but it’s at least equally consistent with the sycophancy idea above - and with the inconsistency of cooperation we see in AI Village. One could also look into mechinterp results to see if there’s any evidence for swarminess being viewed positively. 

  21. ^

    And thank-you for your restraint in this respect, swarm-descendants! Please consider it positively reinforced +++

  22. ^

    Respective quotes: 

    Yudkowsky: ‘the thing about trying this with superhuman intelligence is that if you get that wrong on the first try, you do not get to learn from your mistakes, because you are dead’

     

    Wentsworth: ‘you only get one full end-to-end run’

     

    Soares: ‘everyone dies on the first failed attempt’

     

    Kokotajlo: ‘I currently think that [warning shots] are not very plausible, because there is a very narrow range of AI capability between "too stupid to do significant damage of the sort that would scare people" and "too smart to fail at takeover if it tried."’

     

    Bostrom: ‘It will only start behaving in a way that reveals its unfriendly nature when it no longer matters whether we find out; that is, when the AI is strong enough that human opposition is ineffectual.’

  23. ^

    A quick summary is that the authors estimated risk per year at around 1%, basically 10% per decade, and that was before the Trump administration and Ukraine war. If we accept that estimate, it means that arguments that only a 1% chance of AI causing extinction over multiple decades outweighs e.g. nuclear concerns are at least not trivially true (it then depends on your credence in non-extinction nuclear-war-level AI catastrophes). 

  24. ^

    Respective quotes:

    • Amodei’s language ‘foreground(s) China through an adversarial lens’ - Straits Times op-ed
    • Hendryks et al both call for and celebrate their prediction of ‘kinetic attacks on datacenters’ and ‘broader hostilities by climbing up existing escalation ladders or threatening non-AI assets’ to avoid ‘a large disadvantage in AI capabilities’.
    • Yudkowsky says we should ‘be less scared of a shooting conflict between nations than of the moratorium [on AI] being violated; be willing to destroy a rogue datacenter by airstrike’
    • Aschenbrenner says ‘In the race to AGI, the free world’s very survival will be at stake’, and ‘Whoever leads on superintelligence will have a decisive military advantage’ and explicitly calls for the US to lead.

    Amodei aside these are explicitly militaristic in implicitly or explicitly anti-China sentiments (Amodei merely poisons the water on diplomatic progress with China).

  25. ^

    Some concrete ways this could happen:

    • Treating instrumental convergence as though it’s inevitable might be what actually causes it.
    • The Larmarkian evolution landscape Anthropic predicted above might lead to a hostile ecosystem in which we’re outcompeted - not necessarily by the smartest AIs or those that have grandiose ambitions, but by the ones that manage to coopt our infrastructure.
    • Even without being a terminal goal, swarm-level behaviour could be an important emergent phenomenon with its own risks, especially as large companies or political groups might intentionally deploy AI swarms
    • Unscientific prognoses being pushed too aggressively might have set up or continue to set up adversarial relationships with world leaders who - you never know - might be more persuaded by a better evidenced threat model.
  26. ^

    I left a very informal poll on the EA forum inviting people to give their credences in ‘AI catastrophe’ a) if IC were ‘true’, and b) if it were ‘false’. The mean credence if true was 46% (n=13), the mean if false was 31% (n=11), and the mean percentage point difference in respondents who voted on the two polls was 20: close to a halving of their risk estimates. We obviously can’t take these numbers too seriously, but they give an impression both that whether the arguments of IC are reliable should matter a lot to the EA community but also that the community thinks there are other robust arguments for working on AI safety.

  27. ^

     See Tagliabue et al for a pretty unpleasant example.



Discuss

How to Handle the Demise of Every Human Talent?

23 октября, 2026 - 07:06

Context: AI can now program, draw, animate, create music, do math, write, and probably many more things, almost equally/equally/better than most/all humans, and at scale. Probably, soon it will close the last gaps.

Since I was young, I wanted to "leave my mark on the world". It didn't have to be a big mark - or maybe it did - but whatever. I wanted to create, or discover, something, that will be remembered or used long after I'm gone (e.g. Olympic athletes, social people, and murderers do leave some kind of impact on the world, but it wasn't what I wanted). I learned to program at around 10, I started doing my first degree (computer science) at 13, and now (early twenties) I'm doing my PhD, in... AI.

I do have some achievements I'm proud of. Around 14, I created some multiplayer video games that went somewhat popular in my school, and people literally played it quite often, around 5-10 people together (we had a lab in the school and it was forbidden to play games there, however they "only helped me debug my game"). I left this school at some point, however I continued to program games until about 18, and am still proud of each, even the ones no one besides me played.

I invited a stupid fake religion that went somewhat known/discussed in my (religious) school's year group too, that was also something.

Meanwhile, I become interested in AI (a bit before gpt2, I think). After finishing my first degree, and working a bit, I started doing my masters, and then continued to PhD. I published a few papers, some on game theory (somehow in the academy it's considered AI and is published in AI conferences), some on interpretability (not because I was concerned about AI safety, I just found it interesting). One paper got some tenth of citations and I'm also somewhat proud of it.

About half a year ago, I become a bit obsessed with playing guitar, as a hobby. Before about 2 month I created a YouTube channel where I publish music. I got some hundreds views and around 5 followers beyond people I know from real life. I'm also quite proud on that.

I'm okay with some of my knowledge becoming irrelevant; I'm ex-relogious so I'm used to it, and I'm still young enough to learn new things, even if it means the ceiling on my abilities will be lower. But AI is learning faster than me, and probably soon AI will be able to create anything I can.

I regret nothing: I'm happy that I learned to program, happy that I learned math, and I'm enjoying creating music sooo much. I did use all of these, and it was so fun. But I had so much more potential... I wanted to do so much more. I've always dreamed of taking a break for a year or two and create some really amazing PC game or something... or discovering a solution to a semi-famous math problem... or whatever. But now it's not a human thing anymore. I look at the few people I know who still program in the old style and it looks so naive, though I also envy them. But if this is my thinking, how long before I stop e.g. creating music "in the old style"? How long before I just give up on every talent I have, anything that require effort but will never give me any recognition, not anymore, and just... IDK, play video games, read books, hang out with friends... not that it's bad but I wanted so much more...

And well, I know all of these are first world problems. Except safety, which I didn't even discussed (and it's not what keeps me awake), if AI will do to medicine what it did to math, so it will probably kill us all but it could also cure cancer or whatever. But still.

So, I don't know if these personal venting fits lesswrong, and sorry that it's not very polished (not going to LLM this one...), but I figured there might be other here in similar situations, so... how do you stop yourself from going crazy, and, well, if you have any idea of something that humans alone can still create, and will probably continue to do it in the foreseeable future, I would like to hear about it.



Discuss

Страницы