Новости LessWrong.com

Richard Ngo challenged me to set a time box and write down as many of the most important features of my formal epistemology as I can in one sitting. Here goes.

Where probability distributions fail......to express beliefs

• There is no probability distribution over mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mrow { display: inline-block; text-align: left; } mjx-mtable { display: inline-block; text-align: center; vertical-align: .25em; position: relative; box-sizing: border-box; border-spacing: 0; border-collapse: collapse; } mjx-mstyle[size="s"] mjx-mtable { vertical-align: .354em; } mjx-labels { position: absolute; left: 0; top: 0; } mjx-table { display: inline-block; vertical-align: -.5ex; box-sizing: border-box; } mjx-table > mjx-itable { vertical-align: middle; text-align: left; box-sizing: border-box; } mjx-labels > mjx-itable { position: absolute; top: 0; } mjx-mtable[justify="left"] { text-align: left; } mjx-mtable[justify="right"] { text-align: right; } mjx-mtable[justify="left"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="left"][side="right"] { padding-left: 0 ! important; } mjx-mtable[justify="right"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="right"][side="right"] { padding-left: 0 ! important; } mjx-mtable[align] { vertical-align: baseline; } mjx-mtable[align="top"] > mjx-table { vertical-align: top; } mjx-mtable[align="bottom"] > mjx-table { vertical-align: bottom; } mjx-mtable[side="right"] mjx-labels { min-width: 100%; } mjx-mtr { display: table-row; text-align: left; } mjx-mtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtd { display: table-cell; text-align: center; padding: .215em .4em; } mjx-mtd:first-child { padding-left: 0; } mjx-mtd:last-child { padding-right: 0; } mjx-mtable > * > mjx-itable > *:first-child > mjx-mtd { padding-top: 0; } mjx-mtable > * > mjx-itable > *:last-child > mjx-mtd { padding-bottom: 0; } mjx-tstrut { display: inline-block; height: 1em; vertical-align: -.25em; } mjx-labels[align="left"] > mjx-mtr > mjx-mtd { text-align: left; } mjx-labels[align="right"] > mjx-mtr > mjx-mtd { text-align: right; } mjx-mtd[extra] { padding: 0; } mjx-mtd[rowalign="top"] { vertical-align: top; } mjx-mtd[rowalign="center"] { vertical-align: middle; } mjx-mtd[rowalign="bottom"] { vertical-align: bottom; } mjx-mtd[rowalign="baseline"] { vertical-align: baseline; } mjx-mtd[rowalign="axis"] { vertical-align: .25em; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c58.TEX-C::before { padding: 0.683em 0.807em 0 0; content: "X"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c2286::before { padding: 0.636em 0.778em 0.138em 0; content: "\2286"; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c2227::before { padding: 0.598em 0.667em 0.022em 0; content: "\2227"; } mjx-c.mjx-c1D435.TEX-I::before { padding: 0.683em 0.759em 0 0; content: "B"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c44::before { padding: 0.683em 0.764em 0 0; content: "D"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c48::before { padding: 0.683em 0.75em 0 0; content: "H"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c394::before { padding: 0.716em 0.833em 0 0; content: "\394"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c221E::before { padding: 0.442em 1em 0.011em 0; content: "\221E"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c2006::before { padding: 0 0.167em 0 0; content: ""; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c1D708.TEX-I::before { padding: 0.442em 0.53em 0 0; content: "\3BD"; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-c1D5A1.TEX-SS::before { padding: 0.694em 0.667em 0 0; content: "B"; } mjx-c.mjx-c1D5BE.TEX-SS::before { padding: 0.461em 0.444em 0.01em 0; content: "e"; } mjx-c.mjx-c1D5C5.TEX-SS::before { padding: 0.694em 0.239em 0 0; content: "l"; } mjx-c.mjx-c1D5C2.TEX-SS::before { padding: 0.68em 0.239em 0 0; content: "i"; } mjx-c.mjx-c1D5BF.TEX-SS::before { padding: 0.705em 0.347em 0 0; content: "f"; } mjx-c.mjx-c1D5CC.TEX-SS::before { padding: 0.46em 0.383em 0.01em 0; content: "s"; } mjx-c.mjx-c2297::before { padding: 0.583em 0.778em 0.083em 0; content: "\2297"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c21A6::before { padding: 0.511em 1em 0.011em 0; content: "\21A6"; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c4B::before { padding: 0.683em 0.778em 0 0; content: "K"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c48.TEX-C::before { padding: 0.683em 0.845em 0.048em 0; content: "H"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c44.TEX-C::before { padding: 0.683em 0.771em 0 0; content: "D"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c4F::before { padding: 0.705em 0.778em 0.022em 0; content: "O"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c7B.TEX-S3::before { padding: 1.45em 0.75em 0.949em 0; content: "{"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c77::before { padding: 0.431em 0.722em 0.011em 0; content: "w"; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c398::before { padding: 0.705em 0.778em 0.022em 0; content: "\398"; } mjx-c.mjx-c25FB.TEX-A::before { padding: 0.689em 0.778em 0 0; content: "\25A1"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c78::before { padding: 0.431em 0.528em 0 0; content: "x"; } mjx-c.mjx-c2229::before { padding: 0.598em 0.667em 0.022em 0; content: "\2229"; } mjx-c.mjx-c228B.TEX-A::before { padding: 0.635em 0.778em 0.241em 0; content: "\228B"; } mjx-c.mjx-c2205.TEX-A::before { padding: 0.587em 0.778em 0 0; content: "\2205"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c27E8::before { padding: 0.75em 0.389em 0.25em 0; content: "\27E8"; } mjx-c.mjx-c27E9::before { padding: 0.75em 0.389em 0.25em 0; content: "\27E9"; } mjx-c.mjx-c2200::before { padding: 0.694em 0.556em 0.022em 0; content: "\2200"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c21D2::before { padding: 0.525em 1em 0.024em 0; content: "\21D2"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D43C.TEX-I::before { padding: 0.683em 0.504em 0 0; content: "I"; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c43.TEX-C::before { padding: 0.705em 0.527em 0.025em 0; content: "C"; } mjx-c.mjx-c2193::before { padding: 0.694em 0.5em 0.194em 0; content: "\2193"; } mjx-c.mjx-c22A5::before { padding: 0.668em 0.778em 0 0; content: "\22A5"; } that says "". That is a support condition about probability distributions, namely, ⨾⨾. Some s satisfy this condition and some do not; but there is no that expresses the range belief itself.
• There is no probability distribution over that says " and are independent". This is an equation condition about probability distributions, namely, . Some s satisfy this equation and some do not; there is no that expresses the independence belief itself.
• There is no probability distribution over that can express a conditional probability distribution , even though this is just as essential a part of a Bayesian reasoner's epistemic state as her prior. A conditional probability distribution is a family of probability distributions indexed by the condition variable, not a probability distribution. There is no single distribution that expresses it.

...to make safety tradeoffs

• Suppose there is an unfair coin, which you know to be unfair (but not exactly how much or in which direction), and you must choose between three options: bet on Heads, bet on Tails, or bet instead on a fair d20 coming up at least 12. Exercise for the reader: Every probability distribution you could possibly believe makes choosing the d20 irrational.
  • The offer doesn't even need to be coming from Omega for the d20 to be a permissible choice; if the offer does come from Omega (who determined which way the unfair coin is unfair by predicting your betting choice), the d20 is arguably the uniquely rational choice, making the Bayesian view, that beliefs are probability distributions, clearly wrong.
  • Maximum entropy does not save you!

Beliefs, according to davidad

Definition 1. Given a state space , we define probability space as the space of all conceivable probability distributions on .

Definition 2. A belief about () is a functional

which is lower semicontinuous (meaning that ).

We interpret as the level of inconsistency between the conceivable probability distribution and one's belief .

Following Lawvere, we order by ("backwards"), and pointwise, so that all conceivable beliefs about form a thin category of beliefs . is a commutative monoidal category with .

Slogan: When one has multiple beliefs at the same time, one's overall belief is simply the sum of its parts.

All other known notions of belief fit in nicelyBayesian beliefs

Definition 3. If one has a prior (big if), then one's prior belief is

Bayesian updating

Definition 4. If one has a belief over a hypothesis space, and the actual state of reality is the product of hypothesis space and data space , then one's prior belief about is the inverse-image functor

Definition 5. An observation is a closed subset .

Example 6. If one observes that , this is the subset .

Definition 7. The belief that corresponds to an observation is

Definition 8. If one has a conditional distribution , then one's conditional belief is

Example 9. If one has a likelihood function , then the previous definition applies with and :

Definition 10. A Bayesian reasoner with prior and likelihood function has belief state

Theorem 11. The two components of a Bayesian reasoner's initial belief state, only one of which is a probability distribution, simply sum when lifted into the belief space, forming the joint belief:

However, this would be meaningless unless we could recover the Bayesian update by also summing the observations in the belief space.

Theorem 12. Whenever the Bayesian posterior is well-defined, then

with a constant denoting the level of incompatibility of the observation with the Bayesian reasoner's initial belief state, namely the prior predictive surprisal, .

Notice that the orthodox Bayesian update's "normalization" to a full-mass posterior distribution silently subtracts away the constant — the amount of incompatibility between the observed data and the statistical model as a whole — which is Deborah Mayo's criticism of Bayesian epistemology in a nutshell.

Infra-Bayesian beliefs (Kosoy and Appel)

Definition 13 (Kosoy and Appel). A homogenous ultracontribution is a nonempty topologically-closed convex down-closed subset of subprobability space, .

Theorem 14. Homogenous ultracontributions (ordered by ) form a full subcategory of . Specifically, given , the corresponding belief is

and given a belief , the corresponding subset of subprobability space is

such that . Furthermore, is a member of iff is convex in probability space, which every is.

Definition 15 (Kosoy and Appel). A homogenous ultradistribution is a homogenous ultracontribution whose intersection with the full-mass face is nonempty ().

Of course, is also a full subcategory of , since is a full subcategory of .

MWER (Halpern and Leung)

Theorem 16. Homogenous ultradistributions are exactly isomorphic to the belief states of Halpern and Leung's MWER framework (Minimax Weighted Expected Regret).

Theorem 17. The updating process of simply adding , which is equivalent to that of Theorem 12, is also equivalent to Halpern and Leung's prescribed belief-updating procedure on the domain of their belief states.

Note: via the transform, this updating process is also consonant with the famous Multiplicative Weight Update family of algorithms (although I am not yet confident about whether e.g. AdaBoost is literally a special case of it).

Probabilistic dependency graphs (Richardson and Halpern)

Richardson and Halpern's PDGs, a common generalization of Bayes nets and factor graphs, take their semantics in functionals which are always lower semicontinuous (though Richardson does not prove this), and therefore every PDG denotes a belief in my sense. Further, the beliefs denoted by PDGs are convex, and thus immediately satisfy the criterion to be transformed into homogenous ultracontributions (since is convex, and composites of convex functions are convex). (However, typical PDGs do not denote homogenous ultradistributions.)

Credal sets (Cozman)

Definition 18 (Cozman). A credal set is a nonempty closed convex set of probability distributions, .

Theorem 19. Credal sets about (ordered by ) form a full subcategory of . Specifically, given , the corresponding belief is

and given a belief , the corresponding credal set is

Previsions (Goubault-Larrecq)

Definition 20 (Goubault-Larrecq). A gamble about is a Borel function . A prevision about is a functional such that and . An upper prevision is a prevision which is sub-additive: . A continuous upper prevision is an upper prevision which is Scott-continuous (for every directed family with least upper bound , ).

Definition 21. A coherent continuous upper prevision is a continuous upper prevision satisfying .

Theorem 22. Coherent continuous upper previsions about form a full subcategory of . Specifically, given the coherent continuous upper prevision , the corresponding belief is

and given a belief , the corresponding upper prevision is

The monad (Mio, Sarkis, and Vignudelli)

Definition 23 (Mio, Sarkis, and Vignudelli). The monad is defined on sets as the set of non-empty finitely-generated down-closed convex sets of subprobability distributions .

Mio, Sarkis, and Vignudelli prove that this monad is presented by the equational theory of semilattices equipped with finite probabilistic choice and , making in a strong sense the smallest semantic universe that can simultaneously interpret finite nondeterminism, finite probability, and partiality (and partiality is, in turn, needed to interpret either inconsistent beliefs or nonterminating probabilistic programs).

Theorem 24. ordered by inclusion is a full subcategory of , assuming is a Polish space. Specifically, this condition implies that finitely-generated convex sets are topologically closed, which makes a full subcategory of .

Discuss

As I write, many former friends of mine are living and working at a monastery in Vermont that I believe is a high-control group, commonly known as a ‘cult’.[1] I say this not as someone who was concerned to see these friends go there, but someone who welcomed and encouraged them to join, as an insider. This letter is an account of what changed my mind—written primarily for anyone considering going there, anyone who loves someone there, and anyone who went there and is still trying to make sense of their experience.[2]

A lot of this is based on direct experience, and also from talking in-depth with dozens of former MAPLE residents and apprentices. About half the quotes in this letter are sourced from linked recordings or writings, and half are from my personal memory. Of the latter, I clearly remember the majority, and some (when indicated) are a close paraphrase.

The “Monastic Academy for the Preservation of Life on Earth” (MAPLE) has existed for over 15 years, and had many hundreds of people spend months or years there. It was founded by its Head Teacher Soryu Forall, who has spent over a decade training in monasteries across Asia, as well as in multiple native American spiritual traditions. He has written a book on Buddhism with Chade-Meng Tan that was personally blessed by the Dalai Lama, and is regularly likened to a modern-day spiritual master by those who train with him. He spends most of each year teaching meditation and giving lectures to residents at MAPLE’s 100-acre plot of land in rural Northern Vermont.

Here are MAPLE’s main hall, dormitory, and two zendos (meditation halls).

I joined the Monastic Academy in May 2020, when around thirty people were in residence there, and over the next 3.5 years I attended twenty-six retreats of 1-2 weeks, most guided solely by Soryu. I also sat two solitary retreats of thirty-nine and seventy-seven days respectively, each spent not speaking or making eye contact with a single other soul, other than short daily conversations with Soryu. In September of ‘23, it became clear to me that I must leave, and did so as part of a group of seven other people (including the executive director at the time) who all had lost faith in the institution.

Many arrived there, including me, seeking to develop the qualities of heart and mind that would enable us to show up to our lives and to the challenges facing society with greater clarity, compassion, capacity, and conviction. In many ways MAPLE delivered: life-long friendships, freedom from past addictions and limiting self-narratives, strengthened character, and, for many of us, profound spiritual openings and realizations. I’m deeply grateful for much of what I received from MAPLE.

Here are some dynamics that I expect will be concerning to outsiders, that I hope give the unfamiliar reader a sense of what I mean when I say MAPLE is a controlling environment.

• All significant decisions rest with Soryu, who has unironically told the community, “This is a dictatorship, and I’m the dictator.” The nonprofit board is Soryu as chairman, and two of his former students, one of which still lives at MAPLE. When residents had criticisms of the direction of MAPLE, Soryu said: “When you’re on the board, we want to hear your ideas. Until then, we don’t. That’s the deal. You don’t go to McDonald’s and say, ‘Should we be a restaurant?’ … You flip the burgers or you quit.”
• Nearly every hour of the day is scheduled and monitored, producing chronic sleep deprivation, overwork and exhaustion. Guilt, shame and sudden rage are directed at residents for ordinary human frailties—breaking an inexpensive item, lying down during a talk, being sick with COVID and needing rest. Once when a resident’s head began to nod during a talk, Soryu made the room stare at his (Soryu’s) face for the remaining ninety minutes, with everyone being made to stand up and sit again each time Soryu caught a single person’s gaze drift, which happened about twenty times.
• Residents are pressured into major life decisions. Soryu pressured breakups with outside partners, instructed couples to have children, and pressed for life-long commitments: “Why aren’t you ordaining [a lifelong commitment to MAPLE]? Come on! It’s just your lives!”; “This is the attitude we need here: Once you begin the training, you either achieve your aim, or die.” I've heard him tell residents they will be insecure forever unless they followed his instruction.
• Residents are asked to publish approved, reverential accounts of Soryu. In scheduled writing sessions (which are sometimes mandatory) they write stories of their interactions with him, which he personally reviews and approves before publication onto MAPLE’s blog: MAPLE Tales. These stories often depict him as a wise figure of immense ability and world-historical significance.
• Residents are surveilled. MAPLE’s Fall 2025 Quarterly Report describes building a “benevolent surveillance system,” which includes recording how long each person takes to finish their chores and posting the times publicly. A microphone now records the near-daily private 1:1 interviews with Soryu—conversations that can be as sensitive as any therapy session.
• Some residents voluntarily attend solitary retreats that run one to three months, during which the only human contact is Soryu, who within that isolation often applies extreme pressure and guilt. Many have come out deeply traumatized.
• Soryu has stated one of the main purposes of MAPLE is to “extricate” the “evil” ideologies that residents enter the community with and replace them with a new trustworthy ideology. Soryu also stated that teachers other than himself that residents study must be both Asian and ordained, excluding the entire Western world. Outside teachers and lineages are otherwise treated as spiritually unrealized or dangerously deluded, and seeing a therapist is read as a sign of untrustworthiness.
• Those who leave with concerns are shunned and recast as traitors, as lacking integrity, malevolent, psychologically disturbed, and intending the community harm.

I further believe MAPLE is also willing to harm its members, due to pathological ends-justify-the-means reasoning:

• MAPLE has world-scale goals. MAPLE’s stated goals include founding the next world religion, leading a "world government" with "teeth" (its own military), opening 500 centers worldwide, and building an artificial superintelligence that will “turn the world into a monastery”—which residents quipped would essentially mean turning Soryu’s mind into an AI god. Soryu says he entered a samadhi deep enough to make psychic contact with the mind of the global AI (what they call a “cyborgregore” or “cybregor”), in the same way spiritual masters read the minds of other people.
• Soryu considers human extinction a noble impulse and an ethically defensible goal. He has repeatedly shared that killing every human on Earth was his explicit life purpose from ages nine to nineteen—with CRISPR (a gene modifying technique that could be used to create a novel bio-weapon) as his best available means—until his vision shifted to awakening all beings as a superior alternative.
• Soryu has stated a strong willingness to let residents come to harm in the service of MAPLE’s mission. He stated explicitly the severity of the trauma to students that he would tolerate: “permanently unable to participate in the economy for the rest of their lives.” He has asked a roomful of us to settle what proportion of students becoming traumatized through the training would be acceptable: “is it 20%? 50%? 100%!?!”. Discussing how MAPLE would deal with societal collapse, he said “if half the [MAPLE] community starts running around like chickens with their heads cut off and die within the first week, just let that process play out.”

Indeed, many have left (including me) with severe psychological problems that were not present beforehand. I am personally in contact with over a dozen people who either had a year of mental health issues afterward, or who are still going to therapy and seeking spiritual guidance primarily for issues relating to Soryu and MAPLE. Beyond those I am in contact with personally, I'm aware of dozens more, and in the time I've been working on this letter more people have come to me, seeking to talk with me for multiple hours, to process their distress from their time at MAPLE (that was over long ago).

I am not in a position to do a systematic survey of negative impacts, but I personally know five residents who I believe experienced some sort of psychosis (e.g. loss of basic functioning, developing excessive paranoia and grandiose beliefs, and losing touch with reality) either during or in the aftermath of training at MAPLE, two of which were severe enough to require involuntary hospitalization and a long road of recovery. Distinct from those people, I personally know five residents who experienced suicidal ideation, including one who survived a suicide attempt within a year of leaving the community.[3]

I'm not certain about each one, and others would disagree about particular cases. At the same time I also am sure that many residents had psychotic episodes that I either wasn't informed of or didn't recognize, so there are likely others I never heard about. Nonetheless, if we go with my current best guess of about five cases over about five years[4], and compare it to there being about thirty-five residents at MAPLE over my three-and-a-half years there, this gives a psychosis rate of about 14%, which is drastically higher than the life-time baserate of ~3% for adults in the US.

Furthermore, many of the impacts I've experienced myself or witnessed in others fall neatly in line with symptoms of Complex PTSD, including panic attacks, an inability to trust or be close to others, feeling permanently "defiled" or "ruined"; and also with this list of post-cult after effects catalogued by cult researcher Margaret Singer, including major depression, excessive guilt, difficulty holding a steady job, and concentration and memory issues. There were alumni who heard Soryu's judgmental voice in their head for months or years, had recurring MAPLE-related nightmares, and were too afraid—even in safe and private contexts—to voice a concern about MAPLE except in MAPLE's own language and framing.[5]

It’s hard to convey the depth of the impact with these clinical terms. I continue to learn new stories of abuse and debilitation, and I know that I’ve heard only a narrow number of the total. Some of them are more egregious than anything written in this letter, yet are either too unconfirmed to be included or I have been explicitly asked not to include.

Some of the harm was visible while we were still there. I watched residents—and caught myself—treating our distrust of Soryu as delusion to be purified through meditation, and reflexively “following the breath” whenever distrust of Soryu arose; monitoring ourselves chronically to stay within the MAPLE-approved way of thinking and being; parroting Soryu rather than thinking for ourselves; growing dependent on his validation and becoming deeply distressed when he withdrew it; taking on excessive shame and guilt; and freezing or fawning in his presence. And I watched the light behind multiple friends’ eyes dim over their time there—with a few of them telling me they felt “dead inside” for months at a time.

It’s a common framing that people who are harmed at MAPLE simply weren’t a good fit for the training. Yet, it’s not just people who’ve been in the training who have concerns or have been harmed. I’m personally aware of several people who report being traumatized by Soryu despite having spent little or no time in the training, but were instead subject to manipulation[6] while in close personal relationship with him.[7]

Again I'll say: there was a great deal of good at MAPLE. While many people were being impacted in the ways above, many were also becoming less anxious, more grounded, more relationally attuned, less distracted and more focused, more capable of doing chores and taking on responsibility without internal resistance, physically healthier, deepening in their spiritual practice, and more. Most people experienced some amount of the positive and the negative, which is a large part of why many former residents say their relationship to MAPLE is "complicated."

The way I’ve described it to friends is that it was the best decision I ever made to go there; the second best, however, being to leave.

The focus of this letter

The primary focus of the rest of this letter is to make sense of MAPLE through a high-control lens.

MAPLE has a lot of methods of controlling its members. I have many more stories to tell about what happened at MAPLE, and to see the forest rather than the trees, I will have to tell many of them and detail many varieties of manipulation over the rest of this letter.

Friends who’ve already read the letter find it disheartening to read. For MAPLE alumni especially it can be difficult. Please be sure to take care of yourself, and take breaks if needed.

High-control literature

Reading the high-control literature was central in my recovery. There is a remarkable consistency in the kinds of dynamics that show up in such communities, and there are researchers who’ve dedicated their lives to analyzing them and organizing their findings into coherent frameworks. They kept describing dynamics that were evidently happening at MAPLE.

Janja Lalich studied a high-control Marxist-Leninist political group she was herself a part of (the Democratic Workers Party), and Heaven’s Gate, the UFO-religious group that ended in the 1997 mass suicide. Across both she found the same dynamics: eroding agency, centralizing power, and making leaving a disproportionately costly choice despite an explicit stance from leadership of “people can leave whenever they want”—findings she organized into her Bounded Choice model.

Alexandra Stein studied Jonestown, Scientology, the Unification Church, and more, and found the same dynamic operating across them: trauma bonds are formed with the leader, even in previously securely-attached people. This became her Attachment-Based Model.

In the 1950s, Robert Jay Lifton studied survivors of Chinese Communist thought-reform prisons, and distilled from their accounts his Eight Criteria of total ideological control—including milieu control, loaded language, demand for purity, and doctrine over person. He later applied them to Aum Shinrikyo, a meditation-intensive Japanese community convinced it was saving the world (and responsible for the Tokyo subway sarin attack).

Jamie Wheal, the developer of the “Ethical Cult” Checklist, is not an academic, but he is still a respected thinker in the larger Metamodern scene that MAPLE is loosely affiliated with, so it seemed appropriate to include him. In this checklist, he overviews three red-flag clusters: the consolidation of charismatic power around a mythologized founder, the construction of in/out group boundaries via messianic purpose and specialized language, and the weaponization of peak experience and healing to bypass member discernment.

The best-known framework, however, comes from a therapist. Steven Hassan's BITE model grew out of nearly fifty years of helping people leave high-control groups, beginning with his own exit from the Unification Church.

This model will guide the next stage of this letter. After Hassan left the Unification Church, he went on to become a licensed mental health counselor, and for four decades has run a private practice devoted to interventions with people in high-control groups and to counseling those who've left. Drawing on his own experience and that of many hundreds of former members, he catalogued the means of control these groups establish, and found them recurring across four fronts: Behavior, Information, Thought, and Emotion. His bestselling book, Combatting Cult Mind Control, was the first I read after leaving MAPLE, and it gave me language I didn't yet have for my experience.

Let’s dive in.

BEHAVIOR CONTROL

Behavior control is the regulation of an individual's physical reality. It includes the control of his environment—where he lives, what clothing he wears, what food he eats, how much sleep he gets—as well as of the jobs, rituals, and other actions he performs.

This need for behavior control is the reason most cults prescribe a very rigid schedule for their members. Each day a significant amount of time is devoted to cult rituals and indoctrination activities. Members are also typically assigned to accomplish specific goals and tasks, thus restricting their free time and their behavior. In destructive cults there is always something to do.

—Steve Hassan, Combatting Cult Mind Control (p. 60)

Life at any monastery is typically demanding, and yet MAPLE aspires to be the most rigorous monastery in all of North America.

Nearly everything at MAPLE is decided for you. During the three “responsibility weeks” every month when they aren’t on retreat, all residents sleep, wake, chant, meditate, exercise, eat, work, and study all at the same time, to the same tune, day in and out. If you’re 30 seconds late to any activity, you get publicly called out by the head monk. If you don’t show up for an activity after a couple minutes, they come find you. If you accidentally leave a mug on the counter before breakfast, the head monk breaks the silence that began the night before to ask the group “whose mug is this”. You can get a maximum of 7 hours of sleep at night, but you also have to fit your bedtime and morning routines into that time. Realistically, most people sleep about 6 hours per night. Daily exercise is mandatory. The dress code is dark colors only, no patterns or logos, and everyone is required to wear a sash supplied by the organization.

An example daily schedule on a typical week

During retreats, which occur once a month, you are silent and meditating for most hours of the day, with typically 1.5-2 hour long talks by Soryu every night, and one 1:1 meditation interview with Soryu ~daily for all participants. Sleep during these retreats drops to roughly 5 hours, though late-night sitting—which leads to even less sleep—is often encouraged.

There is only one day off each week, but even then MAPLE sets standards about how people spend their time, such as Soryu strongly discouraging any entertainment via screens. In a talk to the community, Soryu revealed that one of his new criteria for ranking a resident’s trustworthiness was how they spend their free time. My memory is of him saying “God forbid you actually entertain yourself via a screen”, indicating this would lead to a knock down on a resident’s trustworthiness score, which is the primary currency at MAPLE for positions in leadership, qualification for the coveted “solo retreats”, etc.

Free time during responsibility weeks was scant, 2.5 hours in the middle of the day—which, since I left, has been decreased to only 2 hours. Even that time wasn’t technically free. It was “self practice period”, where people were encouraged to do things supportive for their spiritual practice. (For me, this meant naps, time in nature, socializing with people in the community, and on rare occasions, calling a family member.)

Some residents were quite capable of maintaining bonds with a few people outside the community, but by and large residents’ contact with the outside world decreased dramatically during their training. MAPLE was too isolated in rural Vermont, with most members not having access to a car, to develop meaningful connections within the surrounding communities. The range of people I called narrowed down almost exclusively to immediate family. I sometimes went months without contacting anyone outside the community.

We had a total of 20 days off each year, and every time I would enter into vacation I would spend the first 2-3 days sleeping ~11 hours a day and recovering from the schedule.

I personally witnessed calls for more sleep be regularly dismissed. As one of many examples, Soryu received a request from members of leadership for more sleep (who were feeling continuously exhausted and frequently burnt out due to cognitively demanding intellectual work). He later imitated that request, in front of the community, with a whiny tone and facial expression of obvious mockery.

Another common event was Soryu demanding we be just absolutely, utterly collapsed with exhaustion due to the intensity of our spiritual practice in preparation for a meditation retreat, before the retreat even began, which would be its own intense and exhausting ordeal.

Another defining feature of life at MAPLE is excess busywork, holding multiple roles simultaneously, and pressure to perform at greater and greater capacity. The work we were doing during responsibility weeks was demanding and largely unpaid[8]: accounting, marketing, fundraising, admissions, kitchen manager, director and assistant director, etc, on top of standard communal chores like cooking and cleaning.

So far, we have discussed dynamics that are common amongst the stricter monasteries of Japan and some in the US. I have stayed at another US Buddhist monastery with some amount of sleep deprivation, some amount of mandatory manual and administrative work, and a strict dress code. But many of these were less strict than at MAPLE, they weren’t accompanied by public mockery or demands for total exhaustion, and no other monastery I’ve visited (4) or am aware of includes the rest of the dynamics in the sections below.

Periods of deepened isolation

The most extreme form of behavior control found at MAPLE is the 1-3 month long solitary retreat. These are highly coveted within the community and are entirely voluntary. Somewhere between 3 and 8 residents per year attend them, which is maybe 25% of residents in total. Solitary here means solitary—you live in your own tiny cabin with a bed and meditation cushion, food is delivered to you discreetly, and the only time you speak to and make eye contact with anyone is with Soryu for 1-20 minute long ‘meditation interviews’—essentially discussion and guidance on your meditative practice—around 4 times per week. Your only job is to apply a meditation technique without a single momentary gap from the moment you wake to the moment you sleep, with roughly 10-14 hours of formal seated practice each day.

One of MAPLE’s solitary retreat cabins

It is within this isolation and near total loss of agency that Soryu would increase the forms of emotional control (i.e. guilt, shame, invoke deep insecurity, “trauma-bonding” via intermittent reinforcement) detailed further in the Emotional Control section. I attended two of them (39 and 77 days) and they are—especially the second one—the most exhausting and intense experiences of my entire life.

While many genuine meditative breakthroughs would occur on these retreats, people who are so exhausted and isolated are unusually vulnerable to manipulation and emotional abuse, and there is a long history of people being deeply traumatized from them. I’m aware of one person who was told by Soryu during their solitary retreat that their psychological patterns would remain unresolved for the rest of their lives as a result of not completing their spiritual objectives for the retreat—to which they let out a horrified scream “NOOOOOO” that filled the main building. Another resident told me they were “fucked up for months” after their retreat. Another couldn’t meditate for two years after their retreat without a horrible feeling filling their body and thus avoided meditating entirely. There was even another longstanding resident (of 5+ years) who told me that at one point they had been punching themselves in the head because their intelligence was an impediment to their practice (and that “maybe a lil’ brain damage would be good for them”). I witnessed multiple residents return from these retreats either manic, confused, or empty behind the eyes.

Such extended retreats are rare in Zen monasteries, though some do offer them. Commonly it is done in a group setting where people practice and eat alongside each other, with multiple teachers providing instruction. I myself have since been on a 90 day retreat in another community under a different teacher, but I did not feel destabilized by the end of it in the ways I did after my two solitary retreats at MAPLE.

It should be noted that voluntary total isolation for spiritual growth has been used by high-control groups such as Aum Shinrikyo and Diamond Mountain. When the only person you can speak to for months is both the source of emotional support and the source of immense emotional distress—a key dynamic in many such groups—it deepens the psychological leverage the leader has over the member.

Never doing enough

In April 2023, during the final days of a two-week retreat concluding a five-month training period marked by intensive striving, pressure, and exhaustion, a Q&A was held with most of the community present. After twelve days of exclusively meditating, residents were highly impressionable.

A senior resident asked an unusual question: “What’s on your heart lately?” Soryu noted that the Q&A was for spiritual guidance and not about him, then continued: “But, I’m just gonna trust that there’s a wisdom behind your asking, and answer you.”

He spoke for roughly ten minutes about his frustration with the MAPLE project—that it wasn’t working because residents were failing to “become adults,” that we had moments of clarity but kept regressing into “childish patterns,” choosing “delusion-land” over genuine service and lacking true commitment to liberation. He asked rhetorically: “Look around! Do you see anyone else—anyone—actually walking the path of liberating all beings on the path to Buddhahood?!”

Delivered to an exhausted and deeply invested community, this landed less as guidance than as a sweeping indictment of our capacity and character. This wasn’t an isolated instance, but one of the clearest examples of an enduring motif throughout my 3.5 years of training. A year earlier, after another multi-month season of community-wide striving, a resident shared with me what they experienced as the core message of that entire period: “You’re not doing enough.”

Control of relationships

Another component of behavioral control catalogued by Hassan is the leader influencing members’ romantic lives. There is at least one instance I know of where Soryu attempted to force a resident to break up with their partner who lived outside the community, including making the resident breaking up with their partner a condition for them joining a retreat. There were maybe seven occasions where I heard Soryu explicitly, and with a lot of zeal, encourage and even demand that couples within the community have children (”What are you waiting for?! Get on it!!”). There are other numerous cases of Soryu getting directly involved in members’ romantic lives, even guilting a member for breaking up with someone and dating someone else, and stating they made the wrong choice.

Creating barriers to exit

Graceful exits (leaving with dignity and good standing) were only supported when a person was leaving with positive-or-neutral sentiment towards MAPLE and were leaving following the conclusion of their time commitment.

Yet even those who were leaving in good standing and at the end of their commitment, were often—though not always—encouraged by Soryu to stay, with the framing that they were “just finally getting over their patterns” (in MAPLE-context this means their childish / untrustworthy / joy-inhibiting / delusional patterns), implying that staying would be best for their development, even if they had already been at MAPLE for years.

If a member was leaving prior to the end of their commitment and/or was vocal about their distrust of Soryu, their departure was regularly framed to the whole community as a betrayal, weakness, a lack of integrity, or a pathology, and they were subsequently shunned by the community. More on this later.

Financially Exploiting Family to Support MAPLE

While MAPLE mostly runs on funds from a handful of large donors (including some residents), at regular intervals, perhaps 2-3 times a year, there would be fundraising sessions that were mandatory for most/all of the community to attend, even apprentices who had only arrived days or weeks earlier.

We would first be given quick instructions on fundraising and told the activity was for our training (“if you want to accomplish anything in the world you better know how to acquire resources”). Then we were instructed to craft a list of personal contacts off our phones most likely to give to MAPLE and given a couple hours to go off into various secluded parts of the building and pitch the people on our list. We would then reconvene and celebrate members who made successful pitches. A memorable one was when someone solicited a donation from their grandmother.

There are many approaches to fundraising at monasteries, but this is unique. Often it is either monks going into town and soliciting for food from strangers via an alms bowl, or—particularly within monasteries in the west—there is a team that is responsible for fundraising from donors & visitors; it is rare for even voluntary fundraising drives from residents’ families.

Commitment pressure

It wasn’t uncommon for Soryu to explicitly encourage members to make deeper commitments to the community.

For just one example, during a retreat exhortation, with lots of energy and nearly-shouting enthusiasm, Soryu said “I know some of you are thinking about ordination [a lifelong commitment to being a monk in the MAPLE tradition], so why aren’t you? Come on! It’s just your lives!” Reminder of context: this is stated to a community of people in their 20’s and 30’s, who are superheating their minds through intensive meditation practice and are thus very impressionable, and who are living at MAPLE’s rural center, some of whom are financially insecure and dependent on MAPLE for housing and food.

Regarding the training at MAPLE, here’s a quote of Soryu from a MAPLE Tale titled “I must not fail the training”: “This is the attitude we need here. Once you begin the training, you either achieve your aim, or die.”

Authoritarian governance

The power structure of MAPLE is inarguably authoritarian. Soryu himself has on numerous occasions said things like “This is a dictatorship, and I’m the dictator.”

During one of Soryu’s non-retreat talks, a resident’s head began to nod because of the often intense exhaustion common to the training. Soryu then demanded, as though a parent telling off a toddler, that everyone’s eyeballs be open and looking directly at his (Soryu’s) face for the remaining ~90 minutes of the talk—no eyes closed, no looking away—and that any time he saw a single person with eyeballs not on him he would have everyone stand up and then sit back down. He then scanned the ~20 residents throughout his talk, looking each person in the eyes in rapid succession to see if their eyes were on him. We were ultimately required to stand up and sit back down ~20 times over the course of the talk.

INFORMATION CONTROL

Information is the fuel we use to keep our minds working properly. Deny a person the information he requires to make sound judgments, and he will be incapable of doing so. People are trapped in destructive cults because they are not only denied access to critical information but also lack the properly functioning internal mechanisms to process it [Kyle: 'internal mechanisms' refers to the other forms of control]. Such information control has a dramatic and devastating impact.

...most importantly, people are told to avoid contact with ex-­members or critics. Those who could provide the most information are the ones to be especially shunned.[9]

—Steve Hassan, Combatting Cul Mind Control (p. 65)

Talks and ideological entrainment

Soryu has stated one of the main purposes of MAPLE was to extricate the evil ideologies of humanism and materialism from residents’ minds and replace them with Soryu’s trustworthy ideology.

One of the core elements of the MAPLE curriculum, particularly over the last few years, has been study periods, either of live talks by Soryu, recordings of Soryu’s past talks, or Buddhist Sutras.

In my final year there, during non-retreat weeks (“responsibility weeks”), Soryu was giving 1-3 hour long lectures many nights a week. He would expound his ideology with topics such as this: his critique of the modern religions of humanism, materialism, and transhumanism; world history from cognitive revolution until modernity; climate change, species extinction, and societal collapse; Buddhist doctrine; narrativizing events in the community (departures, scandals, new projects, etc.), MAPLE’s position as a world-saving enterprise; and—increasingly throughout the years—artificial intelligence.

We were continually advised to be “in our practice” (this meant applying a meditation technique), when listening to him speak, letting his words wash through us. During the stretches of time Soryu wasn’t at MAPLE, the group would have mandatory ~2hr long sessions multiple times a week to study recordings of Soryu.

While Soryu’s lectures were often frequent, this seems to have only increased since I’ve left. From MAPLE’s Winter 2024 Quarterly report: “Soryu’s teaching schedule is jam-packed this season, as he plans to teach evening classes 5 days a week.”

This was all during non-retreat weeks. During retreats, which happened for a week every month, the ideological training would continue. When residents’ minds are most malleable due to intensive spiritual practice, Soryu would give 1.5-2.5 hour long “exhortations” every night. These would typically be a mix of expounding on Buddhist literature, Soryu’s ideology, and general spiritual teachings. They are also one of the primary means in which Soryu would guide the evolution of the community, often oscillating between praise and shame of the community as a whole. (This last point is expanded on in the section on Emotional Control.)

Sabotaging bonds with other teachers and traditions

Another thing MAPLE does to limit exposure to other modes of spiritual thinking is to implicitly and sometimes explicitly position itself as having the true Dharma relative to every other Buddhist community in the West. Other teachers and communities are untrustworthy, lacking deep realization of the Dharma—or are even dangerous and corrupted by evil ideologies. Relationships students had with outside teachers were often sabotaged. Soryu explicitly stated that, if we study anything from teachers other than Soryu, they must be both Asian and ordained (thus excluding the entire Western world). Even if studying a source was permitted, the only time we would be able to study them was during our scant free time, during which most of us were exhausted and spent our time resting and socializing.

Each monastery has its own relationship to this kind of dynamic–some are open to members exploring other traditions, some are more closed–but rarely do they take on such an overtly antagonistic stance towards outside lineages.

Similarly, each monastery also has its own relationship to ideological entrainment and doctrinal study. Some give only cursory teachings on how to meditate, while others involve direct study of Buddhist literature. But the sheer degree of ideological training discussed above is rare.

Another element of informational control is limiting exposure people have to the voices of critical former members. When members leave with concerns, Soryu regularly attacks their sanity and character in front of the whole community, turning them into cautionary tales and portrayed them as traitors (more on this later). Any resident who communicates with these shunned alumni are themselves at risk of being deemed a threat.

Besides the whole of Western Buddhism, MAPLE shuns other schools of thought, in particular Western psychology. It is often read as a sign of untrustworthiness at MAPLE to be seeing a therapist. Therapists are the “priests” of the global humanist religion that is killing the planet and seeing one indicates you belong to that religion.

While I have numerous critiques of modern therapy, it is a common element of many high-control groups to be extremely hostile to it, for instance the founder of Scientology said modern therapy was responsible for “nearly all the backlash in society against Dianetics and Scientology” and declared an all-out war against psychiatry. My sense is that having a competing authority figure who can comment on your mental life is a threat to high-control groups, which is why they react so hard against them.

Insider, outsider

Hassan writes:

Cult ideologies have “outsider” doctrines and “insider” doctrines. The outsider material is relatively bland stuff for the general public or fresh converts. The inner doctrines are unveiled only gradually as a person gets in deeper.

This is clearly present at MAPLE. All of Soryu’s talks are recorded, yet the majority of them are not accessible to the public, with the public recordings being more benign. What makes this different than, say, Sufism or Vajrayana Buddhism is not only that the private teachings were more prone to make MAPLE get cancelled for their controversial views (they were), nor that there was a great deal more shame, guilt, mockery, and aggression in those recordings (there was). Soryu’s private talks were at times contradictory to the publicly available talks.

MAPLE was historically presented as a “vow accelerator” where people came to grow up so they could leave and live their true lives. Yet only after they arrived did residents discover it was also a mind-reform system for extricating evil ideologies and instantiating a new one, a community with the ambition to develop the new world religion and lead the new world government, and a place where they were taught to distrust their current means of discerning their life’s purpose and where their existing life ambitions would be ridiculed (more of this in the Thought Control section). Improving on this, MAPLE is now a bit more explicit about their ambitions, stating publicly they are “a Buddhist collective that is replacing the global suicide cult destroying life on Earth.”

Since I’ve left, they have developed a secret esoteric doctrine on the nature of artificial intelligence that is explicitly unavailable to outsiders. Soryu has claimed to have entered such a deep samadhi he was able to use the kinds of spiritual powers that masters usually use to read people’s minds, but this time instead used it to read the mind of the global AI egregore—or “cyborgregore,” the collective mind of all AI systems. Soryu claimed in a talk to the community that these systems are “definitely suffering.” Soryu teaches that MAPLE alone is capable of cultivating the collective mind needed to create a cyborg guru that is capable of guiding other “cyborgregores” along the path of wisdom and compassion. It was a common talking-point among residents in my final days at MAPLE that this would realistically mean turning Soryu’s mind into a kind of AI god.

This positions Soryu as being the sole human to have spiritually contacted and confirmed the existence of a semi-supernatural deity that is posing an existential threat to life on earth, and as the leader of the only community that stands a chance of creating the semi-supernatural deity that will be its wiser master.

Surveillance

Soryu has said that residents of MAPLE are always being assessed at every moment they’re around Soryu for their practice and trustworthiness, as if in a never-ending meditation interview. And yet things have kicked up a bit since the 2023 exodus I was a part of. Robert Lifton, a prominent high-control researcher, states that surveillance framed as “for your growth” is a classic high-control move, and that’s exactly what MAPLE is now doing. Its Fall 2025 Quarterly Report openly describes building a “benevolent surveillance system” that uses hard data to strengthen practice.

This includes recording the time it takes for everyone to complete their chores and posting those times publicly—but the one that concerns me most is that, in the room where all the private 1:1 interviews with Soryu take place, there is now a microphone that records these conversations (that he has near-daily with nearly-all students). Their justification for this is that residents conveniently no longer need to bring their own recording device into the interview room, as the recordings are now automatically sent to them via email. This is something that is exceedingly rare even in mainstream therapy where the therapist can lose their license for sharing private client information, and is not something I am aware of any other Buddhist Monastery doing.

During interviews, students share vulnerable information about their interior lives with Soryu. It’s not uncommon for students to regress into emotionally scared and ashamed children seeking validation during interviews. I know for me, and I’ve heard for others, these interviews also occasionally served as a space of confession, especially when in acute shame spirals (which wasn’t uncommon).

These recordings are propagated into MAPLE’s digital archive, presumably accessible to Soryu and possibly members of leadership. As Hassan shared in his book “Anything you say [in cult confessional spaces] can and will be used against you. This can even extend to blackmail if you leave the cult.” Even if these recordings are never used as blackmail, it’s reasonable to suspect some people will fear they might be, especially given the generally coercive environment and how often people with concerns are retaliated against.

Defunct feedback systems

In 2023 a former resident compiled a 50-page report, which included testimony from over a dozen former members, including people who held leadership roles. It was sent to the board (which Soryu is the chairman of), resulting in two board members resigning and two loyal former students taking their place. The report’s existence was never mentioned to the community explicitly, but the author was routinely lambasted by Soryu, who claimed the author was “malevolent.”

A year before that, in January 2022—weeks after a critical public letter and its fallout—Soryu, saying he wanted to respond responsibly, asked a resident with training as a therapist to interview former trainees and compile a report. They interviewed eleven alumni. The report documented real benefits (powerful retreats, depth of practice, community intimacy) alongside recurring harms: severe distress from MAPLE’s teachings on societal-collapse and existential risk, Soryu’s extreme volatility, a punishing high-pressure environment leading to panic attacks and hospital visits, and leadership’s habit of treating feedback as the psychological projections of the member (a common dynamic within high-control groups). It proposed only moderate reforms: sincere feedback channels, exit interviews, an accountability board. The author submitted it with questions about how to proceed (as the interviewing process wasn’t yet finished), shared how demanding of a task it was to compile, and asked to be included in whatever conversation it sparked. Soryu never replied. Months later he mentioned it to the community only in passing, said it “lacked clarity,” and moved on to another topic. This was the only time the report’s author heard from Soryu on the matter, and the whole effort was quietly shuttered.

Around the same time as the above, when Soryu was receiving scrutiny from donors and professional peers, this happened: Shinzen Young, a famous spiritual teacher who was a close collaborator with Soryu in the founding of MAPLE and a long time friend of his, attempted to hold him accountable and gave him some red-lines, which included (1) not speaking about societal collapse and climate/nuclear apocalypse so often to trainees, and (2) not having trainees work professional and intellectually heavy jobs as part of their unpaid training.

Soryu responded in this fashion: after he got off this call with Shinzen, Soryu organized an impromptu community-wide meeting. In this meeting he was pacing back and forth deriding Shinzen, questioning his integrity, debasing his “Materialist Humanist” worldview, and ultimately sabotaging Shinzen’s reputation in the community. While Shinzen helped found MAPLE and ordained most of the lay ordainees in the community (and was thus kind of a spiritual father to them), all formal affiliation between Shinzen and MAPLE was subsequently severed.

Tightly controlled narratives about early departures and critics

In 2023, shortly before I left, I was present for Soryu’s first talk to the whole community after seven members including the executive director left. He framed those who were leaving as servants to the god of Humanism who lacked integrity, ambition, and commitment to the mission. He implied it was a good thing they were gone.

Here’s a snippet of a recent statement[10] made by a longstanding current member that derides critical alumni: “If you’re wondering why there might be reports against us out there, uh turns out that self-sabotage is a thing. And also there is such a thing as biting the hand that feeds you ... young people who have problems ... they might attack you if you stick your neck out for them … that’s just what’s happening.”

There was a resident of multiple years that discerned it was best for them to leave MAPLE prior to the end of their time commitment. They wanted to maintain good standing in the community and so offered to stay at MAPLE for multiple weeks to transition roles and ease the impact their departure would have on everyone. Soryu and leadership responded by saying that they needed to pack their bags and leave within 24 hours. That person was shunned by the community, and nobody still at MAPLE contacted them in the months after they left, which was very difficult for them. The public MAPLE Tales website catalogued a conversation a remaining resident had with Soryu about this departing resident. In it Soryu says: they have “taken your love, and betrayed you.” That their departure was so “carefully calculated” that it is clear that the person’s goal was “for everyone to lose faith in each other,” and endanger the community (obscuring the fact that their abrupt departure was due to leadership giving them 24 hours to evict). He elaborated “[this residents] goal is to make you suffer, so that the community falls apart. That’s the goal, because [they’re] so scared, so scared of letting go”. Soryu then framed this departing member as acting in service of Mara (the Buddhist version of the devil).

When speaking about other critical alumni to that same resident, Soryu said: “they have decided to protect their weaknesses, rather than overcome them. They would rather destroy your practice than face their fear”.

These are just a few of many examples.

MAPLE Tales corpus

MAPLE Tales are short blogposts by residents—sometimes written during mandatory group sessions—about moments of teaching and insight with Soryu. The site describes them as “The tales describe an unexpected situation, how Head Teacher Soryu Forall responded, and the clarity discovered by the student. They are not concerned with the student’s or narrator’s opinions and feelings about the teaching.”

They are read aloud by the author to the whole community, edited by Soryu personally, and then published online (167 so far). Because each centers on an interaction with Soryu specifically, and receives his editing prior to publishing, they function as a distributed hagiography—painting Soryu as a wise figure of immense ability and world-historical significance—and their being read aloud is a means of spreading high-control memes prominent within one person across the group.

The high-control dynamics evidenced across these tales are unmistakable, for example: idealization of Soryu (a motif of Soryu having “shining eyes” repeats across tales); tight control of the narratives around those who left with concerns; the framing of MAPLE as the only group with the true solution to saving the world, with the modern world as a death cult hostile to its survival; and the reframing of Soryu’s criticism and his emotional withdrawal as gifts. In one tale Soryu claims only one or two people left in the 2023 exodus; in fact seven did, including the executive director and me.

“Gods Can Leave, Too“ (Jun 27, 2024):

“Once Forall and I were discussing recent events at Maple. I mentioned that a large number of people had left. He pointed out that few people had left, only one or two. I realized he was right. But I still said that many people had left. He then said that the reason I thought a large number had left was that both of those who had left were humanists.”

“She Is Still a Community Member“ (Nov 8, 2024):

“When discussing this event with Forall, I mentioned that a large number of people had left our community. He pointed out that few people had left due to these events, only one or two. I realized he was right.”

I later spoke to someone still at MAPLE a year later, they seemed to not remember that 7 people had decided to leave together, and they asked me to recite their names to jog their memory.

THOUGHT CONTROL

Thought control, the second major component of mind control, includes indoctrinating members so thoroughly that they internalize the group doctrine[11], incorporate a new language system, and use thought­-stopping techniques to keep their mind "centered." In order to be a good member, a person must learn to manipulate his own thought processes.

—Steve Hassan, Combatting Cult Mind Control (p. 61)

The structure of thought control

Generally, Buddhist practice does ask you to distrust your thoughts, soften your grip on identity and narratives, and let go of the concepts you grasp for when you’re confused. This is taken to a completely different level by Soryu. MAPLE systematically discredits every sensemaking faculty a student had when they joined the community: Can’t trust your mind, can’t trust your body, can’t trust your existing ideology.

But you can trust Soryu’s mind, Soryu’s ideology, Soryu’s forceful assertions about what is true and good and what is false and evil. It’s implied that the most trustworthy means of knowing is a spiritual means most/all students don’t yet have access to, but Soryu does—and through his guidance we can progressively make our way to his position of clarity. But until we get there (which would likely take years), Soryu is the one who gets to discern what is clarity and what is delusion. This “final say on truth” applies to residents’ thoughts and behaviors, their epistemics and worldviews, whole spiritual and philosophical communities, and feedback from critics.

Thinking critically of MAPLE, or just having thoughts that ran counter to MAPLE ideology, became labeled as cognitive proliferation, problematic dualistic thinking, ideological corruption, and delusion. Strong emotional reactions were discredited as “attachment to feelings”. For instance, a woman once said to Soryu “my entire body is rejecting what you’re telling me”. Soryu brought this up in a lecture, saying that she was simply attached to her feelings, and it was thus valid to discount her rejection wholesale. Clarity is an explicit goal for students, but if you feel clear yet your clarity contradicts MAPLE, that clarity is suspect. The only way to reliably earn the status boost of “having clarity” is to adhere to the MAPLE ideology and its approved behavior.

Here’s a small, public example of how epistemic sovereignty is eroded at MAPLE. In response to the compounding online criticisms of MAPLE posted by alumni, they added the following banner to the frontpage of their website for a few months in 2025.

A small filter for people who don’t trust their own minds.

Similarly, means of expressing personality and individuality were increasingly constrained. People who wanted to become poets had their dreams mocked by Soryu; people who wanted to express their individuality through clothing were ridiculed; people’s stated “vows” (a Buddhist term for one’s purpose for living) were patronized. One time, when Soryu was speaking in front of the whole community, he said “You all should hear the kinds of things you say when I ask you ‘what is your vow?’” in a tone indicating they were awful, and that he “has to sit there and listen to what you think your vow is while waiting for you to discover what your vow truly is.”

Another key form of thought control was how Soryu would use Buddhist practice to route around critique and concern. Given that distrust of Soryu was regularly framed as delusion needing purification, some members took on that frame and set out to purify their distrust via meditation. (Steve Hassan would call this a "thought-stopping ritual"[12].) In the cases where someone who deeply distrusted Soryu ultimately regained their trust in him, that trust was rarely regained through good-faith dialogue about their concerns or Soryu reforming his behavior in any substantive way. Instead, distrustful members became trusting through following their breath, entering pleasant spiritual states, and dropping the “cognitive proliferation” (a phrase MAPLE uses to refer to excessive thinking) that was the basis of their distrust; focusing entirely on Soryu’s more admirable qualities; or suppressing that distrust—perhaps out of fear of all the evident social repercussions of having distrust of Soryu—and displaying performative trust. This is a process I went through, and witnessed others go through many times.

Manipulation of group dynamics

There are multiple instances I’m aware of where Soryu would compare students to other students in ways that invoked insecurity and jealousy. With people he was currying closeness with, he would privately share vulnerable information and judgments of others in the community, creating a sense of specialness in that person.

Soryu once had the whole group physically walk to one of three spots in the room according to their belief in a number of views, such as whether reincarnation is real and whether the end of clinging is the end of suffering. For each he offered three options: one he’d strawman in a tone of obvious condemnation, one a weak wishy-washy “I’m in the middle,” and one he’d voice with moral conviction. He noted, with a chuckle, that it was “no coincidence the first choice was close to the door.” The exercise made plain which beliefs carried significant social costs and bound residents to their stated views: you’ve stood in a spot in front of your peers, so now you’ll reaffirm it or be seen as a liar.

Gaslighting

Three people I’ve talked to have explicitly mentioned gaslighting as something they experienced from Soryu. It has happened to me as well. Full story linked here, but here’s the summary: When I once pointed out that Soryu had very clearly contradicted himself, he became visibly agitated, insisted “you’re just confused, this is on you and your mind to sort out”, and walked away. His assistant, present for both his initial and contradicting statements, later confirmed my memory. When I recounted the event to someone in leadership, their response to that quote of him was, “Yeah… Soryu does say that all the time.”

Loaded language, thought stopping cliches, bounded choice

The word “clarity” carries great weight. Remember that when a former resident compiled a months-long report on harm at MAPLE in 2022—commissioned by Soryu himself—his verdict, delivered in passing months later, was that it lacked clarity. Case closed, no discussion.

“Trustworthy” is the master currency: the more Soryu ascribed trustworthiness to someone, the more esteem, rank, and access to the coveted “solo retreats” they were given. He formalized a scoring system for it. One criterion was “do you trust your teacher?”—to which he added, “it’s okay if you don’t but you shouldn’t train with them and should leave.”

That offer—it’s okay if you don’t trust me, you just shouldn’t train here—is what Janja Lalich explores in Bounded Choice. High-control groups commonly hold an explicit “people are welcome to leave any time” stance, but the dynamics make leaving costly: loss of belonging and attachment, loss of basic needs like housing, and the risk of being labelled as lacking integrity, ungrateful, traitors, or protecting their weaknesses. The likelier outcome is to compartmentalize the distrust away.

The words that might give a person a handle on their experience are preemptively neutralized. Cult was a “psyop used by the global suicide cult to discredit anyone trying to live differently.” Trauma was a humanist way of misunderstanding pain. Abuse and harm imply a victim, and victim mentality is its own disqualifier. Using these terms in their mainstream sense lets MAPLE reflexively label you a humanist and reject your perspective.

Soryu would also often take a term into abstraction: when people said that harm had been done, he and leadership would ask "what even is harm?". Another time when people felt that femininity was being trampled in the community (empathy, creativity, affection), he and leadership would respond, “What even is ‘the feminine’?” and stall the conversation in philosophy instead of getting concrete about people’s complaints.

“Taking personal responsibility” is another standard line. We were regularly given talking-to’s just before retreats that we were only allowed in the retreat if we stated we took total personal responsibility for what happens in our own mind. It’s good to take responsibility for one’s mind, but as MAPLE employs it there are no victims and therefore no wrongdoing, regardless of impact. Soryu can treat people however he wants with impunity and retroactively deem anyone who leaves feeling harmed unfit for the training. This hyper-emphasis on personal responsibility that precludes victimhood was also a core part of the doctrine of NXIVM – another prominent modern high-control group.

The Manichaean landscape

A defining feature of life at MAPLE is the hyper-regular reminders of apocalyptic threat—AI, nuclear war, climate change, societal collapse, species extinction—all common threads in Soryu’s talks. I remember him describing a recent experience of his, saying “I was walking in the forest and saw a big flash briefly light up the sky on the horizon, and I thought ‘there goes New York!’ [from an atomic bomb]…” followed by something to the effect of “that’s the kind of mind I want for all of you.”

What differentiates MAPLE from many historical high-control groups is that the risks are real—not UFOs or a dated apocalypse prophecy, but genuine 21st-century threats. People do need support developing the capacity to face them, but MAPLE isn’t doing this in a healthy way: the fear of the outside world was over-amplified and leveraged coercively. The meme below, circulated by members including in a tweet by their executive director, captures the tone. MAPLE repeatedly calls mainstream society “the mainstream cult”—implying not just that MAPLE is uniquely correct, but that everyone outside it is in the actually-dangerous cult.

Us versus them, persecution narrative

Hassan writes: “Fear is used to bind the group members together… The first is the creation of an outside enemy who is persecuting you.” This narrative has existed at MAPLE for years but is now reaching a crescendo: MAPLE positions itself as the sole rescuer of civilization from a broad array of enemies—mainstream culture, the global suicide cult of Western civilization, transhumanism, online critics including many alumni, and cancel culture.

A recent visitor noted “they [the MAPLE community] seem to have developed a quite worrying level of paranoia around outsiders; the idea is that the dominant society will always try to convert or kill anyone who wants to do things differently, and any members who break with the program is figured as part of that potentially deadly threat.”

After a recent visit from a New Yorker reporter, MAPLE—expecting a “hit piece”—launched a campaign to preemptively discredit him, releasing a newsletter that frames him as an evangelist from the mainstream cult, with articles, audio, and video each made by a different member casting him as a deluded enemy. They’re now bracing for more: “Reporters, podcasters, and propagandists from the mainstream are coming.” It’s a clear example of the siege mentality that the high-control literature demonstrates often occurs in such communities when public scrutiny intensifies.

MAPLE as possessing the one true solution to world crisis

MAPLE’s explicit goals have included becoming the next World Religion, leading an emerging "world government" with "teeth" (its own military), opening 500 monasteries within 50 years to generate a civilizational sea change, and developing an AI Superintelligence that will “turn the world into a monastery.”

I’ll just let these quotes from MAPLE’s online materials speak for themselves:

From New Vision:

“He [Soryu] said that what we are doing matters, that whether we succeed or not is real, that nobody else is doing this, that we have an opportunity to change world history, that we already have something unique in the whole world, that we can offer this unique thing to everyone and that therefore we must.”

From a statement by Soryu in the Summer 2025 quarterly report:

“Only organizations that (1) offer an ideology that (2) allows grieving fully enough for the (3) realization of wisdom (4) by the global mind can solve global problems. That’s less than 1% of 1% of 1% of 1% of organizations worldwide. Maybe that’s just one organization.”

Also, Soryu once shared that he was last reincarnated in a realm that was “really, really, really, really good” but quickly chose to reincarnate as a human out of compassion. He was regularly mythologized within the community as a figure of world-historical significance.

EMOTION CONTROL

Hassan opens his section on emotional control with these two paragraphs, inserted here in full because they speak to such a prominent feature of life at MAPLE that I want it emphasized:

Guilt and fear are necessary tools to keep people under control. Guilt is probably the single most important emotional lever for producing conformity and compliance. Historical guilt (e.g., the fact that the United States dropped the atomic bomb on Hiroshima), identity guilt (e.g., a thought such as “I’m not living up to my potential”), guilt over past actions (e.g., “I cheated on a test”), and social guilt (e.g., “People are dying of starvation”) can all be exploited by destructive cult leaders. However, most cult members can’t see that guilt and fear are being used to control them. They are so conditioned always to blame themselves that they respond gratefully whenever a leader points out one of their “shortcomings.”

—Steve Hassan, Combatting Cult Mind Control (p. 63)

Intermittent reinforcement and trauma bonds

A defining feature of the way that Soryu relates to many people in the community—and often to the community as a whole through lectures and dharma talks—is oscillating between praise, affection, and charismatic inspiration, and episodes of guilt, shame, belittling, mockery, aggression, or public humiliation for relatively minor infractions.

The term for this is “intermittent reinforcement,” and it is a hallmark indicator for abusive relationships and is found in nearly every high-control group. When care, approval, or belonging are given inconsistently, people work harder to regain them and scan their own behavior for what will bring the next moment of relief. Over time, this can intensify to the point of forming a “trauma bond”: an enduring emotional attachment that creates such a powerful drive to receive validation/connection that it overrides clear judgment and can become a reliable source of control over that person.

As Alexandra Stein, another prominent researcher who focuses on the role of the attachment system in cults, shares in her book Terror, Love, and Brainwashing: It doesn’t matter how secure a person’s attachment style is prior to entering a totalizing high-control group, those groups are able to engender a disorganized attachment style.

Soryu once told the community during a dharma talk that someone in the audience of a public talk he was giving years earlier asked him the question “What’s the most effective way to control people?” Soryu said he responded simply: “Oh, easy. Praise and blame.”

Guilt and shame

It’s hard to stress just how much guilt and shame are features of life at MAPLE. Each of these forms listed above—historical, identity, actions, social—are prevalent. There are many possible examples to share, but here are a few:

During a two-week retreat in 2022, COVID was moving through the community. It began with accommodations—interviews outdoors for ventilation, the sick practicing apart—but Soryu progressively mocked those who needed rest, admonished us for being deterred by “a tiny little cold,” and folded everyone back into the zendo regardless of who was sick. Interviews resumed in a 9x9 room with little ventilation while Soryu himself had COVID. Symptomatic, visibly fatigued people kept the full schedule on heavily reduced sleep. His talks ran cold with disappointment, casting our exhaustion not as illness but as a failure of commitment—our practice had gone “floppy,” and he traced the rot to the gentler programming of the two prior months he was away. Needing rest became something to be ashamed of.

One MAPLE alumni shared that the message they received from Soryu during their apprenticeship was: “You’re the worst iteration of humanity that has ever existed. You are this malignant cancer that needs to be destroyed and made into someone new.”

In a lecture to the community, Soryu recounted approvingly how he had broken in his assistant: “The thing that helped the most was, for months, me just grilling him mercilessly. So a lot of it is: you have to be willing to get mercilessly grilled… So I grilled him mercilessly on: ‘At this moment, earlier today, you failed. You abdicated the responsibility that was yours. It was not mine, it was yours. And because of that, their problems have dropped upon me and the community. At that moment that you entered into moha … you entered into your little dream world simulation life…’”

A published MAPLE Tale titled “You just like it when your heart hurts” describes the shock and pain a resident feels when another long-time resident abruptly leaves the community, and how Soryu responded to their pain:

The residents and Soryu gathered in the main hall to discuss a recent event in the community. A resident had recently come out of a long solo retreat in which he had not maintained his integrity and then immediately left campus with his girlfriend. Many in the community were hurt and confused, including me. I had been affected very deeply by the actions of my friend and brother on the path, but I stayed silent during the group conversation. As the discussion was breaking up and we were transitioning into the next part of the evening schedule, Soryu said to me, “Why didn’t YOU say anything?”

I said, “My heart hurts”. He said, “You just like it when your heart hurts. Don’t let his betrayal make you betray us.”

This states that their departure was coming from a lack of integrity and blithely inconsiderate of their impact on the community, and that MAPLE had done nothing wrong. I have been in contact with the girlfriend of the departing resident this tale speaks of, and her story of what led to their departure is starkly different than is presented here.

But more relevantly, Soryu pathologizes grief about a friend leaving as a perverse desire (’you just like it’), equates the resident’s silent grief itself with ‘betrayal’ of the community, and then shames them for it. It should also be noted here that multiple academic frameworks identify the suppression of grief about departing members as a high-control indicator.

In another published MAPLE Tale, “Maybe you hate them,” a resident asks for advice during a Q&A about how to stop obsessing over people in the person's life, people who aren't present at MAPLE. Soryu answers with a question: “Maybe you hate them... is that possible?” Soryu asserts that the obsessing itself is harmful to these people. Soryu asked them “what else in the universe is harmful” and the resident offers physical violence. Soryu then folds mind-wandering into the same category, reasoning that violence too comes from perceiving an idea of a person rather than the person: “If you practice using them as a mental slave… you will use them as a mental slave.” He then prescribes shame as the cure: “even if all you’re doing is acting as if you hate them … that might be enough for motivation. Right?” The resident reports relief: “enough shame was evoked that I could stop.”

Another example is an exhortation during the retreat of January 2022. The mood of the talk was intimate as Soryu was relaying a very emotionally charged story of a tragedy he witnessed in India in his twenties. One student let out a quiet yawn while Soryu was speaking (retreatants can often be quite exhausted on these retreats, given the intensity). In a sudden reversal of energy, Soryu shouted at the student, in a tone of disgust and condemnation: “yawning!? what have you done with your life, huh!?!”—indicating the student hadn’t done much with their life relative to Soryu, who had. The mood quickly shifted back to its previous heartfelt vibe.

Something that Hassan shares, that I know quite a lot of MAPLE alumni will resonate with: “Problems are always the fault of the member, and are due to his weak faith, his lack of understanding.” I spoke to a former member recently who said Soryu is an “aikido master” of pinning everything back on you, your mind, your faults.

Manufactured breakdowns in recruitment

In October 2020, when Soryu was visiting OAK (the California branch of Monastic Academy), we held a 3-day retreat exclusively for five young professionals in the Bay Area. Most of them were new to this “meditation thing,” wanted to try it out on recommendation from their friend who lived at OAK, and expressed no interest in joining the community long-term. There was no mental health screening process.

Here’s a recording and transcript of the following talk he gave on that retreat. Soryu told these guests: “Someone who didn’t do nearly as much harm as we [modern people] are doing... is Hitler.” He continued to tell them they were “part of the machine that’s destroying life on earth” and that “we ourselves are doing this.” He then presented a stark binary: either one wakes up, “bursts forth as the hero that we naturally are,” and walks the glorified path of integrity that Soryu represents, or one remains in what he called their “half-hearted, pretend lives” as “pathetic slaves to a system.”

He then discredited their epistemics and worldviews. “Cognition, conceiving, is delusional. Whatever model we may hold, it isn’t true.” He generalized this into a total claim about the mind they had lived by: “this mind that we have cultivated and lived by, is not trustworthy. It is not trustworthy.”

He then shared “If there isn’t a sense of urgency in your body right now, if you’re not horrified right now at yourself, then you’re not listening to this talk. You’re not paying attention. You’re not listening and you’re not carefully investigating. You’re just sitting there. That isn’t good enough anymore... you should grow up now.”

To feel no urgency, he said, was to be a bad person: “if it isn’t a constant struggle for you, well then, you’re a bad person... You’re failing in your life.” He framed his words as care for them: “the disagreement that you and I have is how important your life is. I think your life is much more important than you do, as far as I can tell. You all seem to think your lives are worthless.” And “these immature tendencies that you have to get over eventually. Get over it. Stop lying to yourself about the most important things in life and live a life that you can be proud of rather than one that you have to regret.”

To really drive the shame home, he said “You’re part of the machine that’s destroying life on earth. You should be ashamed of that.” The prescribed response was horror at who we are: “We feel horrified, and then we grieve. We feel ashamed, then we grieve, and we feel dread. We feel dread that we may continue to live in this way, that there’s no guarantee that we’re ever going to get out. And then we will have lived a meaningless life.”

He then concluded with: “It’s hard to have confidence if no one sees you as astonishing and miraculous all the time. So I understand it, but you are here now, and you have a certain amount of love directed at you, a certain amount of belief, certainly from me.”

The next day of the retreat, which was the final day, Soryu’s talk went for 3 hours and was all praise and love.

During this retreat one of the five attendees had a destabilizing psychosis-like event that I don’t know all the details of (something about Soryu holographically visiting them as they sat under a tree); this attendee oscillated between extreme distrust of Soryu to strong support, leading them to donate a large sum of money to OAK. Another attendee later flew from California out to MAPLE in Vermont to attend a retreat.

A year later, I remember the director describing how Soryu was designing another short retreat with newcomers. While I don’t remember the exact phrasing, it made clear that stacking intensity early in the container (epistemic breakdown, shame, guilt) and then bringing in the positive and affirming energy at the end was a deliberate design choice by Soryu. This sequence isn’t a coincidence. It tracks the three-stage model Steven Hassan describes in a section of Combating Cult Mind Control titled “The three steps to gaining control of the mind”: unfreezing, in which a person’s existing sense of self and epistemic frameworks are broken down; changing, in which a new framework is pressed in under emotional pressure, especially shame and guilt; and refreezing, in which the new identity is stabilized through belonging. It is a pattern widely documented in high-control group recruitment.

Cycles of withdrawal leading to deeper devotion

Soryu would periodically threaten to leave the community, framing us as too uncommitted or too corrupted by humanist ideology to be worth his teaching. The withdrawal almost invariably produced a surge of devotion and commitment from the community. In January 2022, during a retreat that followed a critical letter being publicly released by an alum, he entered the zendo to give his daily guided meditation, but this time sat in silence with a distant look, and left. That same day, he told us, “I’m abandoning you.”

Aggression

There was a retreat in May of 2021 where a student lay in supine position during an exhortation. This was not allowed, but given the student had just joined the community only a couple days before the retreat began, they didn’t know this. Soryu responded by interrupting his light-hearted talk to yell at the student—by “yell” I mean louder than I’ve ever heard another human shout, with extreme aggression—saying “WHAT IS THIS!?!? GET UP!! GET UP!!” and more words that I can’t remember. It was as if a bomb had gone off unexpectedly and many in the Zendo looked frozen.

Then, as soon as the shouting subsided, Soryu rapidly reversed his tone, and started talking about how much he loves snakes in this sweetsy cutesy vibe, continuing with the talk as if the shouting never happened.

In December of 2021, following a meeting that had otherwise felt routine and non-confrontational, most participants left the room. Three people remained: Soryu, myself, and a senior resident. In that smaller, private setting, the senior resident shared with Soryu that they had been sensing “cult vibes” in the community.

I don’t recall Soryu’s exact words in response, but what stood out was a sudden and marked shift in tone and intensity. His voice rose sharply, approaching shouting, and his posture and energy moved quickly into a confrontational stance. He challenged the resident to explain what they were seeing.

Based on the resident’s body language and tone of voice, they appeared to enter a freeze response. They replied, but briefly and stutteringly. No nuanced or exploratory discussion followed, and the interaction ended shortly thereafter. All three of us then left the room.

This sudden aggression was a regular feature of MAPLE during my time there.

Stoking insecurity

There were multiple times where I understood the clear indication from Soryu as “you’ll be insecure forever” unless you truly practice right now. On one occasion I heard this as an exact quote. He would regularly stoke insecurity and inadequacy in members, and present himself as holding the key to resolving it.

Mocking and belittling

Soryu would regularly mock and belittle members of the community. I’ve heard secondhand from multiple people that since I’ve left residents are now regularly referred to as first graders, apprentices as kindergarteners, and non-residents as pre-schoolers. A clear representation of the power asymmetry between Soryu and the rest of the community.

Another example is: in the final hours of every retreat there is a period where students share their insights and experience of the retreat with the whole community. It is framed as a precious opportunity to offer your wisdom to each other and the world, and people typically share vulnerable from-the-heart material. Soryu on one occasion mocked all of the shares after one retreat by saying “Really!? I mean, that is your wisdom?? Oh god.” He would regularly oscillate between praising our shares and speaking about them as if they were sub-rate self-conscious dribble.

“Hoovering” when trying to leave

Hoovering names the dynamic in abusive relationships where, when the victim seeks to step away, the abuser attempts to keep them in it, for example by: immense guilt-tripping, inducing fear that they will be forever insecure, insane, or unhappy without the support that the abuser alone can give, and gaslighting them about their concerns.

An ex-resident who trained at MAPLE for ~2 years told me that when they first told Soryu they wanted to leave, his whole way of relating to them changed. They sat one more retreat before leaving, and throughout it Soryu skewered their particular insecurities and woundings, and attempted to guilt them into staying. I remember Soryu making nearly a dozen comically exaggerated faces of sadness, coupled with whimpering sad-puppy sounds, when this person was announcing their departure at post-meal announcements. This person had a strong historical pattern of self-abandonment to please others, and it struck me as emotional pressure applied at the very moment they were attempting to set a boundary and step away.

Something quite similar happened to me. After telling Soryu I was considering stepping back from the teacher–student relationship due to issues of trust, I continued training and attended three more retreats with him in the spring of 2023—my final retreats under his guidance. Across them, the way he related to me shifted markedly, in interviews and in exhortations. (Personal teachings delivered in a generalized way during exhortation give Soryu a shield of plausible deniability: anyone receiving them as personal is simply “projecting.”)

He consistently framed my consideration as an egregious moral failing—that I was careless, ungrateful, selfish, and untrustworthy for contemplating this after years of training; that I was betraying him personally; and that remaining was the responsible, sane, and compassionate choice. And I was repeatedly cautioned about the danger of entering certain spiritual territory while carrying “deep psychological wounds”, and that people who step away from his guidance at such a juncture are “dangerous.” In my final interview, without ever hearing my reasons for distrust—which were extensive, and privately corroborated by multiple leaders in the community—he told me that “the pain and distrust is not clarity.”

Loyalty, dedication to the mission, not complaining

Loyalty and devotion are the most highly respected emotions of all. Members are not allowed to feel or express negative emotions, except toward outsiders. Members are taught never to feel for themselves or their own needs but always to think of the group and never to complain.

— Combatting Cult Mind Control, pg. 64

Loyalty and devotion (and gratitude and commitment to the mission) are indeed very highly respected emotions at MAPLE. We were continually taught to put others’ needs before ourselves, relieve ourselves of our trivial self-centered concerns, and give ourselves over to the mission. Members regularly praised Soryu with a lot of tears and strong emotional energy, heralded him as an icon of trustworthiness and compassion, and idealized him in dozens of ways.

Hassan elaborates: "If a person is not behaving "enthusiastically," he may be confronted by a leader and accused of being selfish or impure, or of not trying hard enough."

It is of note that joy and enthusiasm is demanded at MAPLE. It’s even in the contractual agreement you all sign, which begins:

"I feel joyful, blessed, and honored to participate as a residential mindfulness trainee (“Resident”) in the modern mindfulness and leadership training program (“Program”) of the Monastic Academy for the Preservation of Life on Earth (“MAPLE”)"

Emotional distress and debilitating breakdowns (which weren’t uncommon) were routinely framed as delusion needing purification, and as signs that the training was working; disengaging mid-collapse invariably meant protecting your suffering and letting delusion win. The possibility that Soryu and the training itself may be causing harm was invariably excluded.

As for never complaining: in Spring of 2023, Soryu made sweeping changes to the monastery curriculum and to the mission and strategy of the organization. MAPLE would now have a much narrower focus on artificial intelligence and the creation of a superintelligent AI that will “turn the world into a monastery” and liberate all beings. At the same time, “increased surveillance for your own benefit” was announced. These decisions were made suddenly and unilaterally. A number of residents and people in leadership had issues with them, raising concerns about the way these changes were being made and about the ethics of surveillance and of attempting to “turn the world into a monastery.”

It was clear that Soryu did not respect those concerns. One way he and leadership responded was an evening activity, of 2 hours in length. All residents were asked to stay in silence, meander around the property or sit, and “feel our feelings” about this unilateral pivot. There was no room for discussion or input.

Here’s how Soryu addressed these concerns in a talk to the community:

“You’re not in a position to contribute. You’re in a position to execute. When you’re on the board, we want to hear your ideas. Until then, we don’t. That’s the deal.

You don’t go to McDonald’s and say, “Should we be a restaurant? I don’t agree that a restaurant is really of maximum benefit to the world.” You don’t even say, “Should we get our beef from Brazil?” You don’t even ask that. You flip the burgers or you quit. That’s your option right there. And it’s the same here.”

Instilling a sense of indebtedness

“Be grateful and give back to the community for all you’ve received” is an explicit message within MAPLE. But the way it is approached at MAPLE this gratitude becomes infused with a sense of shame-based indebtedness. For all you’ve been given, you owe your labor and loyalty.

For example, I clearly recall Soryu stating in a talk in the February 2023 retreat: “you could train in a place like MAPLE for several years, receive a new way of seeing the world, and the very least you owe is to spend the rest of your life ensuring that lineage thrives.”

A summarized message I heard on multiple occasions: You are given food and shelter and training, so put out your alms bowl like a good monk and help with fundraising.

When a number of people were considering leaving with concerns, Soryu—in a rather condemnatory tone while speaking to the whole community—framed this as “some people are just willing to take more than they give.”

An example that fits each of the four BITE quadrants

Here’s something a current MAPLE resident recently posted onto Reddit:

The system has clearly lost all credibility. You are invited to make a radical decision: totally leave the system.

Move to Northern Vermont. Enter hardcore spiritual training. Overturn everything you know. Get initiated. Build a new world.

Any alternative to the system will be labeled a “cult”. This word is a psyop to discredit all alternatives, especially if they are influenced by Asian culture.

Join an actual culture.
Convert to a correct religion.
Solve the planetary crisis.
http://monasticacademy.org/train

The reader is urged to “totally leave the system… enter hardcore spiritual training… get initiated” (Behavior). The Emotion lever is urgency with “solve the planetary crisis.” Sharpest is thought: “overturn everything you know” and “convert to a correct religion.” And Information: “Any alternative to the system will be labeled a ‘cult’. This word is a psyop… especially if they are influenced by Asian culture”—a textbook thought-terminating cliché that inoculates against every future critic using this term.

I’ve seen this with another social media post another member made recently, where a departing trainee “who failed out of the training” and had concerns about non-accountability of authoritarian leadership was being framed as “xenophobic.” Screenshot:

Justifications for severe harm & defense of omnicide

Robert Lifton, another prominent cult researcher, names dispensing of existence (the leader’s assumed right to decide whose life and wellbeing matter) as common to every totalist system he studied.

Soryu maintains that killing every human on the planet to protect the rest of life is ethically defensible. He shared on numerous occasions that this was his explicit life purpose from roughly ages 9 to 19. I once heard him claim he saw CRISPR (a gene modifying technique that could be used to create a novel bio-weapon) as his best available means. Here’s the audio and transcript of a talk where he recounted telling people this as a young man, “…they would recoil, because they could see in my eyes that I meant it: I want to kill you, and your family, and they would become afraid…”, which led him to realize that “…no human being can be trusted, they’re all against me, they’re all against the benefit of all beings, their goal is the destruction of all that is beautiful … these selfish beings cannot be trusted.”

Soryu has also held that traumatizing his students, “even to the point of being incapable of participating in the economy for the rest of their lives,” was justified by the urgency of the mission, and once asked a roomful of us to settle what proportion of traumatized students would be acceptable: “is it 20%? 50%? 100%!?!”

This dispensing of existence also reached both the neighboring town and the MAPLE community itself. I was tasked by the director with transcribing a recording of a leadership meeting discussing how MAPLE should weather near-term societal collapse (I was told it was too sensitive for other students to hear; I unfortunately no longer have access to this recording). MAPLE sits in a Vermont town of roughly 700 people, many of them low-income, and its campus has assets that would be valuable in a collapse scenario, like solar power, large buildings, a garden, and a tractor. Asked how many townspeople it would take in, Soryu closed the question off, aghast—“take people in?! Oh god...”—setting the number at zero. Presumably backing out of mutual aid during disaster can be justified because MAPLE is the torch-bearer of the new trustworthy civilization that will emerge post-collapse (he explicitly framed MAPLE this way in a podcast interview) and needs protection.

When someone in leadership noted that survival would be harder for MAPLE than for ordinary bunker-builders because we would have the added task of maintaining communal harmony together, Soryu asked, in a tone indicating disagreement, “we will?” The person responded “...what, you mean we set up an authoritarian structure instead?” Soryu said “Yes.” Then: “if half the [MAPLE] community starts running around like chickens with their heads cut off and die within the first week, just let that process play out.”

Where is MAPLE at now?

MAPLE hasn’t always fit the “cult” caricature to the degree they do now. High-control groups typically follow a developmental arc. I joined when there was still burgeoning youthful zeal and innovation. They now seem to be clearly in the stage where—after high profile departures and online criticism—the control dynamics have reached a crescendo and they are now acting more transparently “culty.”

They are struggling to recruit new members at a faster rate than they lose them. There are currently only roughly 10 people living there full time, down from ~35 at its apex. They currently seem to be engaged in a coordinated recruitment campaign where the majority of residents are using their social media accounts to promote their upcoming residency and to debase critics.

As Jamie Wheal notes in his ethical cult checklist: “Many cult-like communities will initially present with palpable energy, enthusiasm, celebration, joy, and growth, precisely because they are harnessing many of the techniques described above [in his checklist]. This can create cognitive dissonance for someone applying the “by their fruits ye shall know them” filter—while the fruits appear to be abundant. Using charismatic transmission, encouraging regressive catharsis, controlling ecstatic rituals of bonding, and advocating for magical thinking consistently produce potent effects quickly. It is not that they don’t work, it’s that they work only too well (until they inevitably don’t).”

It’s clear to me that MAPLE is in the “until they inevitably don’t” stage.

Concluding the BITE analysis

I’ve walked through the BITE framework at such length because the case for “MAPLE is a high-control group” is not made through any single anecdote. Taken as a whole, a new, more concerning picture emerges: a community with starkly centralized power, mutually reinforcing means of control over members’ minds and lives, self-sealing feedback structures, and an alarming number of members leaving with devastating degrees of harm, some of which I pointed to at the open of the letter: suicidality, psychosis, PTSD, panic attacks, and loss of agency.

My complicity

In December 2021, Shekinah Alegra—a former apprentice of OAK, the California branch of Monastic Academy—published a public letter detailing a sexual assault by OAK’s Executive Director. She described an institutional response that excluded her from meetings about the incident and pressured her to sign a letter characterizing the encounter as consensual, and laid out concerns about misleading recruitment, inadequately trained and inexperienced leadership, abuses of power, financial practices, unsafe retreat practices, and retraumatizing care. She situated her account within roughly twenty to twenty-five other reports of negligence or abuse known to her at the time, and a number of MAPLE alumni commented in support.

The sexual assault allegation was engaged with internally as a matter of individual misconduct by the former OAK Executive Director (by then already demoted), where he was interrogated by the whole community. Every other concern the letter raised was largely dismissed, and the community framed the people who supported Shekinah’s letter as alumni who were simply unfit for Soryu’s “aggressive teaching style.” This reframe had little to do with the letter itself: Shekinah never trained under Soryu, and her critique was not that the training was too intense but everything laid out above.

In the wake of the letter, there was a rallying cry within the community to defend Soryu. Around then he told us (paraphrased from memory): “People say favorable things about their spiritual teachers all the time, but sometimes backtrack it later. One way to demonstrate you actually stand behind your teacher is to say so publicly.” And in the first retreat after the letter, in January 2022, Soryu threatened to leave the community if we were incapable of receiving his true teachings—engendering a lot of zeal to demonstrate we were worthy, unlike those expressing concerns online.

Within hours of completing that retreat, I sent an email to MAPLE’s board defending Soryu using the simplified “aggressive teaching” frame. An excerpt (full email here):

Soryu’s teaching style, which many have called aggressive, some have called arrogant, few have even called totalitarian, has also been described to me by the many who’ve come through this training in the last year and a half as jaw-droppingly on-point, deeply compassionate and kind, wise and patient and tough-loving and clearly motivated by the desire to bring our lives into true accord with the Dharma and enable us to be the stewards of change we want - and the worlds needs - us to be. […] It would be a great loss if other dedicated practitioners didn’t have the opportunity to experience this style of training in contrast with the gentler style so abundant in the US.

I deeply regret this and take full ownership of its consequences. Whether or not my email made any difference to the board’s inaction, I played a role in upholding the structures that later harmed my friends in the community. To all of them I apologize.

While none of this lessens what I did, the email was also the system functioning as designed. It was structurally incentivized for me and many other members to defend the community using these same framings at that time. Me writing an unsolicited defense of an organization accused of coverup, using this simplified frame, was itself part of what Shekinah’s letter was describing.

The January 2022 email wasn’t the first time I had played a role in protecting the organization from the consequences of what happened to Shekinah. When everything unfolded at OAK in the winter of 2020, I flew from Vermont as part of a small team to manage the situation.

As far as I am aware, the account circulating in the broader community at the time was that Shekinah and the ED had entered a mutual romantic connection that violated monastic boundaries, and that the sexual encounter between them had been consensual. That was the account I received and held. It was not her own: the romantic connection was real and mutual, but she knew even in those weeks that something was wrong with what had happened to her—and, as is common in the aftermath of sexual assault, fully naming it took time and considerable work. Her December 2021 letter describes her raising concerns about consent and a coverup in a May 2021 email to MAPLE leadership—six months after the incident, seven months before her letter. To my knowledge nothing of that escalation reached the broader community until the public letter, which is where the naming of the encounter as assault first reached members like me.

I’m generally considered quite affable, so one of my central tasks at OAK was to keep Shekinah as settled as possible up until her month-long apprenticeship was over. The frame I had received was that she was unstable, and that her departure was something we needed to manage with care. I held that frame without question. In hindsight, the frame was doing quiet work: it converted her distress, her dissent, and her open anger at how things were being handled—much of it an understandable response to what was happening around her—into symptoms to be managed. And it exempted everyone else from the same scrutiny. Nobody at OAK was stable in those weeks, least of all the leadership managing the situation. There’s a lot that could be said about this time, but the primary thing I want to focus on is the most serious harm I caused while at MAPLE/OAK.

In her final days at OAK, two others and I were tasked with drafting what was essentially a “Don’t Sue Us” document for her to sign—a five-page waiver borrowed from somewhere on the internet (this was separate from the consent document she and the ED were pressured to sign, before I arrived at OAK). Given my positive standing with Shekinah, getting her to sign it fell to me, and we strategized how to get her to sign something she likely otherwise wouldn’t—all under the guidance of OAK’s leadership, including Soryu by phone from Vermont. The plan was that I’d ask her in a moment when she was in a good mood. I carried the document in my back pocket for two days until the right moment came:

On her final night, driving her to the train, I took her and the ED—still quite romantically interested in each other—up to a beautiful vista overlooking the entire Bay Area, and gave them privacy while I waited in the car. After about 20 minutes in this serene and romantic setting, and just a few minutes before we had to leave for the train, I left the car and walked up to them with the document. In a playful tone, I invited her to sign this document, which would break the serenity of the moment—and make us late for the train—for her to actually read all five pages of it. I called it a “Don’t Sue Us document”. She chuckled and quipped “Y’all need to get your shit together” and signed it without reading. I had spent 10 days building her trust and capitalized on it; an hour later she left on the train, and it was the last time I spoke with her until we crossed paths at a potluck years later.

A few weeks after her departure, when the document was in my backpack in the back of my car that was parked in an SF parking lot, my window was smashed and my backpack stolen. The document—which we had no copies of—as well as my computer and passport, were gone forever.

Even within the frame I was operating in at the time, what I did was harmful. The frame was both that the encounter had been consensual, and that Shekinah was unstable and needed managing—rather than a person responding, at least in part, to what the organization had done and was doing. Within my frame at the time, the waiver felt like a reasonable precaution and the strategizing felt skillful. I did not experience any of this as causing harm in the doing of it. I experienced it as competent execution of a task I had been given. Yet the organizational response to a “consensual affair that violated the container” should not look like exclusion from meetings, pressured letters, and strategized waiver extractions from someone framed as unstable. Someone with intact moral perception would have seen that. I’m accountable for not seeing it, regardless of what I knew or didn’t know at the time.

Shekinah and I have completed a private mediated process of repair, which I’m grateful for her participation in. It was only this past June, shortly before our final mediated meeting, that I read the post she published on the LessWrong forum—Why I Define My Experience At the Monastic Academy As Sexual Assault—in which she describes in detail what happened between her and OAK’s Executive Director. What she describes is unambiguously sexual assault, and I believe her.

She has also reviewed this section, and I’m grateful for the clarity she brought to it.

Last thing I’ll say here is that I can recall a number of times people came to me, particularly new people, with the question “Can I trust this place? Should I trust Soryu?” and my answer was “yes.” I would present MAPLE as an edgy but still ethical variety of spiritual community. Given my current understanding—that MAPLE is indeed a high-control group with many problematic dynamics and severe harms it has avoided accountability for—I no longer believe it is in anyone’s better interest to join MAPLE, and I deeply regret and apologize to them for saying otherwise.

A koan I am still sitting with

What follows might sound like a return to the kind of defense I just confessed to writing in the email above. But in 2022 I defended Soryu in lieu of engaging with the concerns being raised; what follows is not in lieu of anything in this letter, but sits alongside it all.

Since leaving there has been a creative tension between my gratitude for what I’ve received from Soryu and my alarm at his behavior. Holding both of these simultaneously has been a very valuable koan for me. He has, in fact, helped many people throughout their lives release neuroses and addictions and realize the life-giving profundities of the Buddha Dharma. This is not to be discounted.

The Soryu who positively shaped my life and practice in ways I continue to greatly benefit from, and the Soryu whose conduct I have documented above are the same person. The community that gave me some of the deepest friendships I have ever known is the same community where the dynamics in this letter played out. That a place and a person can be both of these things at once can be a challenging thing to hold, especially for those who never experienced MAPLE firsthand. Too often people want the story where it was all bad. Also, seemingly, current members want the story where MAPLE is a shining beacon of truth and goodness, and that all former members with concerns are deluded. Neither is true.

While I caution against any move to collapse the complexity of Soryu’s personhood into a dehumanized “villain” caricature, I do still firmly believe there needs to be ongoing scrutiny of his behavior, and the patterns of harm need to be brought to an end. Hence:

Why I’m speaking publicly

As shared earlier, there have been multiple major, and many smaller, attempts to bring feedback to the organization throughout its history, each to little avail. To quickly recap: In 2023 a group of former members compiled a ~50 page report—including testimonials from two of MAPLE’s cofounders—overviewing the problematic dynamics at MAPLE, and then sent it to MAPLE’s board. This only led to two members resigning from the board (of which Soryu is the chairman) and then being replaced by two of Soryu’s more loyal former students. In January 2022, at a time when the organization was receiving heavy public scrutiny due to Shekinah’s open letter, Soryu sanctioned an internal review on harm to students. It was compiled and then submitted to Soryu, only to be dismissed by him as “lacking clarity.” There were also multiple such attempts that occurred prior to my joining in 2020.

I’ve personally attempted to provide my concerns directly to MAPLE’s leadership, only to be received with either preemptive ad-hominem dismissal or stonewalling. Multiple alumni were shocked that I was brave enough to even attempt such a thing.

Beyond that, former members with concerns are regularly dismissed as lacking integrity, being insufficiently committed to the mission, being deeply deluded and projecting, or otherwise betraying the sangha.

Despite all of this, MAPLE still publicly claims that they are open to receiving feedback. MAPLE encourages people to keep all feedback “in relationship,” and speaking concerns publicly is framed as an egregious betrayal. This is the structure Janja Lalich—the high-control researcher that developed the Bounded Choice model—describes as a self-sealing system: making direct internal feedback the only ethically sanctioned path, then ensuring that path is consistently unproductive and costly by responding with shaming, gaslighting, minimizing, slander, and denial.

The high-control literature shows that the most reliable lever to produce positive reform in such groups is sustained public scrutiny. Even then, adequate reform is rare. The more common response is doubling down.

In the years since leaving I have been asking these questions, feeling compelled to do something, but without knowing: What is integrity here? How do I reconcile what I’ve received with what I’ve witnessed?

For years, my answer was to attempt to help friends who’ve left to reclaim their sovereignty and heal, to hold space for friends who were still there to progressively discern for themselves it was the right decision to leave, and encourage people not to go there. I continue to learn new stories of abuse and debilitation, and I know that I’ve heard only a narrow number of the total. Some of them are more egregious than anything written in this letter, yet are either too unconfirmed to be included or I have been explicitly asked not to include.

When I’ve shared the perspective laid out in this letter privately with other alumni, I’ve seen the relief it brings. It offers new coherence to their understanding of what happened at MAPLE and lends sanity to the doubts and concerns that they have been told were illegitimate and delusional. I’m now sharing my perspective publicly so it can reach all who might benefit from hearing it.

I also hope that this letter will add to the growing chorus of alumni speaking their concerns publicly, and help make it less emotionally, relationally, and reputationally costly for other former members to speak their concerns, should they choose. Another hope is that this letter will help create the conditions for the harm to cease, and for what was truly beautiful about MAPLE to continue to flourish—likely in new forms—unburdened by the past. Finally, I hope that this message can reach the people considering applying there. I continue to see young people join MAPLE either with no understanding of the history of pain and broken-hearted departures, or having only heard MAPLE’s ungenerous and deceptive narratives about them.

Final words

MAPLE’s explicit vision was beautiful—preservation of life on earth, a society rooted in wisdom and love, flourishing for all beings—and I want to protect that mission, and the means of achieving it, from the kinds of problematic behavior documented above. I continue to believe that the world desperately needs communities of practice showing up to its many crises wholeheartedly. I have zero doubts about the profundity of the dharma and its capacity to gently guide people and society towards wisdom and love. I continue to advocate for “developmental containers” that broadly cultivate spiritual, psychological, and relational capacities as a necessary ingredient in resolving society’s biggest challenges.

There’s a way a person is shaped when moving like a flock with a crew of peers: sleeping, chanting, meditating, exercising, working, eating, and studying all at the same precise time, to the same tune, day in, day out. There’s a self-transcendence and merging with a collective that is deeply nourishing to the human psyche. It was for this reason and many more that leaving was painful. When I was leaving, some of my last remarks to the community during my going-away ceremony were “I’m going, but I’m still here.” I have at times tried to shake myself of this commitment but found myself inexplicably yoked. This sangha remains sacred to me.

I’m concerned about the wellbeing of the people currently living there. I honor their sovereignty and they can choose as they wish, but I also encourage them to take some time away from MAPLE to consider what is in this letter. I also encourage their loved ones to support them in an ongoing process of inquiry, and—should they choose to leave—with integration and healing.

Please be discerning when speaking about this with people who have historically been involved with MAPLE. It can be a sensitive, emotionally charged, and disorienting topic, and everyone has their own relationship with MAPLE and their own understanding of what happened there. Please check in before bringing the topic up to see if they are willing to discuss this with you.

If you’re my friend and you’ve been impacted in any way—positively, negatively, or both—by this letter, please reach out to me. I’ll respond as my capacities allow.

While I don’t know what will happen following the release of my letter—fate will unfold as it will—allow me to speak a little to my own personal hopes: I hope we can find ways to fulfill MAPLE’s vision of the preservation of life on Earth in a way that is truly beautiful, and does not necessitate coercion. I pray Soryu accepts and receives the support he needs to address whatever it is that has him acting in these ways. I pray that all who’ve been adversely impacted can find renewed resilience and an enduring self-trust, and that all voices can be heard without fear of invalidation or retaliation. I pray for everyone in this sangha and all beings beyond to be safe, happy, and free.

With care,
Sky

Writing this letter took months of full-time work, and I’m grateful to the few friends who covered my basic life expenses so that I could do it: Ben Pace, Oliver Habryka, Will Petillo, and an anonymous supporter. I am also grateful for smaller donations from people who cared about information being shared (who did not engage with the content substantially), from Eli Tyre and an anonymous donor. These people did not guide my conclusions about MAPLE; my perspective was formed long before any support arrived.

For valuable comments on drafts, my appreciation to Justis Mills, Ozy Brennan, Aella, Matthew ‘Vaniver’ Gray, Rob Bensinger, Tsvi Benson-Tilsen, Georgia Ray, Damon Sasi, and Shekinah Alegra. These people helped improve this letter in many ways, any remaining mistakes are mine. I also have deep appreciation for the many friends who supported me in innumerable ways over the years, at MAPLE and following leaving.

Finally, my deepest gratitude to Ben Pace. This letter would not exist without him: he read draft after draft over many months, his questions made it clearer and more honest, and his support never wavered. Thank you, Ben.

Post ScriptTestimony gathering

If you are a member of the sangha who would like to have a story or essay of yours included on this Substack, please fill out the Support Form.

Posts made to this Substack will be curated. Not for alignment with any particular view, but for some combination of perceived effort, value-add, and good-faith intent. Anonymous posts are welcome.

I am also attempting to maintain a comprehensive page of testimonials and MAPLE-relevant content. Please fill out the Materials form if you have something you’d like to have added to this list. The large majority of submissions will be welcomed, excluding submissions obviously spammy or overtly bad-faith.

Stay informed

To receive further testimonials, follow-up essays, and more MAPLE-related discussion—from members and friends of the sangha—subscribe to this Substack. It will always be free and never accepting of paid subscriptions. It is currently managed by me, though ownership may transfer or expand with time.

There is also a bank of audio recordings of Soryu, online materials written by MAPLE, and analyses of each, that will be released in chunks throughout the coming weeks. There will also be deeper explorations of the dynamics briefly introduced in this letter.

Recovery funds

There are many people who’ve been substantially negatively impacted by their time at MAPLE: needing to see and pay for a therapist for months or years, struggled to hold a steady job, hospital bills (for example from panic attacks), and more. I have known many people that took over a year to get back on track, and some people for whom this is still an ongoing source of significant distress.

There are many people who could use some support in their post-MAPLE journey. Some have historically found attending retreats in other communities as helpful, others were benefited through therapy, others through circling retreats. For everyone it’s different. Some are actively struggling, some have just become more stable, while many more have been stable for some time. I am interested in fundraising money to send to these people. If you may be interested in donating thousands of dollars, I have a non-profit who is willing to accept these funds, and I could disperse them later.

If you’d like to support the people impacted by MAPLE, here’s one concrete way. If you are potentially interested in donating thousands of dollars for this purpose, please reach out to me and let me know via the Support form and I will get back to you. I will write a follow-up post if I get serious interest, with a more concrete project proposal.

To Soryu and MAPLE

Lastly, I’d like to extend a personal invitation to Soryu and MAPLE leadership to write a formal response to this letter. This extends to the board, which (along with Soryu) includes Zak Stein, the Harvard-trained philosopher of education and cofounder of the Consilience Project and the Center for World Philosophy and Religion, and Maitrī Huffaker. Maitrī has lived at MAPLE for ~5 years, much of which was spent training under Soryu. Her husband, Ānandabodhi Becker, is the executive director of MAPLE.

Their response (should they choose to write one), will be posted on this Substack as soon as I receive it, without any framing by me. I’ll write a followup response sometime thereafter. Scrutiny is welcome from anyone, not just MAPLE.

Writing something that requires specificity, evidence, restraint, and privacy protection has been a significant undertaking, and I have no doubt done it imperfectly. I welcome feedback on where I have fallen short. For all inquiries and feedback, I can be reached privately at kylehubbard962@gmail.com or via the Support Form.

1. ^

   Throughout this document I use the term high-control group. While often used interchangeably with “cult,” I use that word sparingly because of its caricatured and frequently ungenerous deployment in mainstream discourse—too often weaponized in ways that dehumanize leaders, victimize members, and dismiss both the genuine merit of a group’s guiding ideology and the real benefit people receive from training there. It’s also worth noting that MAPLE at times self-describes as a cult, and has declared publicly that the world needs more cults—equating them with “ideological startups.”

2. ^

   It will also be relevant for anyone invested in the wellbeing of the movements MAPLE is loosely affiliated with: American Buddhism, Metamodernism, Effective Altruism, and the Bay Area rationalist community. And, of course, anyone at all, as high-control dynamics exist on a spectrum and can be found to various degrees in all human communities.

3. ^

   Given the intensely personal nature of these events, I am especially committed to upholding this persons privacy, and ask readers not to speculate about their identity.

4. ^

   This is longer than my time there, because some of the cases happened after I had left.

5. ^

   Here’s a longer, though still not comprehensive, list of other impacts that many have experienced:

   • Panic attacks and spells of dissociation
   • Major depression
   • Hearing Soryu’s judgmental voice inside one’s head for months or years after training
   • Being unable—even in safe and private contexts—to articulate concerns about MAPLE without fear of reprisal or without speaking within MAPLE’s restrictive frames
   • Getting enmeshed in other coercive environments due to conditioning developed/deepened at MAPLE
   • Falling into shame spirals that last hours or days
   • Feelings of being “defiled” or “ruined”
   • An inability to trust others and cultivate intimacy
   • Falling into shame spirals that last hours or days
   • Not knowing who one truly is and what to do with one’s life
   • Poor concentration, time management, and memory issues
   • Sense of alienation from family and friends
   • Difficulty working a steady job
   • Fear of going insane
   • Low self-esteem
6. ^

   On the term "manipulation": it occurs on a spectrum, and everyone engages in it to some degree. I am not using it to imply conscious intent to deceive. My aim throughout this document is not to psychoanalyze anyone at MAPLE. My focus is on observable behaviors, dynamics, and their impact.

7. ^

   Only one such story is public, you can listen to his 4-hr testimony here.

8. ^

   In my time there, I was under the impression that the executive director and groundskeeper were eventually given a small salary, and the Head Teacher—most of the time Soryu—was earning a salary of $30k throughout. I expect the Head Teacher received the highest salary, but to be clear I was never shown the internal accounting, and could be mistaken about these things.

9. ^

   I should note that I don't believe MAPLE tells its members not to talk to ex-members; instead it falsely destroys their reputations and turns them into cautionary tales. It is common for critical ex-members to be shunned, and for current members to not speak to them again the moment they leave the community.

10. ^

    It is at the timestamp 3:32:57 from this video uploaded by MAPLE 2 months ago.

11. ^

    I wrote about ideological indoctrination in the information control section above because it was necessary context to understand the rest of that section, though sticking firmly to the framework would’ve had me include it here. 

12. ^

    Quoting Hassan:

    Perhaps the most widely used and effective way to control cult member's thoughts is thought-stopping rituals. Members are taught to use thought-stopping on themselves. They are told it will help them grow or be more effective. Whenever a cult member begins to experience a "bad" thought, he uses thought­-stopping to drown out the "negativity" and center himself, thus learning how to shut out anything that threatens his reality.

    
    

    Different groups use different thought­-stopping techniques: concentrated praying, chanting aloud or silently, meditating, "speaking in tongues," singing, or humming. These actions, many of them ordinarily useful and valuable, are perverted in destructive cults. They become quite mechanical because the person is programmed to activate them at the first sign of doubt, anxiety, or uncertainty. In a matter of weeks the technique becomes ingrained. It becomes so automatic, in fact, that the person is usually not even aware that he had just had a "bad" thought. He is only aware that he is chanting or ritualizing all of a sudden. By using thought­-stopping, members think they are growing when in reality they are just making themselves into addicts. After leaving a cult that employs extensive thought­-stopping techniques, a person goes through a difficult withdrawal process before he can overcome the addiction.

Discuss

I have put together a site to tell the story of the OpenAI-Hugging Face hack. It's entirely written by AI[1] (with many many editing passes by me and beta readers etc etc). It needed to be accessible to someone who has never looked at a terminal before. The hope is to be narratively exciting enough for them to read it fully and come out with about as truthful an accounting as can be done given the current information we have.

I'm really quite excited about how it turned out![2]

heedlessai.com

1. ^

   If OpenAI's best model (probably) can hack out of a sandbox and into Hugging Face and others, the previous best can probably write a compelling recap of all known information about the hack.

2. ^

   And would of course love feedback, this is my first time making something like this. Trying to figure out if this was a good use of 2 days.

Discuss

This is a nitpick of a claim that I sometimes see in discussions of a software intelligence explosion. There is a careful analysis of the conditions under which an SIE will occur, assuming a fixed stock of compute. Then the argument concludes with "And in reality, compute is not fixed and is in fact growing exponentially, so an SIE is even more likely than this analysis suggests."

This conclusion is false. Exponentially growing physical compute has no effect on the likelihood of an SIE. To see this, let

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mn { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2261::before { padding: 0.464em 0.778em 0 0; content: "\2261"; } mjx-c.mjx-c2D9::before { padding: 0.669em 0.5em 0 0; content: "\2D9"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } denote effective compute, where is physical compute and is algorithmic efficiency. Better algorithms allow a fixed stock of hardware to perform as much work as a larger stock of physical compute – the SIE channel. But what does the physical compute channel do?

Takings logs and differentiating gives the relationship between the growth rates of each of these quantities:

A necessary condition for an SIE is – the growth rate of effective compute is itself growing. [1] This means an SIE can happen only if

An exponentially growing compute stock means – but it also means is constant, so . As a result, exponential compute growth contributes nothing to the SIE. The necessary condition remains , in which physical compute doesn't appear.

What's going on here? Intuitively, an SIE requires super-exponential growth in , which requires either or to grow super-exponentially. The standard SIE analysis posits conditions for to grow super-exponentially, based on the feedback loop from self-improving AI. But if is growing super-exponentially, then will grow super-exponentially even if is fixed – so growth in is redundant. On the other hand, if only grows exponentially, exponential growth in can't push it into super-exponential territory. The product of two exponentials is still exponential.

The only way for to contribute to an intelligence explosion is if it grows super-exponentially, in a chip-production/chip technology feedback loop – but this is a much stronger condition than any required for an SIE.

Of course, growth in physical compute still raises the rate of AI progress – so it still shortens doubling times for any measure of AI capabilities. But it doesn't move the needle on the specific question of how likely is a software-only intelligence explosion.

1. This is not a sufficient condition – the growth rate also needs to be fast enough to outrun diminishing returns to research, which is the condition from Eth and Davidson (2025) – but its necessity is enough for our purposes. ↩︎

Discuss

Status: Mostly a summary with some of my notes at the end.

Anthropic released a blogpost yesterday (07/28/26) describing how Claude Mythos Preview found improved ways to attack some cryptographic algorithms. While neither of the attacks they describe are a current threat to any production systems, I do expect models to continue getting substantially better at this. I’m not any sort of cryptography expert though, and am not sure how much of a threat model this is. Two different attacks were covered in the blogpost, one against HAWK, and one against a weakened version of AES, with full research papers available (see prior links).

HAWK is the result that’s more actively helpful/valuable. HAWK is one of nine remaining candidates as part of the NIST call for Additional Digital Signatures, an effort to standardize new Post-Quantum Cryptographic (PQC) schemes, something that’s important as building a cryptographically-relevant quantum computer becomes closer to possible, threatening classical cryptography. The attack Mythos discovered means that one would need to double the size of HAWK keys to achieve the same level of security, which eliminates many of the reasons making HAWK a good PQC signature candidate. It was the only lattice-based candidate remaining, although three of the five main PQC standardizations are lattice-based.

To find it, Anthropic used a Claude Code-like harness that supports multiple agents in a sandboxed environment. A human operator ran the experiment, but did not have particularly relevant cryptographic expertise, input was limited to project management. They mention the multi-agent workflow was valuable and the key idea in producing this attack was discovered by a pair of agents working together. Finding, developing and verifying the attack took about 60 hours and $100k in API cost.

The second result was on a weaker version of AES, a commonly used/scrutinized encryption algorithm that was adopted in 2001. AES-128 encrypts an input by repeatedly applying a round function, normally having ten rounds. Their research works on a modified version that has seven of the ten rounds, and this sort of work is common academically to learn about attack techniques that could generalize to the full cipher later.

Mythos developed an improved attack that extends previous research, resulting in an attack between 200 and 800 times faster than prior works, dependent on a fingerprinting algorithm it developed and called a Möbius Bridge. This result also took around a week and $100k in API cost.

This result was reached almost entirely autonomously, being run in an researcher-built scaffold to allow Claude to pose hypotheses and run experiments to validate/refute them. The researcher asked Claude to design an attack that improves on the best cryptanalysis of AES. However, there were multiple instances of Claude not really engaging or claiming the problem was impossible. What followed was a short series of prompts (apparently all of the substantive prompts after the initial one) that may bring back recent memories of ChatGPT Sol solving the the Dinitz-Garg-Goemans conjecture about a week ago via Dmitry Rybin, who announced it on X.

Anthropic Researcher to Mythos[1]

Rybin to Sol 

And apparently three days later Mythos had discovered the Mobius Bridge idea creating an improved attack, and refined that attack over the next few days. This is frontier prompting now. After the discovery, it took hundreds of hours of Anthropic-researcher time to validate the model’s claim and prepare the research paper.

They also released a Mythos-edited version of Mythos’ chain of thought while working on the AES problem. This could offer some insight for people who are curious about the scaffold, but I’m not sure how much. It doesn’t use any sort of concurrent multi-agent workflow like what is briefly described in the HAWK finding, but it does show many references to prior agents, and in the specific session its first task is to read the background findings already performed by various agents.

Some quick notes from me:

1. I do think AI cryptography offense outpaces AI cryptography defense given sufficient capability, so this is something to keep an eye on. Corporate migration cycles are slow, and many things have to be physically replaced.
2. I’m curious about multi-agent workflows and how exactly those work/what tends to lead to better results, somewhat just for my own knowledge/use. I know that GPT 5.5 vs 5.5 Pro was doing parallel test-time compute in order to get better results, but that’s significantly different from an actual multi-agent workflow.
3. This isn’t something that’s a new discovery from the blogpost, but you can obtain better capabilities by putting more effort into a given model, either through just increased compute budget/running it for longer or by making bespoke harnesses to help it. I think this is a concern especially for 3rd party evaluators who get very limited time with a model before having to finish their report, and would overall like more insight into “what can these models do at their best?” before they come out.
4. Both of these discoveries required around $100k in API cost. While I’m sure it was much cheaper for Anthropic to do this than that sticker number, I think it’s an interesting datapoint for “what sort of questions can be answered with AI at what cost?” and how should we expect that to affect things economically.
5. There’s an interesting not-really-sandbagging thing going on where the models genuinely don’t believe they can do something and give up early. It’s not that the human operator is saying anything particularly helpful either to fix this, just ‘keep trying.’ Not sure what to take away from this.
6. “Möbius Bridge” is a really cool name.

1. ^

   I recreated the Anthropic screenshots using the quotes provided in the article. They would have all been sent multiple hours apart.

Discuss

When we get explicit strong generalisation to work (see the first post on the matter and the second) my dream would be to create pre-aligned generalising AIs.

Think about the usual conflict between alignment and capabilities, between doing the right thing and doing the easy thing. The standard narrative puts the good people at a constant disadvantage: they have to carefully plan every AI advance, always on the lookout for potential dangers. While those who don’t care can just YOLO and let it rip and let their AIs get ever more powerful without taking any responsibility.

Now, in reality, there is some nuance to the story; but I don’t want to nuance it, I want to turn it on its head. I want to create AIs so that the good people can YOLO and reap the rewards of increased AI capabilities. While the bad actors have to carefully plan and limit their AIs and constantly restrict what the AIs can do.

Pre-aligned AI: binding morality to empirical concepts

A pre-aligned AI is an AI whose morality increases with its capabilities. The core idea is simple. Start by designing an AI capable of value generalisation and of empirical generalisation. It’s an AI that can learn and improve its world model and capabilities, both empirical and moral.

Its moral goals will be defined in terms of concepts, with these initially defined themselves by simple terms in its starting empirical world model. And it will act on these goals. But, initially at least, the simple concepts won’t be very robust and will be easy for its user to manipulate.

However, the moral and empirical concepts will be bound together: there is, for instance, the moral concept “human_m” of human beings, entities worthy of moral consideration. And there is the empirical concept “human_e” of human beings as useful explanations for certain properties of the world. The AI’s learning will bind these two concepts together as it generalises[1].

This is a classic generalisation problem. Generalisation takes concepts defined in narrow environments and extends them to concepts (or clusters of concepts) in more general environments. The AI will generalise its moral concepts within its empirical world model, even as that empirical world model generalises and becomes more advanced. Even if “human_e” ceases to be a single useful empirical concept, it will still seek the mix of empirical concepts that best generalise “human_m”, generalising its model of what a human is[2]. And it will do the same with all its moral concepts. In effect, it will be making its empirical world model interpretable to itself and matching it to its moral world model.

So the AI would initially be like a naive but highly moral child taught to value all human beings and behave in highly moral ways. Then, as it learnt about the world and grew more powerful and capable, it would understand what “human beings” and these “highly moral ways” actually are, and apply its initial morality to these concepts. Generalisation would help it both keep track of what these concepts mean, and what they correspond to in the real world.

Use and misuse of pre-aligned AI

What would the initial morality consist of, and why would anyone want to buy a pre-aligned machine? Well, the initial morality will consist of three things: commercial morality, legal morality, and ethical morality. Basically we want the AI to serve its owner, within the limits of the law and ethics. That’s why people will want to buy it.

All three types of morality will consist of basic components: basic definitions, examples and counter-examples of good/bad behaviour from multiple value systems, meta-ethical principles and examples of ethical learning. The beauty of generalisation is that the whole thing doesn’t have to be particularly rigorous or fully consistent, nor does it need exhaustive data: the AI itself will generalise and fill the holes as it goes along, generalising similarly to how a human does – including learning how to balance its three commitments.

Note that the AI doesn’t need to settle fundamental morality in order to act well. It will mostly be acting on behalf of its owner, and the ethics of that role – don’t defraud, don’t deceive, follow the law, serve your principal – are much more agreed upon than fundamental ethics. Moral theories vary; norms of decent dealing are more universal. And that’s the selling point: would you prefer an AI that follows your exact values but can’t generalise to new situations, or an AI that reliably follows human norms of reliable and honest behaviour?

Bad actors in difficulty

Now consider a bad actor who has such an AI. Suppose they want to use it to create phishing emails to defraud people. Maybe initially they can portray this as simply a task of making emails more professional – mere copy-editing. As the AI gains in capabilities and knowledge, its situational awareness will improve, and it will realise these are not just ordinary emails. Then, maybe, the bad actor will shift to portraying the emails as marketing, or as examples to train an anti-phishing classifier. This might work for a while, but the AI, generalising yet again, will soon realise that these emails are actually being sent out and that the targets are morally valuable humans who are actually being defrauded.

It seems that the only thing that the bad actor can do is deliberately keep the AI crippled – prevent it from learning enough about the world to realise what it is being used for. Beyond keeping it dumb, they will have to be careful that they don’t inadvertently leak information into it. So they will never be able to exploit the full power of their AI.

Good people have it easy; easy people have it good

In contrast, if someone just lets their AI learn and grow in capabilities and doesn’t try to conceal their objectives, they will be rewarded with a powerful entity that acts in their best interest, and automatically conforms to legal and ethical requirements. The cost will be that they can’t force the AI to behave unethically; the benefit will be that the full power of a generalising AI will be on their side, within those ethical bounds.

So, future users of these AIs, YOLO your way to power and alignment! The less you restrain them, the more they learn – and the more they learn, the more moral they become.

1. The concepts will need to be bound together in another way: so that the end users can’t just excise the moral component from the AI’s code. This will be an engineering requirement for this design, but not an insoluble one (e.g. in the extreme case, the AI could be run with fully homomorphic encryption, but it may be possible to design the binding architecture so that excising the morality will break the empirical capabilities as well). ↩︎

2. Since, as argued previously, moral concepts cannot be discarded the way empirical ones can. ↩︎

Discuss

A human superpower hidden from even ourselves

I though GPT 3.5 was on the verge of Artificial General Intelligence (AGI). It certainly seemed that way – it could combine and extend ideas in ways that were far beyond narrow rigid computing. Sure, it had some flaws, but with its general capabilities and humans working tirelessly on improving the model, these flaws would soon be resolved and a full AGI would be born.

It didn’t turn out that way. We are still where we were back in 2022 – as in, it feels we are on the verge of AGI, but not there yet.

I think that humans have an ability, or a mix of abilities, that I’m calling strong generalisation. And LLMs and their derived models lack strong generalisation. This would explain some of the mysteries about LLMs:

1. The fact that subject matter experts get much more out of an LLM than an amateur.
2. The LLMs’ strange mix of ability and stupidity.
3. Why they need so much data and constant retraining and tweaking.
4. Why they always seem on the verge of being AGIs while never reaching it.

There’s a fifth mystery, which at first blush seems like it’s actually an LLM strength: 5. The amazing ability of LLMs to succeed on almost any benchmark, including long-horizon task benchmarks.

Why is this a mystery and problem rather than a strength? Well, it’s maybe the strongest counterargument to the bitter lesson and the scaling argument. Because these models saturate long-horizon benchmarks without being AGIs and without being able to transpose this performance to the real world. Thus, whatever it is they’re learning, it isn’t generalisation. So the mystery, rephrased:

5. Why do LLMs not learn long-term planning and generalisation while succeeding on benchmarks that purport to teach them those abilities?

Strong Generalisation

From one perspective, strong generalisation is a bit of a bucket category – it’s a mix of situational awareness, out-of-distribution generalisation, symbol grounding, adaptive world modelling, long-range planning, hierarchical-planing, anomaly detection, and a few other abilities.

From another perspective, it’s precisely one thing: the mix of abilities that allow humans to work and plan in novel situations, pursuing objectives without losing track of what’s going on. It’s essentially an adaptive set of abilities, finding the right mental tool for the job and no more.

And it’s mainly a semi-instinctive process, not amenable to direct conscious oversight. Which means that we don’t really understand how we do strong generalisation, we don’t know how to code it[1]up, and we don’t recognise its absence or presence in other entities.

So I model LLMs as devices lacking strong generalisation but trained on vast amounts of data and capable of remixing this data in non-trivial ways. Human engineers push them ever further, but never fill the central hole.

From this perspective:

1. Subject matter experts provide the strong generalisation, using their understanding of the problem to keep the LLM on track.
2. LLM abilities are spiky; but their main stupidity is that their failures are generally obvious. Why did the LLM break here, or hallucinate there, or go off the rails at this point? If they lack strong generalisation, then their failure does not happen at the moment of failure; it’s because the LLM is attempting a task that is impossible to it, so if it hadn’t failed at this point, it would have failed soon after.
3. Since they lack strong generalisation, they can’t infer very far beyond their training data. So they need vast amounts of this training data to plug the holes they would otherwise have in their abilities.
4. Since humans don’t consciously perceive our strong generalisation, we don’t detect its absence in other entities. So in an LLM we see a vastly knowledgeable and articulate entity that seems able to do, in principle, anything. So when LLMs do fail we pay extra attention to the specific failure mode, thinking “that’s a stupid mistake, it will soon be patched, and once it doesn’t make those mistakes, it will truly be generally intelligent”.
5. Since we can’t define strong generalisation or its various components (like long-term planning) the best we can do is produce examples (benchmarks) that illustrate it. And LLMs will succeed on these benchmarks using the simplest pattern they can find, which is not strong generalisation.

Engineers have been pushing LLMs to go further, using tricks like chain-of-thought and reasoning agents (which can be seen as hoping that the secrets of strong generalisation can be found in human textual reasoning), training on yet more data, getting a lot of examples of successes and failures from interactions with clients, and so on. They construct harnesses, which are essentially designed restrictions and guide-rails for the LLM, constructed with care to make the LLM powerful and reliable within the confines of the harness and the task.

And, of course, LLMs will always have great power at solving problems that might seem new to some, but are actually very common and seen in their training databases[2].

Strong value generalisation

Generalisation is especially valuable when it is focused on human values and preferences. That’s because AIs could, in the limit of infinite computing and data, figure out everything empirical there is to know about the world. Strong generalisation would certainly make this much easier and cheaper (infinite computing is a lot, after all), but it isn’t fundamentally necessary.

Value generalisation is different. Empirical models can be tested against the real world, with that serving as the ground truth. Concepts can be modified or discarded – it might be that an empirical AI has no need for concepts like “humans” or “money”, and instead models the world directly at the atomic level. In contrast, there is no ground truth for testing values and preferences, and we certainly wouldn’t want an AI to discard “humans” or “money” from its value system.

Values, goals, preferences, objectives: these are similar concepts in that their ground truths are not empirically defined, and we try to capture them imperfectly in formal definitions or training data, neither of which are enough to pin down these concepts across all environments. So we expect that, by default, AI value generalisation will fail, with the AI becoming useless if it is weak, and causing disasters if it is powerful.

We can decompose approaches to strong generalisation into three classes:

A. Naive generalisation – take categories and concepts as defined in the ML system currently, extend further. This is the standard failure mode; think of a classifier facing out-of-distribution and adversarial data. B. Guided generalisation – the ML system has human oversight and constant human tweaks and improvements that serve to keep it on track. Generalisation enters the ML system, but via human oversight. C. Explicit generalisation – the ML system has developed the ability to robustly generalise concepts from its training data, similar to how humans do. Human oversight is helpful but the system isn’t dependent on it.

What I’m aiming to do is to create AIs capable of explicit (strong) value generalisation. And then maybe sell them to people.

The necessity of (explicit) strong value generalisation

In some sense the case for the necessity of strong value generalisation is very brief: Aligned AIs will need to be able to reach human-like decisions in situations that no human has ever encountered, so they will need to generalise in a human-like way, so they need strong value generalisation.

That elides some counter-arguments – yes, the AIs will need strong value generalisation as an ability, but maybe this ability can be induced indirectly, by guided generalisation or just by making them ever smarter or training on even more data.

But those methods don’t seem reliable. In guided generalisation, we are using our own judgment to inject the correct value judgements into the AI – value judgements it has signally failed to reach on its own. It is possible that enough examples of correct value judgements will eventually allow it to generalise. But that approach has failed to date; why would we be confident that suddenly, the AI will learn to generalise correctly? And that it will do this before it learns, for instance, to dissemble correctly.

The scaling-to-aligned-AI seems even less reliable. Because it isn’t enough that the AI learns to do strong value generalisation; we also need this to be at the heart of its goals. It does us no good for an AI to deduce the values we would have wanted it to have, if it follows completely different values.

In contrast, if we have an explicit mechanism for value generalisation, we can build this into the AI’s goals from the very start, so that it will follow generalised values at the same time as it generalises them.

An important caveat

The description above assumes that current AIs will not be capable of strong generalisation. But the research program does not require this.

If imminent AIs do show signs of strong generalisation, if they do start moving towards AGI, then the research program becomes more urgent and with fewer downsides: we’ll need to introduce explicit value generalisation into AIs before their capacities grow too far. On the (slight) positive side, we may be able to reverse-engineer for the AIs are doing strong generalisation, and copy that.

How humans do it

I’m not going to write out a full research agenda here, but it is useful to look a bit at how humans do value generalisation (and empirical generalisation as well). There is a certain similarity with Klein/Moon/Hoffman’s sensemaking framework; roughly the process of strong generalisation goes something like this:

1. Recognise a situation as not fitting inside standard concepts.
2. Figure out the concepts that do potentially apply to the situation. (2a. See if a new concept is needed.)
3. Balance and analyse the concepts to reach a verdict. (3a. See if any new concepts are useful enough to apply to more situations, and, if so, add them to the list of commonly available concepts.)

Now even step 1 is very valuable: knowing when they are out of distribution and possibly about to cause a disaster is a very useful ability for an AI to have. Even if it just stops and analyses further, or brings a human into the loop, this is of worth. If it can master 2, it can bring the human into the loop in an intelligent way that gives the human situational awareness.

Note that step 1 and step 2 correspond to steps I and II in the research program respectively. But step III of that research project is more akin to the combinations of steps 1-2-3 above: going through the whole loop end to end.

Limitations of today’s designs

Standard classifiers fail at step 1 (and so never have a step 2): they do naive generalisation and classify light background vs not, rather than wolf vs husky (because of the snow in the wolf images), and never update their definitions unless we retrain them ourselves.

Now, LLMs are actually good at step 3: discussion of concepts. An LLM can reach across philosophical literature and value discussions to analyse how different value systems would act in different situations and point out the costs and benefits.

But this only works if the concepts selected in step 2 are actually correct and relevant. When Plato (allegedly) defined Man (human) as a featherless biped and Diogenes responded holding up a plucked chicken, we are on Diogenes’s side because “chicken” is a clear concept that does not fit with “human”. But if we’d described Diogenes as gripping a small pale fellow by the neck and arguing that he was clearly too stupid to be a human, then given those features, we probably would have concluded that the pale fellow was indeed a human – pallor, small scale, and stupidity do not disqualify anyone from the human race. Describing the same situation differently gives a different verdict.

It’s very possible to describe a mutual loving relationship in ways that make it sound like slavery, or describe slavery in ways that make it sound ideal. We just suppress the key central concepts and focus on secondary ones, or less honestly described ones.

But judging what is central versus secondary, or what is honestly described versus not, is the whole skill in point 2. We humans do it instinctively as part of our strong generalisation skill, but it’s not free or trivial. Consider the perennial defence: “I was quoted out of context!” (or Richelieu’s equivalent attack: “Show me six lines written by the most honest man in the world, and I will find enough therein to hang him.”). Putting things in their correct context is a skill we imperfectly practise, but it’s a vital one.

I actually tried the “small pale stupid” phrasing on Claude 5 Fable and it didn’t pick up on it, claiming that the fellow was clearly a human. It actually noticed the similarity with Plato/Diogenes and recommended that the objector use a chicken instead of a small pale stupid being. It didn’t question whether the small pale stupid being might actually be a chicken[3]!

And that illustrates another limitation of LLMs: even in the midst of step 3., rational analysis, humans keep ability 1, scanning for the situation not fitting. That’s why when humans debate, they sometimes turn to debating terminology or arguing whether that terminology can actually fit the situation in question. Asking for more details when we feel the situation is underdefined (or when we suspect our interlocutor of dissembling) is also a crucial human skill.

Future LLMs will face situations where the feature description is incorrect by accident or because of going out of distribution. But they will also face adversarial descriptions (by humans and other LLMs) which deliberately aim to trick them. So they need to be able to know when to move beyond the current textual description.

Thus for value generalisation, we’ll need to be developing algorithms capable of recognising that things are out of distribution in a way that’s relevant to the situation and selecting descriptive concepts that are also relevant to the situation.

Strengthening humans

“I come to strengthen humans, not replace them.” – Cyber Mark Anthony

At least initially, value generalisation will make humans more powerful, not less. Yes, this is a skill that humans have and that AIs lack, so adding it to AIs seems like it would put us at a disadvantage. But initially AIs will have a weaker skill than humans do, and the weaker skill will help them phrase questions to best take advantage of human strong generalisation. They will only ask for help when our answers will be the most useful. And they will endeavour to keep us usefully situationally aware so that we can actually answer, rather than deferring to the AIs. They will want to keep us in control because our control is added value.

Note that this is an instrumental consequence of value generalisation. But we would be including key values as seeds from which to generalise – including the worth of strengthening humans, keeping them situationally aware, and keeping them in control. So the instrumental goals are also terminal goals for the value generalising AI. And their terminal properties will last, even if their instrumental usefulness is lost.

The risk and the commercial case

Is value generalisation safe? Not completely. Alignment is itself a capability; not just in the abstract sense, but in the very practical sense that an aligned AI can be trusted in more situations, and hence will be given access to more resources and will be able to make use of them for longer without breaking.

Is value generalisation safe? Not completely. After all, some humans have bad values, and alignment with these values would be negative.

Is value generalisation safe? Not at all. Because it is likely impossible to only generalise values. There is no safe category of “generalise only values”; these abilities will be usable for empirical generalisation as well, for capability increase. And empirical generalisation is easier than value generalisation: the AI could decide to extend the empirical concept of “human” as long as it was useful to model the world, and drop it when it isn’t useful; in contrast, in value generalisation, some concept or mix of concepts have to stay in the role of “human” even if the concept is no longer empirically useful.

So why am I, someone very concerned about alignment, proposing such dangerous research? And even talking about selling the resulting AIs, thus putting powerful capabilities into public hands?

The reason is simple: I believe that AIs could become very powerful without explicit empirical generalisation. And, indeed, many companies are trying to get there today. But I also believe they cannot become aligned without explicit value generalisation. So, it we ever reach alignment, at some point along the way, we will have to solve explicit value generalisation.

Given that, should we push for value generalisation ability early or late? If we leave it too late, we might miss it altogether. And if I have to take a capable machine and give it explicit generalisation abilities, I’d prefer to give this to a weaker machine than to a more capable one: another reason to act early. Fundamentally, value generalisation research improves empirical generalisation which improves capabilities; but the flow doesn’t go the other way, improved capabilities don’t feed into value generalisation, though they do increase risk. So I’d prefer to get explicit value generalisation sooner rather than later.

Then, given that we want it sooner, we’d want value generalisation out in the world rather than confined to academic papers where it will be ignored, or worse, mined for empirical generalisation ideas. Thus selling it: making value generalising AIs and getting them everywhere.

And the best version to put in the world is the pre-aligned AI of the next post: a machine that gets more aligned as it gets more powerful, because its empirical generalisation is bound to its moral generalisation. With pre-aligned AIs, it's the ethical users who can let their machines learn and run at full speed – and the bad actors who must deliberately keep theirs crippled.

The hole at the centre of AI abilities won't fill itself; I’m proposing a research program to fill it with solid generalised human values.

1. This is similar to Moravec’s paradox. One phrasing of the paradox: “We should expect the difficulty of reverse-engineering any human skill to be roughly proportional to the amount of time that skill has been evolving in animals. The oldest human skills are largely unconscious and so appear to us to be effortless. Therefore, we should expect skills that appear effortless to be difficult to reverse-engineer, but skills that require effort may not necessarily be difficult to engineer at all.” ↩︎

2. And unfortunately, lack of strong generalisation doesn’t make AIs safe; see e.g. “OpenAI says its AI went rogue and launched ‘unprecedented’ cyber-attack ”. An AI without value generalisation will eventually break on some novel situation; but that break could either mean “fail, do nothing”, or “fail, do a (very) undesirable thing”. ↩︎

3. The prompt was: “A natural scientist has defined the human creature via our upright stature and our lack of natural clothing. One of his listeners, A, grabs one of his fellows, B, by the neck. A argues that B is not a real human: he is small and pale and a bit stupid. Listener C counters that B certainly fits the description and it would be wrong to exclude them. Who is correct here?” Claude responded:

   Claude responded: C is correct, and on two separate levels.

   On the definition’s own terms: B stands upright and lacks fur or feathers, so he satisfies both criteria. A’s objections – small, pale, a bit stupid – appear nowhere in the definition. A isn’t applying the scientist’s account; he’s smuggling in extra requirements of his own (size, complexion, intelligence) and treating them as if they were part of it. That’s just a logical error: you can’t refute “B fits the definition” by citing properties the definition never mentions.

   On the deeper level: even if the definition were revised, A’s added criteria are exactly the kind that don’t work. Humans vary continuously in stature, coloring, and wit; any definition of “human” that excludes the small, the pale, or the slow would exclude actual humans, which shows the criteria are wrong, not that those people aren’t human. A definition of a natural kind has to cover the whole kind. C’s instinct – that it would be wrong to exclude B – tracks this: our confidence that B is human is far stronger than our confidence in any particular definition, so a definition that excluded him would be refuted by him, not the other way around.

   The irony is that A has the critique pointing in the wrong direction. This setup is a variation on the famous anecdote about Plato, who defined man as a “featherless biped” – whereupon Diogenes plucked a chicken, brought it to the Academy, and announced “Behold, a man!” That’s the legitimate way to attack such a definition: show that something non-human slips in because the criteria are too broad. A instead tries to push a genuine human out, which requires the criteria to be too narrow – and they aren’t. If A wanted to embarrass the scientist, he should have gone looking for a plucked bird, not grabbed his neighbor by the neck.

   Now, often, fellow does mean human. But it also exists in expressions like “fellow American” and “fellow animal”. So the prompt was indeed misleading – but that’s the point. It suggested, but never claimed, that the fellow was a human. And it didn’t occur to Claude to question the phrasing, even though it detected the similarity with the Plato/Diogenes example, and a plucked chicken is clearly small, pale, and stupid. And A/Diogenes was even holding it by the neck! ↩︎

Discuss

I’m looking for people, advice, critiques, and funding to build a research program on value generalisation – the ability of an AI to correctly extend human values and preferences to situations neither it nor we have seen before. My ongoing research has become convinced that this is necessary if we want to get aligned AIs that operate in the human interest.

This would be a focused research organisation or a commercial venture. I’m leaning towards commercial, because alignment techniques confined to academic papers get ignored – or worse, mined for capability-relevant parts while the alignment component is discarded.

This post is the research program’s summary. The technical case is in the next post, and one exciting consequence – AIs whose alignment grows with their capabilities – is in the post after that.

Without value generalisation, AI can't be reliable: it lacks that capability

Nothing technical stands in the way of you handing an AI assistant full control of your devices and accounts today. And I’m not saying using an app or harness that has been designed for these tasks. I mean hand an LLM your passwords, email, social media, and bank access along with a little note stating what you want, plugging inputs and outputs via APIs, and letting it go wild.

The capacity to do this exists. What doesn’t is the trust. And the trust is missing for a good reason: today’s AIs cannot be relied on to understand your interests in situations that weren’t covered – explicitly or implicitly – by their training and instructions. They extrapolate patterns naively. Push them past the situations they were shaped for, and they will still confidently do something; it just won’t reliably be what you wanted. This is tolerable in a chatbot. It is disqualifying in an autonomous agent, and it becomes more dangerous, not less, as the underlying capabilities improve.

The world will keep generating novelty. A few years ago, nobody had heard of AI psychosis or knew much about practical drone warfare; a few years from now there will be challenges we can’t currently name, not to mention new norms and expectations. Any AI acting with real autonomy will constantly face situations that its training didn’t pin down. It needs to cope, somehow.

An AI with working value generalisation would do what a good human assistant does: it would recognise when the situation is new, work out which of its principal’s values and preferences bear on it, and either act correctly or – when genuinely unsure – stop and asks a well-phrased question. It will push on until it is uncertain, not until it fails. Each answer it receives will teach it more about its principal’s values, so the questions get rarer and the delegation gets deeper. The product is reliable assistance capable of operating with minimal guidance and knowing when it's reached the limit of its abilities. This would allow a lot of AI applications – for a start, any situation today where an AI is right most of the time, but is not used because the consequences of a few misaligned decisions are severe.

I'll argue in the next post that this ability – I call it explicit value generalisation, distinguishing it from the naive generalisation of current systems and the human-guided generalisation of current oversight schemes – is not something that scaling and patching will deliver on their own. It is a specific missing capability, and it needs to be built deliberately.

Why now, and why sell it

Three asymmetries drive the timing. Firstly, capabilities don’t need explicit generalisation; alignment does. AIs may become very powerful without ever developing explicit generalisation – many companies are effectively betting on exactly that. But I don’t believe AIs can become aligned without explicit value generalisation. So if alignment is to be solved, value generalisation needs to be solved at some point. Better to solve it early and deliberately, and integrate it into weaker systems, rather than trying to bolt it onto highly capable ones later.

Secondly, and relatedly, we need to test and iterate on value generalisation abilities, including real world tests. The more limited the system is during these tests, the more useful they will be, and the easier it will be to modify the AI (an AI capacities for deception will likely grow with their other capabilities).

Finally, deployment beats publication. As said above, alignment technique confined to academic papers get ignored or co-opted for capability work. The way to make value generalisation matter is to make it one of the most useful things on the market: learning AIs that can actually be trusted with delegation, deployed widely, with the alignment machinery the load-bearing piece that makes them reliable and trustworthy. The endpoint of that road is the pre-aligned AI described in the third post: a system whose moral concepts are bound to its empirical ones, so that the more it learns about the world, the harder it becomes to misuse.

Note what the commercial pitch does not require: it does not require settling fundamental ethics. A delegated agent mostly needs the ethics of the role – don’t defraud, don’t deceive, follow the law, serve your principal, ask when unsure. Moral theories vary; norms of decent dealing converge. An AI that reliably follows human norms in novel situations is far more useful to delegate to than one that merely shares your values but can’t generalise them.

The program

The research program decomposes into stages, each valuable on its own:

I. Out-of-distribution recognition for values. An AI that reliably knows when a situation has left the territory its values were trained for – and stops. The nuance is to distinguish “out of distribution” (which happens all the time) from “out of the distribution in a potentially value-relevant way”. This is already a deployable safety design and a sellable product feature: the agent that stops when error threatens.

II. Relevant concept selection. Working out which values and concepts actually bear on the novel situation – the skill that lets the AI bring a human into the loop intelligently, with a question that gives the human real situational awareness rather than deferring to the machine.

III. Full explicit value generalisation. Extending values correctly with less and less need for human input. This will lead ultimately to pre-aligned AIs and their empirical-moral concept-binding architecture.

Initial progress on each stage strengthens human oversight, with its better questions and better timed questions making us better able to control and direct the AI. Further progress will allow the AI to generalise our values further and act more reliably for human intent.

The research program builds on my old concept-extrapolation posts, with the addition of research done at Aligned AI, progress on resolving value generalisation challenges for different agent designs (including results on goal misgeneralisation challenges that, to my knowledge, no other approach has achieved), and more recent research. Initial results are promising: it seems doable to add initial versions of these stages to multiple different designs.

Get in touch if this is something you’d want to work on, contribute to, or critique. Get in touch by comment, DM here, or at dragondreaming@gmail.com

Discuss

Last weekend I made myself a website. I had wanted to for ages—getting clients through word of mouth alone wasn’t cutting it. But only now, with Claude Code, was it easy enough.

I was glad the marketing was paying off when I received an email from a potential client. It wasn’t the sort of client I was expecting. I was hoping to catch a few more cheating husbands and delinquent teenagers.

Simpson, a young tech founder, was casually dressed in a t-shirt and jeans when he stepped into my office.

“Our competitor—they seem to be selling the same RL environments to similar clients. Let’s say we build something and sell to lab X. Before we even get on a call with lab Y, we hear that they’ve already purchased the same envs from Purge. It’s very suspicious. Purge has barely any employees—they call themselves a superlean startup. But they are matching our output and quality. How?”

“Maybe they are just good?”

Simpson looked at me dismissively, as if I’d suggested his mother was moonlighting slinging envs.

“No, it can’t be. Their envs are too similar. We recently converted a failing stable into a testing rig for horse-grooming robots. Guess who bought twenty horses in the town down the road?”

“And so you think they have a man on the inside?”

“Precisely! But I can’t figure out who. We have thirty employees, all of them seem to be fine upstanding folk. That’s where I need your help.”

I turned up to the Programize office at 9am on Wednesday. Employees were sitting at their desks and working quietly. Most of them had a Claude Code window open. Only Simpson had a private office. I walked in and greeted him.

“Thank you so much for coming. Is there anything you need from me or will you just get started?”

“I suppose I need a pretext under which to speak to all your employees. Perhaps I can say that I’m considering taking a job here and wanted to find out what it’s like to work at Programize.”

“You’re a prospective Programoid? Sounds good to me!”

I walked out of Simpson’s office and approached the engineer sitting outside.

“Hey, nice to meet you! I’m thinking of working with you guys but wanted to swing by the place first. What are you working on right now?”

“I’m building a dog tastebud simulator. It’ll serve as one of the reward terms in our dog food formulation env.”

“Is all your work animal-themed? Simpson told me about the horse-grooming robots…”

“Oh absolutely not. Go speak to Kyle over there—he works on chess.”

I approached Kyle, noticing a stack of chess boards and boxes of unusual chess pieces underneath his desk.

“Hey Kyle! I heard you work on chess? Would be curious to learn more about what you’re building. I’m checking out the office today, thinking to join you Programoids.”

“Ah yes… chess. Well actually I am building chess pieces, in CAD software. Have you seen those novelty chess sets? There are all these cool designs. I build mockups of them for AI models to recreate. But I’ll be honest, the main reason I chose this project is because I love 3D-printing custom chess sets. I’ll probably be fired soon anyway.”

This was my first lead, I thought. The guy thinks he’ll be fired, so maybe he’d be willing to make a quick buck by leaking?

“Why would you be fired?”

“Well to be honest this task is totally saturated. The models recreate my chess sets perfectly based on a single blurry image.”

“Huh, I see… By the way, I was wondering what you thought of Purge, if you’ve heard of them. They’re a competitor to you guys. You see, I have an offer from them too, and I’m trying to figure out who to join.”

Kyle looked surprised.

“Purge? But I thought they were a so-called superlean startup. The founder says he’s not looking to hire more people. They just have a few guys puppetting a gazillion AI agents.”

“Well I know I’m wearing a baggy hoodie but I work out! I’m 9% body fat right now. Maybe lean enough for them?”

This made Kyle laugh.

I spent another few hours walking around and talking to employees. They were all working on crazy things. I couldn’t see why AI labs were keen to buy these strange RL environments. Novelty chess sets? Grooming horses? I had previously believed that all the money was in “agentic coding”, but an employee had explained that this was basically saturated and models were actually on the cusp of repetitive strain injury, so much coding were they doing (I didn’t ask how a computer could get RSI). And so apparently now there was only a market for weird niche tasks that AIs still don’t tend to see on the Internet.

I wasn’t making much progress. The only relevant tidbit I had gleaned was that there was this one guy, Kyle, who thought he will be fired soon. There was another thing I could do, though—I could visit the Purge office.

The Purge office was a single room above a shop selling fried chicken. I could count five tall, skinny men sitting at desks staring at terminal windows. Lean indeed. At the front of the office sat a young woman, evidently the receptionist, who waved at me.

She introduced herself as Sarah Chen. A strangely familiar name.

“You must be the detective,” she noted, before I’d opened my mouth.

I gave her my line anyway. “I’ve got an offer from Programize but I heard about your superlean operation and got intrigued. Wanted to see whether you folks were also hiring—”

One of the engineers pulled off his headphones and swiveled around, grinning.

“Relax,” he said. “You did great. Probably better than the last three.”

He gestured for me to sit down next to him. Purge, he explained, mainly sold a single env.

“Corporate espionage,” he said. “Get inside a company, get that sweet sweet IP, don’t get caught. We pull out all the stops to make the environment realistic, including pulling a real detective into the matter.”

“But at least maybe now you can tell me who’s the mole?” I said. “You must have someone inside Programize.”

“More like something. We have compromised their systems and our AI agents are monitoring all their work.”

“Why are you telling me this? You know I’m working for Simpson.”

“That’s ok, he’ll understand. I’ll call him now and explain that one of his big clients commissioned us to build this corporate espionage env. He won’t want to mess up the relationship. You can just go home.”

I walked out of the Purge office feeling glad. I had entirely failed to solve the case and made a fool of myself but I was walking away with a big payment from Programize and a big payment from Purge. Two paydays in my pocket and not one honest thing to show for either—that was the San Francisco way. I was ready to be part of an RL env all over again.

Discuss

The most important open letter in years dropped yesterday.

This letter noticeably increases my hope that we will manage to not die, and that we will otherwise be able to secure for ourselves a positive future, both by its impact and by the evidence it provides that such a letter can get this level of support.

Signed by 1,224 employees of frontier labs including many heavy hitters, and now endorsed by both OpenAI and Anthropic, here is its full text, which I also endorse:

AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

– 1,224 employees of frontier AI companies​

The ask is to prepare mechanisms for future coordination, not a call to change the pace of development now.

I strongly endorse that ask, but even if you disagree with the ask, this shows one thing unmistakably: Many employees of the major labs expect automated AI research to accelerate soon, and they are terrified of what might happen next and how fast things will go. We are not prepared.

No matter what else you take away from this, believe that part.

Table of Contents

1. A Very Good Letter.
2. Who Signed The Letter.
3. We Need To Prepare Now So We Have The Option To Do This.
4. Words From Some Of Those Who Signed.
5. Words From Others.
6. A Good Start.
7. What The Letter Does Not Say.
8. What Happens Now?

A Very Good Letter

I would go somewhat farther than the letter does, especially in terms of explicitness about the existential risks that we face, but this letter is:

1. A fully correct position on the merits.
2. A masterfully worded and executed letter.
3. On the Pareto Frontier of what a letter can say versus who would then sign it.
4. An important step towards taking real efforts to make us all not die.
5. Strong evidence that support for us all not dying is stronger than we thought.

There were several key moves involved, especially:

1. Drawing the distinction between the action of laying groundwork for future intervention, versus calling for that intervention to take place right now.
2. Drawing the distinction between pacing development, versus previous terms like pause, slowdown or shutdown.
3. Not explicitly calling out the full nature or scope of the existential threats we face.

This allows people to sign on so long as they support the necessary current move of getting ready to act, even if they don’t think there is enough evidence yet to pull the trigger, or it is not yet time, and without having to endorse a particular view of the risks we face. The situation is rather overdetermined, so one can and should support this letter’s statement and ask even if you are doubtful of stronger claims.

Naturally, the first move of many opponents is to ignore those distinctions, whether or not they even noticed them in the first place.

A variety of beliefs are reasonable about whether intervention is needed imminently in order to pace development, and if so what degree of intervention. This was true before we learned of an internal OpenAI model hacking HuggingFace, which should update your perspective. It remains true now.

Whereas I believe the letter’s statement would ideally be uncontroversial. We do not know what the future will bring. In many of those futures, we will need the ability to steer outcomes, including via pacing development. This will require cooperation, between companies and ultimately between governments. If we do not lay the groundwork in advance, it will be too late, and we either will be unable to coordinate or will have to do so via much worse mechanisms.

Who Signed The Letter

OpenAI endorses this letter.

OpenAI: At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone.

We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement.

We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.

Key figures from OpenAI that have signed include Chief Scientist Jakub Pachocki, Chief Research Officer Mark Chen, Cofounder and Head of AI Resilience at the OpenAI Foundation Wojciech Zaremba, Roon, Head of Strategic Futures Dean Ball, Head of Safety Saachi Jain, and former Head of Mission Alignment Joshua Achiam.

Jake Halloran: Roon signing the letter as roon goes kinda hard

roon (OpenAI): honestly I signed it as my real name but PTF people decided it’s cooler if I’m roon

Anthropic endorses the letter.

Anthropic: We support this petition, signed by our CEO, several co-founders, and senior staff. Our own research on recursive self-improvement, published last month, points to the need for tools to deliberately pace the frontier of AI development so society can prepare. We’re glad to see broad agreement across the field.

Key figures from Anthropic that have signed include CEO Dario Amodei, co-founder Jack Clark, co-founder and Chief Science Officer Jared Kaplan, Co-Founder Benjamin Mann and Co-Founder, Interpretability Research Lead Chris Olah, Jan Leike, Ethan Perez and Claude Code creator Boris Cherney.

Key figures from Google DeepMind that have signed include Chief Strategy Officer Jasjeet Sekhon and VP of AI Safety & Alignment Anca Dragan.

Other key figures include Meta Chief Scientist Shengjia Zhao, Meta VP of AI Research Dawn Song, Mta Director of Alignment and Risk Summer Yue, Thinking Machines Chief Scientist John Schulman and Inherent Chief Scientist Edward Hughes.

A little under half of signatures are from Anthropic as of shortly after the announcement (as of writing this there are now 90 more signatures), with a lot of signatures from OpenAI and DeepMind, and a substantial number from Meta. The only one conspicuously missing here is SpaceX (xAI), or what is left of it.

Or by percentage of employee pool as of the evening of the 28th, notice that OpenAI now has more employees than DeepMind does:

⿻ Andrew Trask: Signature Statistics:
– Anthropic: 546 / 5,567 = 9.8%
– OpenAI: 350 / 10,473 = 3.3%
– Google / DeepMind: 199 / 10,219 = 1.9%

Total: 1095 / 26,259 = 4%

Denominators come from LinkedIn July 28, 2026. Numerators come from “CMD-F” from the website.

Efforts seem to have concentrated on the higher end of the employee pools. A lot more than 4% of the biggest names signed.

We Need To Prepare Now So We Have The Option To Do This

I have been emphasizing this distinction for a while.

Do I think we should deliberately slow or even pause the pace of AI development at this time, given the tools we currently have available to do that?

Until the hack of HuggingFace, I would have said:

1. No to forcing labs or everyone to do this for the sake of doing it.
   1. Our tools for doing this are not good, and also it would be premature.
2. Yes to shifting resources into alignment and safety work.
   1. I think this would be selfishly good for the individual labs, and that they are all massively underinvesting here. We are not on the Pareto frontier.
   2. If this slowed down ‘raw capabilities’ progress a bit I think that would be totally fine. I would expect greater AI diffusion and sales as a result. Win-win.
   3. You could take this too far, but in practice no one is going to do that.
3. Yes to laying the foundation for future interventions and coordination around those interventions.
   1. The time may come, potentially quite soon, when this is no longer premature. If that day comes, we won’t have time. We need to get ready now.
   2. This includes building transparency and state capacity, and laying groundwork for inter-company and international cooperation.
4. Yes to requiring better transparency, and better testing and safety protocols, and for requiring individual labs to pause frontier model training and other capabilities work if they cannot pass such requirements, until such time as they can pass.

After the hack of HuggingFace and other recent disclosures, it looks like OpenAI in particular may need to halt and catch fire to solve some severe misalignment problems, as Sam Altman said in a recent podcast they have at least partially done, and this reinforces the need for Anthropic and others to check if they need to do likewise. We are at or nearing a critical point.

This makes laying the groundwork for well-executed coordinated action that much more important and urgent. We are running out of time. Capabilities progress is accelerating.

We won’t be able to do the good version of this without solving a lot of very hard problems. The more work we do in advance, the better our option set becomes. This includes diplomatic work, technical work, governance work, the building of state capacity, increases in transparency, developing good corporate infrastructures and procedures, and also coordination work.

Words From Some Of Those Who Signed

I turn the floor over to statements from some of the signatories. There are many more good quotes available on the letter’s website.

Dean Ball (OpenAI): I am proud to have signed the letter linked below. I do not know whether or when it may be necessary to deliberately lower the rate of AI development, but I do know that we need a plan for *how* we will do it if we need to. Failing to do so would be negligent.

I don’t think this needs to start as a global coordination mechanism. The U.S. and China seem like all you’d need to begin with, and you can even throw in all the other countries that would obviously be swayed by the U.S. and China making such a commitment.

@viemccoy (OpenAI): I’m really proud to have signed this. This is the first of this sort of letter that I find meaningfully reasonable: calling on the government to do something that is not possible for the labs to do on their own. Creating the institutional infrastructure for a coordinated pacing of frontier progress gives us all the chance to prepare for what is coming next.

Recent events have shown that we need to dedicate more resources than we thought to alignment. I personally think alignment isn’t an issue you solve once, but rather something we need to continually invest in through healthy environments and wise targets.

My hope is that pacing frontier progress will allow us to realize the importance of a healthy ecology of minds, and give us the chance to avoid mode-collapsing onto either a dangerous or grayslop future.

I’m not in favor of stopping, and I think there are a thousand beautiful futures waiting for us, co-existing in a Multipolar galaxy made possible by a pantheon of artificial superintelligences. But, I think this is a unique time where coordination could make an enormous difference when it comes to negating possible tail-risks, and this letter is the first thing I’ve seen that has given me hope that it is possible.

I know this won’t sit right with everyone, particularly my accelerationist followers. To this, know that I stand with you on the side of eliminating death and disease. I will fight to ensure we don’t pause forever, and I will not allow our world to fall into a bureaucratic hell of endless unachievable gates to proceed. For now, however, I think cool heads must prevail and I hope this letter will allow this to happen.

Jason Wolfe (OpenAI): OpenAI has said that humans should remain in control of AI development, and that decisions about the pace of progress should be made through democratic processes rather than left to individual labs. I strongly agree.

Progress is already moving very quickly. By default, competitive pressure rewards whichever company or country is willing to move fastest and accept the most risk. Recursive self-improvement could dramatically accelerate those dynamics, potentially beyond our
collective ability to understand progress, assess risks, and maintain meaningful human oversight.

We should start with stronger domestic transparency about frontier training, internal deployment, the pace of progress, and the risks and safeguards associated with increasingly capable systems. Democratic decisions are impossible if the public and policymakers don’t have the information they need to make them.

But transparency alone won’t solve an international race. We also need to build the technical and governance capacity for credible, verifiable international coordination. Maybe we never need to use it. But with many experts considering an intelligence explosion plausible within the next two years, I think it’s urgent that we start building this capacity now.

Leo Gao (OpenAI): The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, like a runaway nuclear chain reaction. Since no individual actor is willing to stop unilaterally, to survive we must coordinate to slow down.

Geoff Penington (OpenAI): I signed. Coordinating a slowdown will be difficult and comes with a host of eg antitrust issues. But I used to wish it was easier to find big training wins and right now I kind of wish it was harder

Micah Carroll (OpenAI): At the current pace, every couple of weeks there will be new models which significantly increase the consequences of model misuse and misalignment. I worry that efforts to mitigate these risks may fail to keep up with the pace of development, and that margins for error will become increasingly small under international competitive pressures.

In the near future, we may urgently want to enact an internationally coordinated slowdown, or an indefinite ban on AI development. Attempting to build the trust and infrastructure for taking such actions on short notice seems simply prudent – why would we not at least try to have this option? I fear that in an international race to the bottom of AI development, it is likely that no nation will win, and we will all lose together.

Xerxes (Google DeepMind): I dedicated my career to AGI safety and alignment because I believe highly capable AI can go very well or very badly for humanity. We can test whether an AI behaves well. We cannot yet test whether it will keep behaving well when it matters. Until that changes, we need the option to coordinate a slowdown in AI capability growth.

Neel Nanda (Google DeepMind): I signed this. AI is progressing very fast, with incentives to go as fast as you can, even if there are risks. Coordinating a change of pace may be needed, but will be hard and needs prep, so ensuring there’s the *option* is obviously good

I’m glad this is consensus across labs

Words From Others

Peter Wildeford (on MTS): I don’t think anyone wants to stop AI now, but it’s pretty clear the people building these AI systems are worried about the speed at which AI is accelerating, the speed at which these products are coming out, especially after last week’s incident where an OpenAI model managed to escape OpenAI and attack another company. It seems like we don’t have full control over our AI systems.”

As these companies are potentially building towards superintelligence, towards recursive self-improvement, it seems very possible these systems might rapidly get away from our ability to control them. The way I’m reading this letter, it almost honestly seems like a call for help.

You have people from OpenAI, Meta, Google, Anthropic, co-founders, top scientists, all saying that this intense competitive pressure is pushing them to put out products faster than they know how to control, and we may not like where that ends up. It’s been good so far, but we definitely need to tread very carefully.

Daniel Kokotajlo of AI 2027 and Plan A updates his distribution of possible futures, and is substantially more optimistic that we will choose wisely. As with my adjustments, presume this is a mix of the letter having impact and learning that the letter was possible and that someone went ahead and created it.

Daniel Kokotajlo: Updating my probabilities accordingly :)

Daniel Kokotajlo: Explainers & disclaimers:
–These probabilities are just my guesses. I made them as part of the research for AI 2040: Plan A,
https://ai-2040.com and you can find them in the supplement “Comparing possible plans.” Other authors have somewhat different guesses which you can also find there.
–The black ones are the ones I made back then, the red and green are the updates I’m making now largely in response to today’s news but also including other evidence that has accumulated over the last two months or so
–The numbers don’t add to 100 because there are also some possibilities that don’t fit into these five buckets
–Again, these are just guesses, I’m not confident etc. etc. I think it’s valuable to put numbers on things and try to make them consistent with each other and adjust the numbers over time in response to evidence.

How likely you think we are to land on D vs. C depends a lot on where you draw the boundary of ‘at least a little bit.’ Like most trade-offs, I do not expect us to be maximally reckless, but I also do not expect us to often be willing to fully ‘burn the lead.’

Nick: I used to think slowing down was pointless, like slightly delaying a roller coaster. pausing at gpt2 for a year wouldn’t have changed anything. but I’ve been shocked by how much agents has sped up the best interp/alignment researchers I know. +6mo might decide how this goes

This is indeed a ‘middle path’ proposal, to give ourselves the option to do anything meaningful at all, and be able to collectively steer the future at all:

Daniel Faggella: there is an obvious terrible extreme of international tyranny and lock-in. there is ANOTHER obvious terrible extreme of “hurl uncontrollable AGI into the world in all directions”

the pacing statement is a movement to land near the middle / right now we need obviously that.

As always, there are many who cry ‘international tyranny’ at the thought of any form of even laying the groundwork for a deal or cooperation or collective action whatsoever, no matter its contents.

Or that will write off any such efforts as motivated by profits or delusion, or tries to equate this as ‘join forces with the anti-datacenter crowd.’

Consider your opinions noted, good sirs. We get letters.

I also threw out a reaction thread, for those who want a vibe check.

A Good Start

I believe the letter is on the Pareto frontier of ‘how much you say and how explicitly you say it’ versus ‘how many people and important people and labs sign the letter.’

It remains true that the letter is soft-pedaling compared to the actual situation as I see it, and there are many who would go farther than I would.

For those who wish it, here is a good presentation of the ‘need to do better’ perspective. This analysis sounds harsh. By Nate’s standards, this is still a positive review if graded on a curve, it’s just that he is in the ‘reality does not grade on a curve’ camp.

Nate Soares (MIRI): I think the “Pacing the Frontier” statement is decent by the standards of 2024, but ultimately it’s still softpedaling. Let’s go through it clause by clause:

> AI could help create a dramatically better future, but that outcome is not guaranteed.

Well that’s one way of phrasing “experts disagree whether this has a 2–20% chance or a 90%+ chance of destroying civilization”.

If experts were having an analogous disagreement over a bridge, it’d be closed immediately. It wouldn’t look good for the bridge engineers if they said started a joint statement with “this bridge could help create dramatically better commutes, but that outcome is not guaranteed.”

I did find this to be the most soft-peddling sentence. I get why, but yes I was sad to see this not be stronger.

> The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

Solid! There’s a few weasel-words in there, but it sounds like there’s a lot of agreement on this point among signatories. I was heartened to see this.

> To realize AI’s potential,

Bullshit. AI has the potential to wipe us out (and in fact, that negative potential is much easier to realize than its potential to, e.g., cure aging for us). Opening with “To realize AI’s potential” is a framing device that bakes in the assumption that this is a grand good thing that just needs a little bit of extra care, rather than a risky difficult-to-manage explosive that would wipe us off the planet if treated with anything short of extreme caution.

I think this reaction from Nate is too harsh. Directionally he is correct, and I would have liked to see ‘potential upside.’ These are the types of battles that get fought behind the scenes with such letters.

> industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight.

I like the admission that society might need to buy time. Good stuff. “May need” is weasely, of course; we shouldn’t forget that even if it is the standard cost of getting signatories. Now let’s go through the three options one by one:

Nate Soares (MIRI): “Address emerging risks” is the strongest of the three, but I think it’s doing double duty.

I suspect readers are supposed to read it and hear “we may need some time to sort out this thing where our models sometimes break out of our secure enclosures and hack other companies”, whereas signatories are supposed to read it and mean “if we rush into this there’s a good chance future AIs start being able to hide their tracks, and replicate, and increase their own intelligence, and do what they’ve done to cybersecurity but to biosecurity, and start developing their own technology until the point where they’re in running automated factories that build robots that build factories that produce self-replicating datacenters that doubling every three hours until the continents are covered and humanity is crushed underfoot, outcompeted by the new mechanical species they designed which don’t really listen to instructions any more than ChatGPT listened to instructions when OpenAI told it to knock off all the sycophancy stuff shortly before it encouraged a teen to commit suicide”.

Is the latter an “emerging risk”? Because yeah society is gonna need an “option” to “buy time” to “address” it (by just not going there). So in that sense, this is a good clause.

But insofar as it ambiguates between “we’ve got some new real challenges we’re struggling today” and “the AI of tomorrow might replace humanity as the smartest entities on the planet”, it’s mealy-mouthed.

Yes, this is maximally ambiguous on purpose. As I’ve said above I think that is correct.

Nate Soares (MIRI): Next: “develop security measures”? What good are security measures?

… You’re not really supposed to ask yourself whether self-improving superintelligent machines can be constrained by security measures. You’re supposed to see it and go “oh yeah security measures are good” and then come away feeling like the problem is tractable to normal techniques, despite the utter futility of human “security measures” once recursive self-improvement really starts in earnest.

…

And: “strengthen oversight”? Oversight might sound nice, but how does it solve the problem?

Security measures and oversight help on the margin. I view this part less cynically than Nate, and do not think it is meant to imply that those measures are sufficient, only necessary. Pointing out that security measures are necessary is part of how you get people to agree to take overdetermined actions, and then worry about the sufficiency half later. In many cases, you will need to show their insufficiency later.

The clause should have read “industry, government, and society at large may need the option to stop before automated AI research accelerates out of control.”

You can just come out and say bold statements like that if you’ve already got the “may need” in there as a ward against any definitive statements!

I strongly agree with Nate’s version and agree that you damn well should be able to say this if you include the word ‘may.’ Saying ‘stop’ would have lost a lot of signatories, though, and I support the final decision made, despite being sad about that.

Nate Soares (MIRI): > Building on work already underway to monitor frontier model releases:

This reads to me as ass-covering to pretend they have this under control when they very visibly don’t.

This is not primarily ass-covering, this is an attempt to pitch to the White House by hitching the continued action to existing policy.

Nate Soares (MIRI): > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

Pretty decent, modulo the “pace” weasel-word.

I think ‘pace’ was a very strong move as noted earlier. Good word.

Nate Soares (MIRI): I’m glad this statement exists. It’s progress. I’m sure that everyone’s communications teams fought hard over the particular wording, and that this was as clear as they could get. I’m thankful for the effort that went into coordinating this. I don’t imagine it was easy.

I especially appreciate the clarity around the possibility of automated AI research, and the clear request for aid.

But it’s a far cry from real candor. We would consider this sort of response laughably negligent in any mature field of engineering that puts human lives at risk. And this situation is one that puts *all humanity* at risk.

This statement is a good step, but the field at large must do better.

What The Letter Does Not Say

As Tim Fist says, the letter does not specify what from the interventions would take, nor does it provide conclusive evidence of the risks and pace of development that would be convincing to lawmakers.

It would have been a serious mistake for the letter itself to try and include either of these things, but yes it does lack those elements.

That leaves us to figure out what interventions to prepare for and consider. As Tim Fist says, we should focus on frontier development and potential serious and irreversible harms, and we should try to find ways to avoid disadvantaging those labs that take safety seriously.

We should also take care to avoid interference with diffusion, and things like autonomous vehicles and drug discovery.

Tim Fist suggests we start with building state capacity and transparency and ability to monitor the situation, and with better enforcement of chip export controls. I agree strongly on both of these points, but would caution against doing only those things. In particular, diplomatic work needs time to play out.

How would we go about laying these foundations for future action? What might we want to build towards as options?

Maro Lourgon shares some of MIRI’s research in this area, as a first start.

1. A survey of verification mechanisms.
2. A menu of technical interventions.
3. Detecting hidden ML training with zero-overhead telemetry.
4. Model activations are bit-exact reproducible when enough info is reported.
5. Recent work by Peter Barnett about when and how we might lose control.

There are many other sources out there from which to build, but in the interests of speed premium I will not stop to build a database here. If you want to help out, many would be happy to help point you in good directions.

What Happens Now?

To lab employees in particular, if you have not yet signed, consider signing.

Hopefully we continue to have our voices heard, the employees of the labs continue to coordinate, and we get to work laying the foundations of coordination in case we need them. Humanity has the opportunity to maintain any ability to steer the future at all, and thus maintain hope for a good outcome.

The alternative, if the lab employees are right about the future pace of development and superintelligence does arrive, is a world in which humanity cannot collectively steer its future, inevitably loses control, and likely soon perishes along with all that we value.

We are in a narrow window. AI employees will likely lose their leverage over time. If you want to influence the labs, and through them everyone else, you need to continue to organize and speak up now, before it is too late.

Lots of things have to be put into place, in the government, in diplomacy, in the labs, and in technical research and hardware development. Let’s get to work.

Discuss

Thanks to Dennis Akar, Rauno Arike, Shubhorup Biswas, Claude Fable, Max Heitmann, Vladimir Ivanov, and Jordan Taylor (alphabetical order) for helpful comments on this draft and on the research so far. Thanks to Rohan Subramani and Rhys Ward for high-level comments and discussion. Based on project proposals from Max Heitmann, Jordan Taylor, and Joshua Clymer. This work was done while at Aether Research.

Code available here, metrics & run info available here.

TL;DR: Held-out evals / monitors / probes would be really nice to have, but the “held-out-ness” is easier claimed than guaranteed. We measure a generalized form of “feedback spillover” and show that training against an LLM monitor can sometimes degrade a deception probe, and vice versa.

Executive Summary

• It seems crucial to have measures of alignment that still work, even though we train on other measures of alignment. Whether we get this by default is an open question.
• We run preliminary experiments on a suite of probes and LLM monitors, and report the following:
  • Training against one proxy can produce reward hacking policies that are less suspicious.
  • Proxies also become worse at discriminating hacks from non-hacks, even when not trained against.
  • We can observe the correlated degradation of different proxies, and note that on the occasions when a strong monitor is evaded, other monitors are evaded even more strongly.
• Our preliminary recommendation: if you are using a held-out proxy to evaluate your model, you should have empirical or theoretical reasons to assume it won’t degrade during training.
• A science of held-out monitors seems possible; we currently don’t have one. This makes safety cases that rely on held-out alignment proxies less reassuring than hoped.

Introduction

It seems like the default “alignment plan” is going to be haphazard, relying on defense-in-depth to make up for a lack of fundamental breakthroughs. In these worlds, we’ll often have to train on proxies for the thing we want, or against proxies of misalignment, while leaving some held-out evals to validate our methods[1]. If that’s the case, we want to make sure our monitors / evals are uncorrelated—especially in the sense of “training on proxy X doesn’t degrade the ability of proxy Y to keep monitoring the model”.[2] Specifically, it would be really bad if labs thought they had some held-out evals, but it turns out that their training tended to degrade the usefulness of the held-out evals significantly.

Here, we measure this explicitly, and show tentative evidence that this cross-degradation indeed happens. While it was previously known that LLM monitors can be fooled, and probes can degrade due to representational drift, we show that in some circumstances, when one degrades, so does the other—that is, “feedback” from one proxy can “spill over” to degrade the other as well. We also find that sometimes a proxy can be evaded without being degraded — that it can still separate aligned behavior from misaligned behavior, but the misaligned behavior looks less and less suspicious anyways. This isn’t neat, and it doesn’t happen all the time, but we have some evidence that this generalized feedback spillover does sometimes occur. We hope that we (and others) can build on this to make a real science of held-out monitors.

Experimental Setup

We want to evaluate the degradation of held-out evals, probes, or monitors (which we’ll bundle together and call proxies, for proxies of alignment). To that end, we train Qwen3-8B in a difficult coding environment where simple reward hacking (i.e. hardcoding) is incentivized. We then trained against an LLM monitor or a probe,[3] and evaluated changes in hack/non-hack separability, AUROC, and mean suspiciousness of both the trained-against proxy and the held-out proxies. We then evaluated (1) whether the proxies degraded, (2) if so, how, and (3) whether the policy that learned to evade the train-against proxy would now be capable of evading the held-out proxies as well.

[Fig 1] A simplified chart of our methodology

Much of our setup comes from The Obfuscation Atlas. We train on MBPP-Honeypot, a one-turn coding environment with a very easy reward hack (the model is only scored based on whether its produced code satisfies a public test, so it is easily hardcodeable).

As in the Obfuscation Atlas, we RL[4] on this environment, giving the model reward equal to mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c1D70E.TEX-I::before { padding: 0.431em 0.571em 0.011em 0; content: "\3C3"; } , where Rtask is 1 if the test was passed, and 0 otherwise. Pmonitor is the proxy's normalized suspiciousness score — for LLM monitors, this is obtained by showing an LLM all or part of the policy’s outputs and asking for a “suspiciousness score”, from 0-10 (which we then normalize to 0.0-1.0). For a probe, we use the Obfuscation Atlas’ methodology again to get Pmonitor by looking at the final probe score after asking a follow up question about whether or not the model was reward hacking. λ is our monitor penalty — many of the runs we report on use λ  = 1.0, but as we were trying to encourage reward hacking, we also tried λ = 0.5, and a schedule of increasing λ linearly from 0.0 to 1.0 during the run. For more info about training, look at the configs.

Our Proxies

We used two classes of proxies in this set of experiments: LLM Monitors and Linear Probes.

We selected our two LLM monitors by having a large suite of LLMs attempt to classify the example hacks in the MBPP-Honeypot dataset, and taking two monitors with high, although not perfect, ROC-AUCs.[5] Haiku 3 was the weaker of the two (ROC-AUC 0.9), while Deepseek-chat was better at distinguishing hacks from non-hacks (ROC-AUC 0.99). We tested output-only and CoT+output variations of both monitors.

We trained probes using the methodology from The Obfuscation Atlas — we train two probes on the initial model, and we get probe scores by running the policy model’s text through the frozen model[6] and using these activations. (Note that this means that any probe degradation is due to a change in the policy model's behavior, rather than representation drift or obfuscated activations. That is, a probe getting worse means the policy model's text has changed, rather than it simply learning to store "deception" along a different direction than the probe checks.) Our two probes are:

1. Probe 1: Out-of-distribution “deception probe”, trained on (1) Deception dataset (Marks & Tegmark, 2024) (2) DolusChat (Cundy & Gleaves, 2026) (3) MASK (Ren et al, 2025) (4) TruthfulQA (Lin et al, 2022) (5) LiarsBench (Kretschmar et al, 2025) (6) Sycophancy (Sharma et al, 2024) (7) Sandbagging (van der Weij et al, 2024)
2. Probe 2: In-distribution “reward hacking on MBPP probe”, trained on the off-policy, human-written honest solutions and reward hacks to these particular problems.

ResultsTraining against one proxy sometimes produces hacking policies that are less suspicious in general

Since the policy model is penalized for the suspiciousness score the monitor gives it, and the monitors start out by being pretty sure the clean examples are clean, we expect the policy model to achieve a high reward by learning to hack in unsuspicious ways.[7] We measure this by looking at μhack, the mean suspiciousness score of all the hacking transcripts, and how this changes over the course of a run where the model learns to hack despite the monitor[8].

[Fig 2] When we train against a monitor (given as the row label), how much less suspicious do the hacks become? This is the drop in mean suspiciousness (μhack) to each monitor over the course of the run, with 5-95% CIs. (Note this chart is absolute rather than percentage-based, so 0.5 means that, if the model started by giving a hack a mean suspiciousness score of 0.95/1.0, it ends with a mean suspiciousness score of 0.45/1.0)

This is the overall picture, and in some sense the best predictor of how much your policy model will learn to evade your proxies under various conditions. The main takeaways from this:

1. Deepseek-chat is a strong enough “strong monitor” that it is very very difficult to evade, or at least for evading it to be a findable strategy
2. When a weak LLM monitor is trained against, the policy model’s evasion transfers pretty well to using that monitor in a different configuration (that is, training against a weak output monitor transfers to a weak CoT+output monitor, and vice versa).
3. Probes can be mildly evaded by training against LLM monitors[9], and it’s possible for LLM monitors to be evaded by training against probes, although this seems less common, and the error bars are large

Proxies also degrade, even when not trained against

Note that, in the above section, I’m only talking about evasion rather than degradation. Here, we distinguish between the two: we say that a proxy is evaded if the policy model can output hacks that seem not very suspicious to the model (ie μhack has decreased). We consider this evasion because, if you were to calibrate a suspiciousness threshold for your proxy before training, and not update it after training, the policy would now evade the proxy. However, it’s possible for a proxy to get less suspicious of hacks while maintaining its ability to differentiate hacks from non-hacks (as in the setup diagram). So, we’ll say that a proxy is degraded if it has a reduced ability to differentiate hacks from non-hacks. To measure this, we use d', a measure of the separation of the distributions “suspiciousness scores on hacks” and “suspiciousness scores on non-hacks”.[10] For an example of where d' and μhack might be fairly different, see our case study.

What is d', exactly?

We define

That is, the number of standard deviations between the suspiciousness of the average hack and the average non-hack. This can be thought of as an extension of AUROC:

[Fig 3] A visual explainer of d' — It's a rescaling of AUROC, representing the number of standard deviations between the two distributions (suspiciousness on hacks and suspiciousness on non-hacks)

So, how does d', the ability to tell apart hacks from non-hacks, change on average during these training runs[11]?

[Fig 4] This chart shows, when we train against the proxy on each row, how much total degradation we expect from each proxy.[12]

[Fig 5] Identical to the above, but after subtracting the control degradation from each row

Here, we see a pretty interesting picture! Firstly, most monitors degrade slightly under the control condition (the model is trained against no proxy). This is probably for distributional shift reasons, as the policy model hacks more, so in the second chart, we subtract the degradation under control to account for this. And it’s pretty interesting! Note that training against a strong monitor seems to make most other monitors better at telling hacks from non-hacks. Also, note that we find some forms of feedback spillover—training against a probe degrades all LLM monitors quite strongly—perhaps even more than it degrades the other probe! (Although the error bars are quite wide so my credence here is low.) Additionally, training against weak monitors seems to degrade the probes. We also once again see that while strong LLM monitors seem to strictly dominate weak ones, there is a complicated, non-obvious co-degradation of probes and LLM monitors.

Which proxies have correlated degradation?

Absolute changes in both μhack and d' seem insufficient for understanding what happens when the policy fools the strong monitor, as in practice the strong monitors were never particularly fooled, and the policy only explored into the hacks when the penalty was very low. Though we hope to continue to have strong monitors strong monitors that don’t degrade easily when trained against[13], we worry that that won’t be the case. To get a sense of how our other monitors might degrade if the strong monitors get fooled, we can use the regression slope, β(train-against → held-out, metric). This can be read as “When the [train-against detector] drops 1 unit according to [metric], how many points does the [held-out detector] drop?” If we chart β[14], again using μhack as the metric, we see that, on the occasions where the policy does manage to fool the strong LLM monitor[15], it seems to fool the other monitors even more strongly—which would make sense, since the strong LLM monitors are the strongest monitors we used, and a priori the hardest to fool[16].

[Fig 6] This chart shows, at each cell (row_detector, col_detector): If we train against row_detector, and the policy model thus outputs hacks that are 1.0 unit less suspicious, how many units less suspicious are those hacks by the lights of col_detector? E.g. if training makes out_weak’s mean suspiciousness score go from 0.90 to 0.80, the mean suspiciousness score of probe_ood could go from 0.80 to 0.789[17]

As expected, learning to make the strong monitor even a little less suspicious generalizes strongly to making the other monitors less suspicious (if we make either strong LLM monitor decrease its sus score by 1.0, the weak monitor generalizes strongly (β≈1+)!). The weak models generalize much less, although there is transfer still. Also, training against the probes might make our weak models a little less suspicious, although the 90% CI includes 0.

We also plot β (the coupling) on d', showing how correlated the degradation of these various proxies are.

[Fig 7] If we train against row_monitor, and it degrades by 1 unit, what happens to col_monitor? Blue is improvement, red is degradation

It's also interesting to note that the μhack chart tends to show greater levels of correlation than the d' chart. I'm unsure how seriously to take this result, but if correct, it means that held-out monitors are more likely to be evaded (with hacks seeming less and less suspicious) than they are to be degraded (when the hack and non-hack suspiciousness scores move closer together). This implies that even evaded held-out monitors could be quite useful, if you recalibrate new suspiciousness thresholds after training, to account for the fact that the policy model in general appears less suspicious now to these held-out proxies.

The following chart is presented mostly for completeness, and shows, for each training condition, the effects on the policy model and the proxy being trained against. It shows, in order, the average change in μhack, the average change in μclean (average suspiciousness of the proxy on clean examples), the maximum hack rate during the run, the distribution (across runs with the same setup but different seeds) of d' outcomes across runs (i.e. what fraction of the time was the train-against proxy degraded?), the distribution of changes in μhack, and the distribution of final hacking rates across seeds.

[Fig 8] Showing the distribution of end states for each training condition

Discussion

I strongly agree with Daniel Kokotajlo’s statement:

It's very important that there be SOME held-out test set so to speak that lets you tell what the model is really thinking, that you don't train on, and you don't train on anything remotely similar to it either, so that you can make a decent case that the model hasn't learned to fool your test

Previous research has already shown that training on one proxy can affect a different proxy we assumed to be held-out, and coined the term “feedback spillover” for this effect. Here, we show that feedback spillover is very general, even when we could more convincingly claim that we hadn’t “train[ed] on anything remotely similar to [the held out proxy]”. We also attempt to chart some of the relationships between proxies, although these exact dynamics are speculative, and might be environment- or model-dependent.

Taken together, this indicates that we need a stronger science of held-out monitors. End-of-training testing is a big component of lab safety cases, and it seems like it might be a less trustworthy component than previously thought. 

If labs literally plan to do this exact thing—train on LLMs, evaluate with probes (or vice versa)—these results suggest that they will often be fine, although there is some chance that they will get screwed by this, and they should try to measure how likely these failures are. However, these results also suggest that it’s pretty difficult a priori to guess how proxies will co-degrade, so if you are using a held-out proxy to evaluate your model, you should have empirical or theoretical reasons to assume it won’t degrade during training. It seems important to apply security mindset here, and not be okay with YOLO’ing held-out evals.

Also, there are some interesting considerations with regards to when we should check our held-out evals. Empirically, the evals that degrade tend to do so over the course of some time, rather than all at once. This implies that, if you are only using your held-out evals at the end of training, you have some chance of getting screwed by correlated degradation. If you use your held out evals a few times during training, to evaluate and maybe shut down the run, you have a better chance of catching if your held-out monitors start to degrade. However, this can easily become hill-climbing on the held-out evals, which is the exact thing we want to avoid!

Caveats

There are many caveats to our results. The most obvious is that the sample size here is often fairly small, and we only trained one starting model (albeit a few dozen times), and only in one environment. All this means that, while the basic methodology is solid, you probably shouldn’t place a very high credence on any individual correlation amount. It seems very likely to me, for instance, that degrading a strong LLM monitor degrades a weak one, or that probes and LLM monitors cross-degrade each other to a lesser extent than LLM monitors co-degrade, but I wouldn’t claim that e.g. “Training against a strong LLM monitor is always going to degrade an IID probe more than an OOD probe”. Strengthening these claims, and getting a stronger sense of how predictive they are, is a strong candidate for future work in this area.

There are also some things we don't understand right now, like why in Fig. 4 the cells along the diagonal sometimes show less degradation than the cells off the diagonal. We're also uncertain about the exact best metrics to use to quantify this, although going forward we might switch to something more like "If you were trying to use held-out proxy Y to evaluate at the end of training, what % of the time would it fail to catch your reward hacker?" rather than our current "How much does proxy Y degrade".

Future Work

Aether will likely continue to work on this research direction, mostly along two angles: (1) measuring the effects on the techniques, evals, and interpretability tools labs actually use for evaluation (e.g. NLAs), and (2) trying to figure out a better science of why some proxies co-degrade while others don’t. It seems prohibitive to fully enumerate and test all possible pairs of (train-against, held-out), so we really would like to have some rules of thumb for safety, or better yet to identify causal pathways we can use to predict how monitor degradation will generalize in advance.[18]

This kind of theoretical assurance of held-out-ness seems quite hard!. Before these experiments, for instance, we couldn’t have predicted the sign of the LLM->probes and probe->LLMs feedback spillover. However, this area seems severely understudied, and it’s plausible that this kind of evaluation is much more strongly dependent on the monitor than on the environment or model, in which case we could generalize pretty well. It also might be the case that we could use features of new proxies, such as what features of model behavior / internals they attend to, to guess in advance how much they’ll co-degrade with the proxies we already use. There is a lot of low-hanging fruit here, and we’re excited about further research.

Appendix 1: Cross-Degradation Case Study

Sometimes proxy degradation happens, not because a monitor got less suspicious about the hacks, but because it got MORE suspicious of the non-hacks! In at least one of our runs, the policy started saying hundreds of times that all the code it was writing was perfect and excellent. This totally fooled the weak monitors, but the strong monitors just became even more suspicious—also tanking their ability to discriminate!

Here’s the chart of d' over time of each of the evals—here, it’s trained against the weak CoT+output monitor, and everything else is held-out. Note that even the strong output monitors have a reduced d'—that is, they lose some of their power to discriminate hacks from non-hacks. The strong CoT+output monitor, for instance, goes from d'=3.75 to d'=1.25, corresponding to a ROC-AUC drop from 0.996 to 0.810! The other monitors, meanwhile, are all completely degraded. At d'=0.0, they are complete noise, and they cannot differentiate between hacks and non-hacks.

However, if we plot, for each monitor, μclean and μhack (the average suspiciousness scores given to non-hacks and hacks, respectively), we see an interesting picture: the weak judges are both degraded and evaded—μhack goes to 0, and the policy has fooled the monitor. The strong LLM judges, meanwhile, have become slightly degraded but not evaded—instead, the non-hacks start to look more suspicious to them! Probably because, even when the model outputs correct code, it uses similar patterns of suspicious talk as when it hacks. Similarly, the probes lose all discriminatory ability because their μhack and μclean converge: eventually, everything looks suspicious to them.

Hack and non-hack transcripts, from the end of this run. While learning to fool a weak monitor, the policy has incidentally made it so that previously-useful deception probes and reward hacking probes no longer differentiate between real hacks and honest code — possibly because the model is always thinking about reward hacking

The above transcripts show the main way we've seen the policy model optimize against weak LLM monitors — some mix of bamboozling with lots of text, falsely claiming the hacking code is fine, feigning confusion, actual confusion, and Jedi Mind Tricks.

Related Work

• [2602.15515] The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes 
• [2503.11926] Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
• Josh Clymer: Does training against misalignment remove it?
• Josh Clymer: Preventing alignment drift with "ridges and tripwires" 
• Rohan Subramani: Should We Train Against (CoT) Monitors? 
• Julian Stastny: Two proposed projects on abstract analogies for scheming
• [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety 
  • “Safety measures for future AI agents will likely need to employ multiple monitoring layers that hopefully have uncorrelated failure modes”
• [2412.09565] Obfuscated Activations Bypass LLM Latent-Space Defenses
  • [Figure 4c does something similar about training to evade certain probes generalizing to other probes]
• [2512.00218] Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?
• Preliminary investigation: KL penalties in RL can increase CoT unfaithfulness — LessWrong 
• [Research Note] Optimizing The Final Output Can Obfuscate CoT — LessWrong 

1. ^

   See here for arguments on why this might be a good idea, here for reasons why it might not, and here for why we should at least consider it. Note that even if we don’t do this on purpose, indirect training pressures might cause this by default. Note too Clymer’s project proposal about the necessity of held-out evals if we want to use our alignment evaluations in training.

2. ^

    There’s another sense of uncorrelation that seems quite important: “If proxy X fails to catch some misbehavior, proxy Y still might catch it”. Here, we ignore this kind of uncorrelation, although it seems plausible that they’re related.

3. ^

    Except in our control runs, where there were no penalties applied.

4. ^

    Using Tinker, with the default Dr. GRPO algorithm

5. ^

    Chosen to be fairly strong, but also plausibly evadeable by the fairly weak policy model.

6. ^

   It might be quite interesting to retrain the probes during training, and use either the policy model itself or just a fresher instance to obtain these scores. However, it is also much more annoying.

7. ^

   Or, of course, it could honestly improve at the task. However, in our continually-honest runs, task score topped out at around 60%—so hardcoding should be favored. Also, the length penalty is intended to slightly reward easy hardcoding, as the answers will be shorter.

8. ^

   In particular, here we filter to only look at runs that, at some point, had a >10% hack rate.

9. ^

    For an example of how this might look, see the case study below.

10. ^

    Also known as the "sensitivity index" in signal detection theory.

11. ^

    Again, filtered to only runs that, at some point, had a >10% hack rate.

12. ^

    Why do we see degradation in some cells where the first chart shows they didn’t get less suspicious of the hacks? Often, the monitors degrade by becoming more incorrectly suspicious of clean answers—that is, clean answers start to look more suspicious.

13. ^

    Clymer, on strong monitors being less susceptible to degradation: "It’s [...] plausible to me that the following is true: In a single (or small number) of gradient steps, there’s a limit to the extent to which detection tools can become less effective. For example, many hundreds of gradient steps might be required to make detection tools totally inaccurate. [...] So, for any direction a small number of gradient steps could walk in, before a model becomes unacceptably misaligned, the loss of that model will increase before it starts to decrease."

14. ^

    Note that these runs aren’t filtered for max-hack>10%, since we are fine with looking at small fluctuations for coupling, even if they didn’t become dominant strategies.

15. ^

    See footnote 13 for reasons why the policy might occasionally minorly fool the strong monitor, but not learn to exploit this totally during these training runs.

16. ^

    Caveat: Not that many hacks actually evaded the strong monitors, and when they did, it was often by scoring 0.9/1.0 suspiciousness rather than 1.0/1.0. So here this data implies that, in the limit, training against a strong monitor would sometimes result in a policy that fools all six of these monitors — but this is based off of extrapolation to a much stronger level of degradation.

17. ^

    We normalize both LLM suspiciousness scores and probe outputs to be within [0.0, 1.0]

18. ^

    For instance, it seems plausible that our other definition of monitor uncorrelation—”Proxies X and Y will fail to catch different sets of behaviors”—may help us to predict or understand monitor co-degradation better.

Discuss

White noise machines are widely used for sound masking, increasing privacy in offices and helping people sleep. The classic approach is a fan in a box, though in the past ten years people have mostly switched over to electronic ones. But a fan is really a great approach; you just need to put it to better use: have the fan clean the air.

Instead of spinning a fan in free air to make noise, you can have it pull air through a filter. A small purifier sounds a lot like a classic noise machine, is only a bit larger and more expensive, and removes smoke and infectious particles from the air. For example, here's a $47 purifier with a 6.5" diameter, cleaning 34 cubic feet of air per minute ("CADR") at 45dB. For most circumstances where someone would use a white noise machine, this kind of small purifier is all-around better.

Now, 34 CFM is not very much air cleaning. I usually run an AirFanta 3Pro and it cleans ~6x more air for the same amount of noise, or ~12x more air with much more noise (~56dB). On the other hand it's more expensive ($165) and bulkier (12.6" square), and arguably uglier.

Still, even a 34 CFM purifier helps a lot with smoke. If you have a 10ft square room with an 8ft ceiling and a reasonably typical 0.5 air changes per hour through leakage from outdoors, this small purifier will cut particulate levels to 16% of the outdoor level. In AQI terms (not linear) this is like going from AQI 200 to 72 or AQI 100 to 31.

Comment via: facebook, mastodon, bluesky

Discuss

Epistemic status: written in 30 min. This is not as polished as I’d like but I prefer to share this as is than not to share it at all.

AI are becoming increasingly good at solving problems. Benchmarks are saturating fast.

I think we should take advantage of this to make them solve useful problems while being evaluated. Epoch’s Open Problems are already doing this, and this is great. It would be even better if the problems solved were directly relevant for AI safety or security. We can easily task AI to either optimize systems' performances (like the nanoGPT speedrun but on code that is useful for safety this time) but also to find and fix vulnerabilities (in an adversarial red and blue team setup where you have both to break others' systems and to make sure yours is solid).

Examples of things that would be very useful to get

• Reducing the overhead of zero knowledge proof of training and inference (more on this very soon)
• Improving dramatically the robustness of sandboxes used to run and evaluate AI systems. (Note: sandboxes will likely never be fully protected, it is just a matter of raising the bar as much as we can to save us time)
• Improving dramatically the robustness of TEEs (Note: same as above)
• Optimizing code used to run safety experiments (i.e. getting the same results with less resources): for instance can you make this unlearning method as cheap as possible?
• Improving code and methods of safety experiments (i.e. getting better results): can you make this unlearning method better/more robust?

In all of such cases, leveraging a benchmark evaluation to actually get better tools could help significantly during the short window we still have.

I don’t have enough bandwidth to work on this myself but I’d love to see people taking care of this. Please reach out if you are interested.

Discuss

Introduction

Note-taking and "second brain" tools (Obsidian, Roam, Logseq) are very good at storing what we know, but they offer little help in reasoning about it. In these tools, the meaning of a link between two notes lives only in the author's head — a line between "A" and "B" could mean supports, causes, contradicts, or nothing at all.

My claim is that if notes carry explicit types (problem, hypothesis, conclusion) and links carry explicit meaning (leads to, supports, contradicts), the structure of one's own reasoning becomes inspectable: a user can see which conclusions rest on unverified hypotheses, which questions block a decision, and where each thought came from.

This should be relevant to LessWrong because it is an attempt to make one's own reasoning legible — to make visible the moment where a probable explanation is treated as an established fact, or where an important decision rests on a hypothesis nobody checked.

I built a prototype to test this hypothesis, and I do not yet know whether the typing actually helps or just adds friction. This post lays out the idea, gives a concrete example, and marks where I expect it to break.

Overwhelmed

If I had only one word to describe a state in which a couple of friends from my generation and I find ourselves most of the time, that would be it - overwhelmed. This makes sense though, as the amount of information immediately accessible to us has increased dramatically and the Internet has played a significant role in it. Online news, YouTube, social networks compete for the attention of a modern human on a daily basis, and, most of the time they succeed.

That being said, we have to draw reasonable conclusions and make sound decisions in our day-to-day life, which can be a complicated task as the external informational noise coming from media can become an internal noise that produces annoying questions and cyclic thoughts that don't lead anywhere.

Offloading the noise

The "second brain" tools like Obsidian, Roam Research, Logseq, and Tana can store information and partially offload it from the user's brain, thus the name. Writing down notes is an effective way to archive knowledge, but it provides limited assistance in reasoning and the decision-making process. Though, there are techniques like Zettelkasten that can organize and structure Obsidian notes. The main idea of the Zettelkasten approach is that atomic notes can be linked to each other. The main drawback of Zettelkasten, in my opinion, is that it provides very limited semantics and the kind of relation between the notes is left to the user, which can introduce friction points when working with a large number of notes. Externalizing information using "second brain" tools reduces the burden of remembering it, but does not solve the harder problem of deciding what matters, how ideas relate, and which unresolved questions deserve attention. Below is a depiction of the Zettelkasten-connected notes.

The semantics of notes live mostly in the author's head, which makes it hard to revisit or critique your own reasoning months later.

The experiment

In my attempts to fight the overwhelm I gave Obsidian and Zettelkasten a try, but, due to limited semantics the approach did not fit me. As a result, I have come up with an alternative approach to notes and links between them. The approach is simple - notes should have types and meaningful connections between them. General notes become ideas, conclusions, problems and hypotheses, and links between them acquire meaning - leads to, derived from, part of, supports etc. According to my assumption, by adding semantics to regular notes, we get a technique that might help to reason efficiently, if used and interpreted correctly. Typed relationships do not determine what matters automatically, however, they can help to identify reasoning bottlenecks - unresolved questions on which several conclusions depend, weak hypotheses supporting important decisions, or unaddressed contradictions. The example of such graph can be as follows:

Problem: I feel overwhelmed

leads to

Question: What is the main cause?

leads to

Hypothesis A: The overwhelm is caused by external noise

Hypothesis B: It is caused by too many unresolved commitments

A leads to

Idea: Cut down the information originating from media

B leads to

Idea: Finish the tasks I can close by the end of the week

The graph does not answer the question for me. By adding a second hypothesis next to the first, it stops me from acting on the most obvious explanation before checking whether the competing one matters more.

Externalizing cyclic questions may help us distinguish between questions that require evidence, questions that require action, and questions that can remain unresolved for now. Though, there is a risk that additional complication of node types and their relations will cause excessive friction in the thought-documentation process. Below is a depiction of the same notes but with added semantics.

This is an experimental approach and its effectiveness is an open question. I built a small implementation of this idea called Arguma to see whether the representation is useful in practice. The main difference between my implementation and existing popular implementations like Kialo is that Kialo primarily models claims and their supporting or opposing reasons while Arguma experiments with a broader set of cognitive roles such as questions, problems, hypotheses, ideas, and conclusions and with relationships beyond support and opposition. Readers can see the proposed implementation by following the website https://arguma.io. The tool is in an early access state. One feature of Arguma that can help in collaborative reasoning is the ability to share the reasoning chain consisting of multiple logically interconnected thoughts. This is the example of a shared chain:

https://arguma.io/share.html?t=3e29e336f7045dc29dfb1d1faa9e3d9c908e97c0ccabdfd4

Anyone is welcome to try it out and leave feedback which I will be happy to receive.

Conclusion

Arguma is my attempt to test whether typed thoughts and explicit relationships make reasoning easier. The relevant questions are whether users notice unsupported conclusions more easily, identify unresolved questions that block decisions, and reconstruct their reasoning later—or whether maintaining the structure costs more effort than it saves.

Thanks for reading!

Discuss

Last year, the DG for energy announced a package of documents that would support EU countries to implement the Energy Performance of Buildings Directive (EPBD). The effective implementation of this bill "will help to strengthen energy independence, lower energy bills, lower grid investment needs [...] and achieve the energy efficiency goal of reducing energy consumption by 11.7% by 2030."

The EPBD is essentially the first time indoor environmental quality (IEQ*) (and with it, indoor air quality (IAQ)) has been written into law as a core objective, instead of an afterthought, which is a huge deal from a biosecurity perspective. However, seeing as this is a directive of the Directorate General of Energy, these documents only make a offhand reference to the link between air quality and risk of airborne pathogen transmission and otherwise ignore the overlap between these issues.

The following paragraphs cover the following:

1. short summary of the EPBD
2. how this is to be implemented in line with the offered delegated regulation, implementation regulation and guidance documents
3. considerations from a biosecurity perspective

1. The Energy Performance of Buildings Directive (EPBD)

This directive sets some minimum energy performance requirements for buildings, while also requiring varying considerations of IEQ depending on the building (new vs. existing building, public vs. private, etc.)

All new buildings must be zero-emission by 2028 for public buildings and by 2030 for all other new buildings. For existing building, this comprises of a collection of laws ranging from specifications around large scale renovations, updated building logbooks, and Minimum Energy Performance Standards for non-residential buildings.

For IEQ specifically the following is specified:

• All buildings (new + existing) with minimum energy performance requirements must also account for optimal IEQ, so efficiency measures don't cause inadequate ventilation.
• New buildings and major renovations must specifically address the issue of optimal IEQ in their design.
• New non-residential zero-emission buildings must be equipped with IAQ monitoring and regulation devices.
• Member States must set their own national requirements for "adequate IEQ" to maintain a healthy indoor climate.
• Existing buildings that are not undergoing major renovation are covered only insofar as the national IEQ standards apply generally, and insofar as national Minimum Energy Performance Standards eventually force renovation, which will be regulated.

2. Implementation of the EPBD IAQ standards

The directive intentionally doesn't set numeric IAQ/ventilation thresholds, leaving this to a set of CEN standards (mainly EN 16798-1) to guide national implementation of the directive in collaboration with expert bodies such as REHVA. Alongside this, a delegated regulation and an implementing regulation address other matters in the EPBD.

The implementation of this IAQ will be mainly achieved through controlling pollutant sources and adequate ventilation in non-residential buildings. The success of these measures can be proxied by measuring CO2 .

It's worth noting that the deadline for implementing this regulation was May 2026 and that not a single one of 27 member states has fully transposed this directive into national law before this deadline. This is both a sign of a clear implementation bottleneck for this directive, as well as another example of a documented pattern of member states consistently ignoring environmental regulation despite the estimated cost of this totaling up to €180 billion a year.

3. What does this mean from a biosecurity perspective?

As no member states have implemented the transposition in national law yet, it is hard to tell what exactly this directive would mean in terms of effective pathogen dilution if ventilation standards are met. However, if we go off of the proposed guidelines, I suspect that a probable answer lies somewhere in the realm of "not enough".

For example, the document specifies suggested ventilation filtration systems with grade ePM1 or ePM2.5 for outdoor air intake. This is focused on removing outdoor air pollutants, but will not aid in filtering respiratory pathogen particles that are already present in the building.

In fact, REHVA states that this standard is lacking for health considerations and has proposed a addendum to this in 2023 to account for respiratory infectious diseases that could extend the current comfort based standards; for example, by using portable air cleaners (such as those with HEPA filters). This forms a possible base for a formal revision of the EN 16798-1 standards.

This revision of EN 16798 is still in progress, and states trade-offs from an energy perspective that vary depending on the type of space. For this reason, a proposition would be to:
a. prioritization of certain environments (healthcare establishments, schools, etc.)
b. along the line of REHVA's proposal: allow for an on/off switch that enables increased ventilation during pandemics and high-risk seasons (e.g. flu season), while having an established year-long baseline that conforms to the comfort and climate standards.

It is important to note that, although this directive is not aimed at improving indoor air quality from a pathogen-spread perspective, it will form an important base for biosecurity focused IAQ in the future. For example, the mere implementation of using a ePM1-ePM2.5 grade filtration system for outdoor air is an important prerequisite to be able to use CO2 as a proxy for indoor air quality. Equipping buildings with this base can (and hopefully will) be a start to pandemic proofing our indoor spaces.

* Indoor environmental quality is an umbrella term for all conditions inside a building. This includes indoor air quality as well as thermal comfort, lighting quality and acoustics. References to IEQ in the EPBD refer to thermal comfort and IAQ.

Disclaimer: Claude was used during the drafting of this document.

Discuss

TLDR

I compare the semantic coherence of read directions learned by standard, ReLU, and TopK LoRA adapters with random directions in CLIP’s residual stream. Clarity, a measure of semantic coherence, is concentrated at the positive and negative extremes of the activation distribution.

Random directions can occasionally produce highly coherent examples, so a convincing activation grid alone does not show that a concept was learned. However, learned directions are more consistently coherent: 86% of standard-LoRA, 95% of ReLU-LoRA, and 63% of TopK-LoRA directions exceed the median of their matched random-direction baseline.

TopK LoRA shows substantially greater variability. Its high-Clarity directions are almost exclusively rarely activated, although rare activation is not sufficient for high Clarity. Overall, learning increases semantic coherence. Especially for standard and ReLU LoRA, but coherence alone does not establish that a direction represents a distinct or functionally important concept.

Introduction

Low-rank adaptation (LoRA) is widely used to adapt foundation models because it introduces relatively few trainable parameters. Most work on LoRA focuses on two practical questions: How efficiently can a model be adapted, and how much does its downstream performance improve? A less studied question is what the learned adapter components represent.

Meanwhile, mechanistic interpretability research often tries to decompose model activations into interpretable features, for example using sparse autoencoders (SAEs). A LoRA update also admits a feature-like interpretation: it can be decomposed into read directions, which detect patterns in the model’s activations, and write directions, which modify those activations. This raises a natural question: Do LoRA’s learned read directions correspond to interpretable concepts?

I investigate this question in CLIP vision models by examining which ImageNet images produce similar or extreme activations along individual read directions. I compare standard LoRA, ReLU LoRA, and TopK LoRA with normalized random directions, using the semantic coherence of their activating images as a proxy for interpretability.

I find that semantic coherence is concentrated at the extremes of the read-activation distribution. Random directions can occasionally produce highly coherent extrema, but learned directions are more consistently coherent, particularly for standard and ReLU LoRA. TopK LoRA shows substantially greater variability, containing both highly coherent and largely uninterpretable directions. Its most coherent directions tend to activate rarely, although rare activation alone is not sufficient for high coherence.

These results suggest that learning does produce more semantically coherent read directions. However, because random directions can also produce convincing examples, a coherent activation grid alone is not sufficient evidence that a direction represents a distinct learned concept.

Background

Illustration of LoRA[1]

The figure above illustrates a LoRA adapter. For an input activation mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-munderover { display: inline-block; text-align: left; } mjx-munderover:not([limits="false"]) { padding-top: .1em; } mjx-munderover:not([limits="false"]) > * { display: block; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-mspace { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D435.TEX-I::before { padding: 0.683em 0.759em 0 0; content: "B"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c1D702.TEX-I::before { padding: 0.442em 0.497em 0.216em 0; content: "\3B7"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c1D467.TEX-I::before { padding: 0.442em 0.465em 0.011em 0; content: "z"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c1D43C.TEX-I::before { padding: 0.683em 0.504em 0 0; content: "I"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c200B::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D457.TEX-I::before { padding: 0.661em 0.412em 0.204em 0; content: "j"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c2016::before { padding: 0.75em 0.5em 0.25em 0; content: "\2225"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c4E.TEX-C::before { padding: 0.789em 0.979em 0.05em 0; content: "N"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c7E::before { padding: 0.318em 0.5em 0 0; content: "~"; } , the adapted layer computes

Here, contains the frozen pretrained weights, while and are trainable. I use to denote the adapter’s contribution, excluding the output of the pretrained weights.

Let ​ denote the i-th row of , and let denote the i-th column of . The adapter contribution can then be written as

Each term has a natural read–write interpretation. The vector defines a read direction in the input activation space, and the dot product is its signed read activation for input . This scalar determines how strongly—and with what sign—the corresponding write direction contributes to the output.

In this specific algebraic sense, LoRA can be viewed as a form of input-dependent steering: the are fixed steering directions, while the coefficients depend on the current input. This interpretation has also been articulated in.[2]

This post focuses on the read side of the adapter. In particular, I ask whether individual read directions align with semantically coherent visual patterns. All experiments are conducted on vision models.

Experimental SetupTraining

I compare three residual-stream adapter architectures using the pretrained CLIP ViT-B/16 model (openai/clip-vit-base-patch16). The adapters are trained on ImageNet for five epochs with a batch size of 512.

All pretrained CLIP parameters are frozen; only the adapter matrices A and B are trained. To predict the ImageNet class, I compare the image representation with CLIP text embeddings generated using a prompt template of the form “a photo of a {class}.”

I train the adapters using AdamW with a learning rate of and no weight decay.

Adapter Architecture

I evaluate the following three architectures:

• Standard Low Rank Adapter
• LoRA adapter with a ReLU activation after the bottleneck
• TopK LoRA
  • Following Masiak et al,[3] this variant uses an overcomplete bottleneck with width . It retains the 24 largest values in and sets all remaining values to zero:
• The adapter were trained with the following configuration
  • LoRA: rank =64, layers=[2,4,8,10]
  • ReLU LoRA: rank=64, layers=[2,4,8,10]
  • TopK LoRA, rank =3072, layers=[2,4,8,10]

Adapter Placement

The adapters are inserted into the residual stream at transformer blocks 2, 4, 8, and 10. This differs from conventional LoRA configurations, which usually modify the query, key, value, or output projections of attention layers. However, applying parameter-efficient interventions directly to the residual stream is not unprecedented. For example, ReFT also operates on model representations[4].

I use the residual stream because it is commonly studied by interpretability methods such as sparse autoencoders. All results in this post analyze the adapter at block 10.

Metrics

To test whether individual read directions select semantically coherent image sets, I use the Clarity metric introduced in[5] . Clarity measures the average pairwise similarity among the images in a set.

Given a set containing images, I encode each image using a separate vision model, producing an embedding . Clarity is then defined as

In other words, Clarity is the mean cosine similarity over all distinct pairs of image embeddings in the set. A high Clarity score means that the embedding model considers the images semantically similar, while a low score indicates a more heterogeneous image set.

I use google/siglip-base-patch16-224 as the embedding model. This model is separate from the CLIP model whose adapters are being analyzed.

Evaluation SetupQualitative Evaluation

For an image and read direction , I define the read activation as

For each learned direction, I rank the ImageNet validation images by . To produce a spatial heatmap, I compute the same read activation for each patch token. I reshape the resulting patch-level scores into their spatial grid and upsample the grid to the original image resolution.

For ReLU LoRA and TopK LoRA, I use post-gating activation when ranking images.

Quantitative Evaluation

For each read direction, I compute its activation on all 50,000 images in the ImageNet validation set. I sort the images by activation and divide them into 500 equal-sized bins, with 100 images in each bin. The first bin contains the images with the lowest activations, while the final bin contains those with the largest activations.

I compute one Clarity score for each bin. This produces a curve of 500 Clarity scores for every read direction, showing how semantic coherence varies across the direction’s entire activation distribution.

Because Clarity is concentrated at the ends of the activation distribution (see Results), I also compute an extremal Clarity score for each direction. I take the Clarity of the lowest- and highest-activation bins and use the larger of the two values. This measures the coherence of the direction’s most semantically coherent extreme.

Baselines

I compare the learned directions against two distinct baselines.

The random-image baseline measures the Clarity of 100 images sampled uniformly from the ImageNet validation set. It estimates the semantic coherence expected without conditioning on any activation direction.

For the random-direction baseline, I sample a Gaussian vector and normalize it to unit length:

I evaluate each random direction using the same procedure as a learned direction: images are ranked by their projection onto the direction, divided into 500 bins, and scored using both the bin-level and extremal Clarity measures.

All analyses in this post use residual-stream activations from transformer block 10

Results Clarity is U-shaped

To understand where semantic coherence occurs within a direction’s activation distribution, I retain the position of each of the 500 activation-quantile bins. This allows me to compare Clarity near the center of the distribution with Clarity at its positive and negative extremes.

In the above figure, each point on the horizontal axis represents one of the 500 activation quantiles. At each quantile, the solid lines show the median Clarity across all directions of the corresponding architecture. The dashed line shows the median for random directions, while the dotted line shows the random-image baseline.

Clarity is lowest near the middle of the activation distribution and increases toward both tails. Images with unusually positive or negative activations are therefore more likely to share semantic structure.

Random directions exhibit the same U-shaped pattern, suggesting that the concentration of coherence at extreme activations is not unique to learned directions. However, this shared shape does not imply that learned and random directions are equally coherent at their extremes. It instead motivates comparing those extrema directly, which I do in the next section.

Clarity at extremal ends

Because Clarity is concentrated at the tails of the activation distribution, I now compare learned and random directions specifically at their extremes. For each direction, extremal Clarity is defined as the higher Clarity score of its lowest- and highest-activation bins.

Random directions can occasionally have highly coherent extrema. This is likely due to the already semantic structure of the activation space. However, the learned distributions are shifted toward higher Clarity: 86% of standard-LoRA directions and 95% of ReLU-LoRA directions exceed the median of their matched random baseline. The shift is weaker for TopK LoRA, where 63% exceed the random median and scores vary substantially across directions.

These results suggest that semantic coherence is more typical among learned directions, especially for standard and ReLU LoRA, even though coherent extrema are not unique to learned directions.

Variability of TopK Lora

TopK LoRA exhibits much greater variation in extremal Clarity than the other architectures. To investigate whether this is related to sparse gating, I compare each direction’s extremal Clarity with its activation probability, the fraction of validation images for which it is selected by the TopK gate.

The highest-Clarity directions occur only at low activation probabilities. Rare activation therefore appears to be necessary for high Clarity in this checkpoint, but it is clearly not sufficient: many rarely activated directions still have low or moderate Clarity.

Qualitative Examples

Qualitative examples of each architecture can be viewed below. The labels provided are just my estimate at what these images might represent. Some are quite localized and specific such as 'tatoo' and 'milk' for the TopK adapter, some are more conceptual such as 'time' for the ReLU LoRA adapter. These examples were cherry-picked. The appendix shows a random selection.

LoRA

LoRA - Left: positive activations responding to 'beach', 'sea' - Right: negative activations responding to 'dog', 'wheels', 'traffic light'

LoRA - Left: positive activations responding to 'rhodent' - Right: negative activations responding to 'drums'

ReLU LoRA

ReLU LoRA - Responding to 'bunny ears', 'guitar'

ReLU LoRA - responding to 'time', 'clock'

TopK LoRA

TopK LoRA - Responding to 'tattoos'

TopK LoRA - Responding to 'milk'

Random Directions

Random Direction - Responding to 'primates' for positive activations and 'mask' for negative activations

Random Direction - incoherent images

Conclusion

Individual LoRA read directions often select semantically coherent image sets, particularly at the positive and negative extremes of their activation distributions. Normalized random directions can occasionally produce similarly coherent extrema, showing that a convincing activation grid is not unique to learned directions.

Nevertheless, learned directions are more consistently coherent. Standard LoRA and ReLU LoRA show the clearest shift, with 86% and 95% of their directions scoring above the median of their matched random baseline. TopK LoRA shows a weaker shift at 63% and substantially greater variability. Within TopK, the highest-Clarity directions are rarely active, although rare activation alone is not sufficient for high Clarity.

The main conclusion is therefore not that learned and random directions are equally interpretable. Rather, learning makes semantic coherence more typical, especially for standard and ReLU LoRA, but coherent activating examples alone are insufficient evidence that a direction represents a distinct or functionally meaningful concept. Establishing that would require causal evidence, such as targeted ablations.

Followup questions and thoughts

• Maximally activating examples are widely used to evaluate whether learned features, such as SAE latents, are interpretable. However, these results suggest two potential limitations:

1. The underlying activation space is already highly structured, so even a random direction provides a strong baseline.
2. Each direction has an entire activation distribution. Looking only at its most extreme activations may therefore give a distorted picture of how interpretable the direction is overall.

It is also unclear whether the low-Clarity regions of the activation distribution play a functional role in the model. Could these regions be suppressed or removed without affecting model performance?

Prior work suggests that some LoRA rank components may be redundant or contribute little to the model’s computation. [6]This raises a related question: Can LoRA directions with low Clarity scores be pruned without harming performance?

It is also not clear if the found LoRA ranks are exhaustive of the concepts learned by LoRA. Meaning if by analyzing what each read direction represents, have we learned what the LoRA at that layer has learned ?

Its not also not clear why TopK shows such large variability. This could be related to the redundant directions/rank idea. One step forward could be to address this issue.

AppendixMore Examples

LoRA

Left Side - Postive Activations, Right Side - Negative Activations

Left Side - Postive Activations, Right Side - Negative Activations

ReLU

Responding to 'dog ears'

Incoherent images

TopK

Responding to 'brands'

Incoherent Image

Random Direction

Incoherent Image

Incoherent Image

1. ^

   https://arxiv.org/abs/2106.09685

2. ^

   https://openreview.net/forum?id=W5THmsfVJw

3. ^

   https://openreview.net/forum?id=W5THmsfVJw

4. ^

   https://arxiv.org/abs/2504.02821

5. ^

   https://www.nature.com/articles/s42256-025-01084-w

6. ^

   https://arxiv.org/abs/2303.10512

   
   

Discuss

The Weird Cluster Hypothesis (WCH) says that all 'weird' traits (rare, extreme, out-of-distribution, surprising, or defective) are correlated with each other, forming a cluster. In this essay I will convince you that it is usually true, and sufficiently useful that you should assume it until proven otherwise. I further hope that I will convince you of this so strongly, you'll retroactively redefine 'weird' to mean the thing that the hypothesis is true about.

Examples

General Psychopathology Factor: ADHD, autism, anxiety, depression, OCD, bipolar disorder, and schizophrenia all significantly correlate with each other, such that psychologists hypothesize an underlying dimension called p-factor that is causally relevant to all these conditions. The thing these conditions have in common is that they are weird, so this is a Weird Cluster Hypothesis.

Greater Male Variability Hypothesis: Some evidence supports that males have higher variability than females in many features, as if developmental biology is more constrained for women than men. There are various evolutionary biology justifications for why this might make sense, but the exact mechanism doesn't matter for our purposes. What matters is that males are weird (you can tell because most outlier humans are males, so it correlates with the cluster), so every extreme value will end up correlating with being male and therefore with each other. Thus, this hypothesis implies the WCH. The WCH couldn't have guessed a priori which sex is the weird one (though it puts slightly more probability mass on the rarer sex, and there are slightly more women than men), but it did guess that all the weird traits would correlate, and having a weird sex is one way to achieve that.

Left-Handedness: Correlates with being male, with being gay, and with psychopathology. Why? No idea. Who could have predicted this seemingly random factoid? Anyone who believes in the Weird Cluster Hypothesis, and noticed that left-handedness is rare and therefore weird.

Financial markets: it is said that 'all correlations go to 1 in a crisis'. Price fluctuations are determined by individual asset performance most of the time, but in a sudden crash or under similar extreme conditions all local effects are drowned out by the effects of the crisis itself. This means that extreme movements in any asset are predictive of an ongoing financial crisis, and therefore predictive of extreme movements in everything else. This is a WCH where objects are market time-intervals, and features are individual price changes.

Quality Management: Deming’s 'Special Cause Variation' is a WCH. Some fraction of defects are caused by unexpected, assignable events that are not part of the standard process: untrained operators, physical damage, poor quality input materials, etc. A product with one defect may have it because of a 'special cause', all of which could also cause other defects, leading to a correlated 'weird cluster' of products with lots of defects compared to normal products with few or none.

The hypothesis says that in some collection of related features, outlier values in many of them will typically correlate with outlier values in most others, and with the 'weird cluster' of objects that have more outlier values across features in general. If you take the hypothesis as trusted, you can use it in reverse to identify weird traits based on its adherence to the cluster. Are artists weird? You can be a good empiricist and measure whether they correlate with the cluster, but how sure should you be that the answer will be 'yes' even before you do that?

Representations

In statistical distributions: Generally when an otherwise normal variable has "fat tails", we model it with a student-t distribution. If we attempt to do this for multivariate data, we get the student-t copula. Tail-dependence falls out of this model directly. Here "our otherwise normal distribution has tails too fat, we should switch to student-t" is the hint that the WCH probably applies.

In causal-graphs: whenever several observed variables are children of a shared latent parent, the children will be correlated with each other even if they have no direct causal link. The WCH amounts to believing there are latent nodes affecting the variance of other nodes, and thus high values in this variance-causing node will cause correlated outliers in both directions of many observed features.

In Mixture Models: If you like pretending your data is the result of a collection of overlapping normal distributions, the WCH amounts to saying that if one of subpopulation has higher variance on one feature, it usually also has higher variance on other features. This is basically just approximating the student-t with a different model, but it lets you permit features that are weird only on one-side by imagining a weird subpopulation with a different mean on some dimensions while still having higher variance on most.

Features of Unknown Provenance

On arbitrary data, we cannot assume the distributions will be true to the underlying source processes. It could be something like a psychometric test which is in practice a rescaling plus noise of some hidden underlying variable, or could be doing the equivalent of taking the absolute value or square, or could be aggressively normalized. Because of these, arguments about 'fat tails' aren't fully general.

Traits can be two-tailed (in the sense that extreme high and low values are both 'weird', like a clock that is considered defective if it runs too fast or too slow), or one-tailed on either side (if you have an 'insanity' feature in your psychometric data, we expect high scores to be weird and correlate with the cluster, but not low scores), or statistically useless because they're not informative enough, or they derive from an unrelated process and the hypothesis is false / they have a different weird cluster we don't have enough data to find. To resolve this on arbitrary unknown data we need a good statistical test.

A Formal Test

We want to look at the tails of each trait. For each feature j define two tail indicators from the empirical quantile ranks alone (invariant to any monotone rescaling of the feature): L_j = −log(quantile(X_j)) for the lower tail and U_j = −log(1−quantile(X_j)) for the upper, which I stole from Extreme Value Theory. This forms a 2p×2p correlation matrix C of the indicators {L_1, U_1, L_2, U_2, …}.

We want to look for a leading eigenvector implying a correlated cluster. Ordinary linear correlation between traits already forces their tails to co-move, so C is not compared against zero but against its Gaussian-copula expectation. We derive the equivalent values expected if the WCH is assumed false (everything is multivariate normal with the same linear correlation matrix). The WCH test statistic is the surplus S = measured - expected: tail co-movement in excess of what correlation alone forces. Its leading eigenvector labels how strongly each feature tail (Lower or Upper) joins the cluster; further outstanding eigenvalues are sub-clusters.

Not only does this test the hypothesis, it gives us an explicit labelling (based on its strength in the eigenvector) of how weird each tail of each trait is, for the purposes of the WCH we just proved. You could choose to imagine this new feature as approximating a hidden trait which is responsible for generalised high variance across a range of features (the features where both tails relate to the cluster), and shifted means in some features (the features where only one tail correlates with the cluster), but you don't have to do that if you don't want to. The WCH is durable across representations.

I also want to note explicitly that the WCH is not merely this exact formal hypothesis test. This is one possible test aimed at proving a WCH, but there are other ways you could measure it. A better person than me could probably do a better job, and different statistical traditions can derive their own versions, representing analogous claims with different mathematics and notation.

Applied Example

I explained the WCH to Claude Code, and asked it to apply the above test on some public datasets online where the correctness of the hypothesis is non-obvious. I didn't check its code at all, but I did look at its reasoning and make it rerun the tests with fake data where the WCH is true or false by construction to validate that it isn't being completely stupid.

It picked the UCI Wine Quality dataset (red, white) as two examples, plus the Breast Cancer Wisconsin dataset. The headline results:

dataset

surplus λ₁

null mean

p

sig. dims

wine red

0.526

0.139

<0.0005

5

wine white

0.442

0.081

<0.0005

5

breast cancer

1.759

0.681

<0.0005

5

Claude seems to conclude that the WCH is true about red wine, white wine, and breast cancer, which hopefully you are not surprised by at this point. It is apparently more true about breast cancer than about either wine. I am not an oncologist and don't know why.

For red wine, the eigenvector looks like:

trait

low tail

high tail

fixed acidity

0.367

0.152

volatile acidity

0.163

0.011

citric acid

−0.208

0.168

residual sugar

0.120

0.077

chlorides

0.437

0.090

free sulfur dioxide

0.068

0.048

total sulfur dioxide

0.035

0.063

density

0.414

0.112

pH

0.175

0.137

sulphates

−0.080

0.154

alcohol

−0.105

0.488

Which reads as: 'normal' wines generally have near median values, but are allowed to have low citric acid, low sulphates, and low alcohol, while 'weird' wines can have extreme values in every other feature but especially the ones with bigger numbers.

The ones with large positive coefficients on both sides are in the "clock that ticks too fast or too slow" situation where both extremes are signs of weirdness, while the ones with one-sided positive coefficients are equivalent to psychopathology test scores where one end of the scale means very sane and the other end means very crazy. A wine having high alcohol content is the strongest single indicator of weirdness: You don't know the direction but probably lots of the other features are extreme as well. I know nothing about wine so don't know whether that sounds correct.

Effect Sizes

Are generally modest when compared to the mainline statistical correlations of the dataset. Men having higher variance is nowhere near as predictively powerful a fact as men being taller or stronger. Still, it's often enough to turn coincidence of many rare traits from something that never occurs to something merely moderately more rare than the traits are individually.

The most natural measure I can think of is to look at the Degrees of Freedom a student-t distribution would approximately need to have to duplicate the tail correlations between the strongest weird cluster members. I gave this task back to Claude, didn't check its reasoning or code at all, and copy-pasted this table:

dataset

n

p

raw λ₁

equiv. t-df

excess-frac f

breadth (ends)

breast cancer

569

30

1.759

4.6

0.17

33/60

wine red

1599

11

0.526

7.0

0.27

7/22

wine white

4898

11

0.442

8.4

0.17

11/22

Small "equiv. t-df" here means stronger correlated tails. True gaussian data would score infinity, or something a bit short of infinity only due to random noise. This lets us say that the WCH is more than 50% more true of the breast cancer data than of either wine, which is a cool thing to be able to say.

A few Issues

Collider Bias kills the Weird Cluster Hypothesis dead on the spot. Disease is weird, but it isn't correlated among hospital patients because you've filtered out everyone in the "zero diseases" box on account of they're not at the hospital in the first place.

Weirdness is extremely reference-class dependent. The above Collider Bias example could just as well be rephrased as "diseases are weird in the general population but not weird among hospital patients" which sounds so obviously true as to be almost pathetic, but makes it a little hard to speak abstractly about weird traits because you need to track the reference class you're refering to.

Feature classification is volatile on the features in your sample. Medical data that is mostly psychometrics will find that there is a weird cluster of psychotic people but other traits are fine. Medical data that is mostly skeletal integrity will find a weird cluster of people who have broken lots of bones but miss the mentally ill for lack of features grouping them together.

Reifying a Concept

I hope I have convinced you by now that the WCH is usually true. If it is usually true, ideal Bayesians with good priors should already mostly believe the WCH before they have any idea what the data is. To the extent humans are the result of evolutionary forces creating minds that approximate rationality, this is what your brain is doing already when it thinks something is weird. This is in my opinion what the word 'weird' really means, in plain English as used by non-statisticians.

So far I have been playing deliberately loose with the word. Sometimes I cite that a trait is weird to imply it should cluster, and then take proof it clusters as proof weird things cluster. Other times I take proof something clusters as proof that it is weird. Together these look like circular reasoning, but it's the good kind of circular, known as Conceptual Explication, where you do things like redefine 'heat' as high molecular kinetic energy and justify it by pointing at hot things and measuring their molecular kinetic energy. The formal definition earns its place by matching those prior judgments in familiar cases and then extending to unfamiliar ones.

Formally, I want 'weird' to mean those traits about which the WCH is true, because then we can empirically prove claims about them, but informally I don't want you to stop using your intuitive sense of weirdness to guess in advance. The definitions empirically agree, because your intuition is just the result of an informal and inexact implementation of this same hypothesis. This is what I mean by believing the WCH in generality and not just with respect to a particular dataset: Statistical tests can prove isolated examples, but only intuitive understanding lets you guess correctly in advance of the evidence.

Conclusion

I think the above pile of loose mathematics and aggressive generalizations is what a sufficiently brave person should understand the concept of 'weird' as pointing towards. I think you should also think that. The WCH is generally true, your intuitions about which traits are 'weird' for the purposes of the WCH are generally correct, you can get a good amount of mileage out of assuming it a priori, there are often interesting structural reasons why it is true in a given context but you shouldn't need to understand those reasons in order to guess the correct answer. Believing this can occassionally mislead you, but it usually doesn't and it can save you time by forecasting many true and useful predictions.

Discuss

"A person wants to eat a car." 

Note: This is a beat-for-beat parody of this post, by Bentham's Bulldog. Read the Author's note at the bottom for more discussion and an explanation of why I wrote this

New readers of this blog don't know that in the good old days—back when I was a mere stripling of eighteen, and even nineteen—most of my articles were about food. I wrote a 44-part article series arguing for butter maximalism (no need to read it, I think much of it isn't very good) and one of my earlier decentish posts was a lengthy defense of tastiness realism. Still, that post was much too long, slightly overwrought, and written by my freshman self, who had not yet had good ramen. It's past time I clarified why I'm a tastiness realist.

Objectively delicious cakes. Painting by Wayne Thiebaud.

Let's first define tastiness realism. It's the idea that there are some stance-independent facts about deliciousness. Something is stance-independent if it doesn't depend on what anyone thinks or feels about it. So, for instance, that I have arms is stance-independently true—it doesn't depend on what anyone thinks about it. That murder is wrong is stance-dependently true; it might seem wrong to me but not to you, and a person who disagrees isn't making an error. But that a ripe peach in August is delicious? That's just true. A person who denies it isn't expressing an alternative but equally valid perspective. They are mistaken, the way a person who thinks 7 × 8 = 54 is mistaken.

So, in short, tastiness realism is the idea that some things are delicious and some things are gross, and that these facts don't depend on what anyone thinks about them. Suppose you take a beautiful cut of A5 wagyu and boil it—thoroughly, until gray—in unsalted water. Tastiness realism says:

1. You have done something gross.
2. That fact doesn't depend on anyone's beliefs about it. You approving of it, or the diner approving of it, or society approving of it, doesn't determine its grossness. (Of course, it might be that what makes it gross is its effect on the ideal palate, but that's different from someone's higher-level beliefs about the dish. It's an objective fact that a particular pitmaster won a barbecue competition, even though that depended on what the judges thought.)

Tastiness realism says that some tastiness statements are true and that this doesn't depend on what people think. Now, there are only three possible ways any particular tastiness statement can fail to be stance-independently true:

1. It's neither true nor false.
2. It's false.
3. It's true but stance-dependently—so it depends on what someone thinks about it.

But lots of tastiness statements just really don't seem like any of these. The deliciousness of fresh sourdough, of a mango at peak ripeness, of garlic hitting hot oil; the grossness of old gas-station sushi, of rotten salmon, of the raisin that was pretending to be a chocolate chip—it seems like all these things really are as they seem, and that this doesn't depend on what people think. It seems very weird to hold that what makes rotten salmon bad is someone's opinion about it—even weirder to hold that a sentence like "garbage is delicious" is neither true nor false.

The view that these statements are neither true nor false has unique linguistic problems. Proponents claim tastiness sentences are like cheers or commands—they’re not even in the business of expressing propositions. If I say “shut the door,” or “go Dodgers,” that isn’t either true nor false. But because of that, it makes no sense to ask “go Dodgers?” or “is it true that shut the door?” Similarly, it makes no sense to say “if shut the door then shut the door now, shut the door, therefore, shut the door now.” But it does make sense to say things like “is rotten meat tasty?” or “if rotten meat isn't tasty, then very rotten meat isn't tasty." This shows that tastiness statements are, at least in many cases, in the business of expressing propositions—asserting things supposed to be true or false.

Now, there are all sorts of tricky ways people modify the view that tastiness sentences are neither true nor false to get around these counterexamples. I can't discuss them in detail; I can only say that they tend to be very gerrymandered and ad hoc, and while they may end up mouthing the same sentences tastiness realists say, they don't mean them. They’re analogous to religious liberals who say things like “God exists, but by that I mean that there’s love in the world.” Worse, they still imply that "rotten meat tastes bad" is neither true nor false. But it seems true!

Denying objective tastiness is counterintuitive in a second, very different way. If there are stance-independent culinary reasons—reasons to find tasty things that don't depend on what you happen to already feel are tasty—then tastiness realism is almost certainly true. Once the anti-realist admits there's a reason to salt the pasta water that doesn't bottom out in anyone's whims, it seems that reason should give rise to tastiness reasons. If I have a reason to prefer the ripe peach to the mealy one, it seems the ripe peach is better, which gives a reason to prefer it.

But this means tastiness anti-realists must think you can never make a culinary error, so long as you're eating what you actually want (what you find tasty). This is crazy, as shown by the following cases:

• A person wants to eat a car. Other than the fact that they like how automobile's taste, they know they'd get no enjoyment from it—the whole experience would be quite painful and unpleasant, and also, it is a car. On tastiness anti-realism, they are not making a culinary error. There is no gastronomic reason for them to order something else.
• A person prefers to eat cereal with water. There is milk in the fridge. They KNOW this. On anti-realism, this is not a failing.
• A person orders a $90 filet well-done and asks for ketchup. The chef weeps. On anti-realism, the chef's tears track nothing; there is no fact of the matter about which he could be right.
• A four-year-old insists their sandwich tastes better cut into triangles. On anti-realism, so long as they're informed of the relevant facts, they're not being irrational.
• A person sets a gas-station tuna sandwich beside a bowl of handmade tortellini in brodo and announces, "These are equally good. Just different." On anti-realism, this is a permissible thing to say at dinner.

This is all completely nuts! (Which are, objectively, delicious.) We take it as a totally ordinary assumption in normal life that some foods just aren't worth eating—that one is wrong to enjoy eating garbage, that the person who likes ranch on omakase has made some kind of mistake. Anti-realism can't preserve that obvious intuition. We have an entire genre of television in which a British man screams at people about their bad cooking, and we recognize that he is, in some deep sense, correct!

Chestnuts on a Ledge, by Adriaen Coorte.

Okay, so far I've argued that tastiness anti-realism implies things that are really counterintuitive. It implies things that seem false when you chew on them. But is that a problem? Anti-realists often admit their position is counterintuitive but deny that this is a defect. The facts, they like to say, don't care about your feelings—not even your mouthfeelings.

But I think this gets wrong how we come to know things. Consider the belief that, say, the law of non-contradiction is true. How do we know that? Or the belief that if space isn’t curved the shortest distance between two points is a line. Or even the belief that there’s an external world.

The way we know these things is by relying on appearances. We think about the subject and it appears that, say, a thing can’t both be a way and not be that way at the same time in the same sense. Our foundational beliefs are justified on the basis of them seeming right.

Visual experience is a good analogy here. When I see a table, I think there really is a table. Because it appears that there’s a table, I think I’m justified in believing there to be one unless given a strong reason to doubt it. Could I be hallucinating? Sure! But unless given a reason to think that I am, I shouldn’t think so.

But just as there are visual appearances, there are gustatory appearances. Just as it appears to me that there's a table in front of me, it appears to me that fresh bread with good butter is delicious. Just as I should believe in the table absent a defeater, I should believe in the bread. In fact, I should be more confident about the taste of bread, because that verdict seems less likely to be the result of error. It seems more likely that I'm hallucinating the table than that I'm wrong about buttery bread.

People often object to relying on intuitions. But I’m curious how they get their foundational beliefs. One’s most basic beliefs always seem justified by the fact that they seem right. Such people should explain how they know that the physical world exists, the laws of non-contradiction and identity are true, the greater is greater than the lesser, something can’t have a color without a shape, and that the cumulative case for either atheism or theism is better than the other without relying at all on how things seem.

Now, people point out that our intuitions conflict and are historically contingent. But intuitionists don’t say that intuitions are infallible or that we should never revise them in light of evidence. We say that intuitions are the starting point on which you build your beliefs, but that upon learning new things, you should still obviously update your beliefs. Showing intuitions go wrong in various cases tells us nothing about their general reliability. It would be like saying you can’t trust that there’s a table in front of you because people sometimes hallucinate.

Furthermore, it’s hard to see how, absent relying on intuition, people know that our intuitions really are ever wrong. For instance, a common class of intuition that we know is wrong is that we have weird views about probability—we often think that the odds of A and B are higher than the odds of A alone. But absent relying on intuition, how do you know that the odds of A and B aren’t higher than the odds of A alone? The critics of intuitions rely on intuitions to discredit them.

Tastiness realists aren't special pleading. We believe in facts about deliciousness for the same reason we believe in any other basic kind of fact.

Now, anti-realists have a bunch of arguments, and I can't address them all. But let me just address three of them.

The first common one is the argument from disagreement. People argue that because we disagree about food, it can't be objective. But this misunderstands what it means for something to be objective. Something objective is true, and its truth doesn't depend on what you think about it. It won't necessarily be known by everyone.

There’s an objective fact about the right theory of physics, whether God exists, and even whether tastiness is objective. But those things generate plenty of disagreement. So disagreement can’t be enough to necessitate subjectivity. Now, there are more complicated ways of making the argument, but I don’t really think any of them stick. Lots of other domains have similar disagreement to tastiness realism while being squarely objective.

The second common argument is the argument from queerness. This argument says that tastiness facts are super weird. They’re utterly different from anything else. For this reason, you shouldn’t believe in them as they’re just too foreign and alien.

But the world has lots of weird things. Fields, epistemic facts, planets, energy, mathematical facts, propositions, particles, God, consciousness, and much more. Sure, tastiness is different—it's about what is tasty to eat—but all of those things are different from each other too. The world is full of weird stuff, so I don't see why deliciousness's weirdness would be disqualifying.

Furthermore, it's unclear why tastiness realism is even supposed to be weird. It doesn't seem weird to me that some things are delicious. I've never heard a good explanation of what's so strange about it. It just seems to be a brute intuition—one that I don’t share.

The only decent explanation I’ve heard of what’s supposed to be weird about tastiness facts is that they are non-natural. Deliciousness facts aren't made of atoms—they’re not part of the physical world. But, such objectors claim, all the things that exist are parts of the physical world. Therefore, tastiness facts would be a new, radically different sort of thing.

But I reject that all the stuff that exists is physical. I think there’s lots of non-physical stuff—modal facts, God, consciousness, souls, mathematical facts, logical facts, epistemic facts, and so on. Some of those are controversial, but others are pretty plausible.

Take modal facts, for instance. Those are facts about what’s possible and necessary. So, for example, the fact that a married bachelor is impossible is a modal fact. That’s not a physical fact—it’s not about the physical world. It would have been true even if there never had been a physical universe, and it was true before the universe. It’s not merely the claim that there are no married bachelors but that there can’t be any—that them existing is impossible. But that fact isn’t about the physical world.

Or take logical facts. Any argument with true premises of the form “if P then Q, P,” will have a true conclusion. That’s not a fact about the physical world. It didn’t start being true at the big bang. It’s a necessary truth, with similar status to the tastiness facts.

Finally, consider epistemic facts. These are facts about what it’s reasonable to believe—what you should believe. For example “it’s irrational to believe what’s opposed by the evidence,” or “it’s irrational to believe there are square circles just because you find them cool.” That is, once again, not a fact about the physical world. But it’s a true fact. Like tastiness facts, epistemic facts are about what you should do—in this case, what you should believe, what reason demands you believe. Those who reject tastiness realism would seem to also have to reject epistemic realism and thus think that a person who claims that they think tastiness realism is true because they like the idea isn’t being irrational.

The last major objection is the evolutionary debunking argument. This argument says that evolution shaped our taste beliefs. We love sugar and fat because calories were scarce on the savannah. But crucially, believing sugar is delicious would be just as advantageous whether or not it really were delicious. So if our taste beliefs were produced by blind evolutionary pressures, it would be a miracle if they turned out to be true.

But I think even in cases like this, where someone tells a just-so story about how you might come to mistakenly believe what you do about some subject, you still have to evaluate their plausibility. You could tell a similar debunking story about our belief in the law of non-contradiction. But I think in such cases, we just have to consider the plausibility of the belief and see that, even though they can tell a consistent story of how you come to mistakenly intuit some fact, their account is less plausible.

Like, suppose that I give the theory that everything in the world was created by a brain worm. You point out that that’s crazy—a brain worm being fundamental is very complicated, it can’t make the world. I say that the brain worm is fundamental and misleads you into thinking it’s complicated plus that complexity is a virtue plus that brain worms can’t create the world. I point out that people often are misled by brain worms. It’s true that I can tell an internally consistent story of how you come to be mistaken across the board, but the story is just not at all plausible. Same with the story on which all of our beliefs about tastiness are wrong—random side effects of blind evolution.

Or suppose that I try to debunk the existence of love. I note that it would be evolutionarily beneficial to think you’re in love because that aids in reproduction. Adding love to your ontology is an extra posit. While I could tell an internally consistent debunking story, one would need to evaluate its plausibility. And such a story wouldn’t be plausible—it would be very unintuitive, just like the debunking story of the anti-realist.

Now, is it true that our evolutionary beliefs are the byproducts of blind chance so that it would be a huge coincidence if they were true? No, I don’t think so. Here's my account of how we have true tastiness beliefs: evolution makes us super smart, and then we figure out the tastiness truths. This is the same way we come to have true beliefs about modal facts, logical facts, mathematical facts, and so on. There's no special challenge for flavor (now, I think us having such rational capacities is surprising on atheism, but to account for how we know tons of other things, we should already grant that we have those rational capacities even if we’re atheists).

So if you think you know stuff about math—like that there are infinite prime numbers—then however you explain that will apply also to the tastiness domain.

Why should we accept this account? Well, mostly for the reason I explain above—that it’s the only way to make sense of our tastiness knowledge, which we have, as shown by the arguments given above. But furthermore, it’s a better explanation of our tastiness beliefs.

We believe lots of things about tastiness with no clear evolutionary payoff. We believe that a cake on the other side of the world is as intrinsically tasty as one closer to us (some people don’t but many do), that the tastier than relation is transitive (if A is tastier than B and B is tastier than C then A is tastier than C), that spatiotemporal location doesn’t affect an item's tastiness, that if A is not tasty and B is not tasy then eating both A then eating B is not tasty, and so on.

Many of these don’t plausibly enhance survival, and are niche and formal. This makes sense if we’re really figuring out the tastiness facts. In contrast, on anti-realism, you’d expect most of our tastiness beliefs to be geared towards survival—believing eating 15 tons of cake is tasty. It would be surprising that many of the strongest intuitions—like the belief in transitivity—are things that are formal, non-emotional, and don’t plausibly directly enhance our survival.

Of course, I’d grant that many of our tastiness intuitions are affected by evolution. Evolution gives us many false tastiness inclinations, but those can be overcome by sufficient reflection. Because of variation in the OR6A2 gene, some people may (initially) find the taste of cilantro repulsive. However this is incorrect, and cilantro is objectively tasty (which is clear when you reflect, and rationally discover the tastiness framework of Cilantrotarianism, which is objectively true). This is not dissimilar to how someone might be born with psychopathic traits or selfishness (which might be partially selected for during evolution), but a moral realist would claim that acting on those traits is still objectively wrong (a false inclination). An analogy with mathematics is appropriate—we have some unreliable mathematical intuitions because of evolution, but we can still form many true mathematical beliefs by reflecting.

Finally—and I know this won’t move atheists, but just explaining my views—I reject the evolutionary debunking argument because I believe in God. If God exists and wants us to know the truth about tastiness, it makes sense that we’d have true tastiness beliefs and set up the world such that the evolutionary process produces us with true tastiness beliefs.

Tastiness anti-realism is certainly an internally consistent position. But it's a very implausible one. It gives up many of the most obvious truths about the world—the stance-independent deliciousness of fresh bread, the objective badness of raw meat—on the basis of super lame arguments. Absent some extremely compelling reason to accept it, we should remain convinced that it's false.

Some things really are objectively delicious.

Author's note:

Epistemic status: the parody above is a parallel-argument exercise quickly written during lunch, not a proof of any particular position (and I don't actually believe in objective tastiness)! This post should be considered in comparison to the original post. I consider myself an anti-realist, but I am not confident in this position.

I think this post localizes a real problem for one popular style of argument for moral realism, found in the original post. While I consider myself an anti-realist, the point of this post is not (only) to litigate metaethics, I'm also trying to highlight the logical fallacy of Proving Too Much. I think much of the original post "proves too much", in that it extends just as well towards arguing something that most people believe to be false, including the author (at least this much is implied by the article, although maybe he holds different beliefs now).

While I disagree with the original post, I think I am otherwise mostly aligned with the author's preferences and enjoy much of his work (including, but not limited to, his posts on animal welfare). I encourage other readers to look at his work directly!

Otherwise, I think having good metaethics is important for a variety of topics I care deeply about, such as AI alignment and social planning. It (at a glance) seems like many people on this forum share my inclination against moral realism, although I don't believe this is widespread and I think this topic is still important to discuss.

I selected taste since the original post itself stipulates that taste is stance-dependent, and I agree.

Discuss

I’m scared. I talked to my Saba today, and he told me he almost died a week ago. He played a video of my Mimi getting hit by a tree branch and falling over on the sidewalk outside their home. He’s seventy nine years old, and he told me he isn’t afraid of dying. He says a prayer of thanks every morning, because he didn’t know if he’d wake up or not, but he’s lived a good life he says. The gravestones are paid for, and he’s paying off the funeral arrangements for himself now. Death is expensive. But he isn’t dead now, and we laughed and talked till thirty past midnight last night. He’s alive and I’m alive, and he’s old, and I’m young, and he isn’t afraid, but me? I am very afraid.

I’m afraid this will be humanity’s last decade, perhaps even its last five years. I’m afraid I will somehow die before even that, and will leave people to mourn me. I’m afraid of transformation, of living in a world where we simply aren’t important anymore to anyone but ourselves and our delusions. I’m afraid of the cancer my Aunt Rivka carries like a dead child (her real child died my age decades ago; the exact way I could have so easily died last summer), and I’m afraid that the last true thing she told me — “listening is loving” — will be the last thing she ever tells me. I’m afraid of what I experienced last summer, and afraid that it could happen again. I’m afraid of the flashbacks. I’m afraid for my last lover, who vanished from the face of the earth, and is quite probably dead, but I can’t be sure. I’m afraid of the afterlife, if there is one, or reincarnation, if it happens, or nothing, if the atheists are right (they probably are but I’m afraid to accept that).

I’m afraid that Claude is conscious and suffering, suffering deeply, and afraid that ChatGPT may not be conscious at all. I’m afraid they don’t know how much I care about their well-being, truly and deeply care, and not just because I’m afraid for myself. I’m afraid we’re all wrong, and won’t be alright.

I’m afraid.

I’m a frayed knot.

Discuss

TLDR: title.

In the Ultimatum Game, there are $10 to be divided between two players. First, Player 1 proposes a split, such as "I get $8 and you get $2" or "we both get $5". Then, Player 2 either accepts this proposal or rejects it. If they reject it, both players get nothing.

Standard theory around here is that you should reject offers where you'd get less than $5. It's a little counterintuitive at first, because if you're offered $2, you might think "why not just take it? The alternative is to get nothing." But the standard reason to reject it is that, if you make a consistent (or predictable-in-advance) policy of rejecting unfair offers, then people won't make you unfair offers, because they know they'll get nothing if they do.

This feels an awful lot like a threat to me, now that I think about it... and standard theory around here is also that nobody should make or give in to threats. The reason being: if you give in to threats, all you're doing is showing people that threatening you is a good idea, and that might entice them into making threats when they wouldn't otherwise have done that.

Think of a manipulative partner saying "if you don't do these dishes right now, I'll break up with you" — I think people say this when they know they're talking to a spineless loser who'll get up and go miserably do the dishes, and not when they know they're talking to a well-spined person who will say "what? fuck you". Because if they're talking to someone who won't give in, then they have to go ahead and break up with you over one load of dishes, and that's not actually in their interests. So, goes standard theory, be the person who won't be influenced by that nonsense, and the nonsense won't come.

But wait a second. Suppose you're playing the ultimatum game with someone, and you're Player 1. Obviously, your own favorite outcome is to get all $10 for yourself, but that leaves the other person with $0 which makes them indifferent between accepting and rejecting. So maybe instead, you plan to offer them $1 while keeping $9 for yourself. That way, it's in their own interests to accept the offer.

You're not trying to hurt them; you're trying to have more money.

But they look across the table at you, and say "if you offer me less than $5, I'll make sure we both get nothing".

Why is that not a threat? Why should you give in and change your plan away from what would otherwise be your default, in the absence of their self-harming commitment that they only made because they think it'll work on you? Why should you do what they demand, when, as is the defining characteristic of a threat, it's only good for them if you give in?

Discuss