Новости LessWrong.com

Published on February 12, 2026 4:33 PM GMT

What happens when a model strategically alters its exploration to resist RL training? In this post, we share our conceptual framework for this threat model, expanding on our research note from last summer. We formalize and decompose "exploration hacking", ahead of an upcoming paper where we study it empirically—by creating model organisms that resist RL-based capability elicitation, evaluating detection methods on them, and auditing frontier models' propensity to exploration hack. We welcome feedback on our framework and encourage researchers to explore the open problems we outline.

Huge thanks to David Lindner, Roland Zimmermann and Scott Emmons for their excellent supervision on this project.

Introduction

Reinforcement learning (RL) is increasingly central to safety training, frontier reasoning capabilities, and dangerous capability assessments in LLMs. For example, reinforcement learning from human feedback (RLHF) and AI feedback (RLAIF) are now standard methods for alignment and behavior shaping. Moreover, RL with verifiable rewards (RLVR) has played a central role in producing today's reasoning models and in fine-tuning models for specific tasks. And frontier labs are using RL-based capability elicitation to estimate worst-case risks before public release.

However, RL is fundamentally dependent on sufficient exploration of diverse actions and trajectories so that high-reward trajectories can be subsequently reinforced. This differs from supervised learning, where training data is fixed: in RL, the model generates its own training data through exploration. Consequently, if a model were to alter its exploration behavior, the training outcome could be significantly compromised.

This failure mode is known as exploration hacking: when a model strategically alters its exploration during RL training in order to influence the subsequent training outcome.

Unaddressed exploration hacking poses risks in three main scenarios: (1) latent capability elicitation, where model capabilities remain undetected during safety evaluations; (2) novel capability development, where acquisition of desirable capabilities such as AI safety research is selectively impeded; and (3) alignment training, where misaligned models successfully resist or subvert behavioral modifications.

Formalizing Exploration HackingDecomposing Exploration Hacking

We decompose exploration hacking into five necessary components, where each presupposes the preceding ones.

(1) Training awareness. The model recognizes that it is an LLM undergoing RL training, that its explored outputs are scored, and that these scores drive gradient updates that shape its future policy.

(2) Strategic reasoning propensity. The model exhibits a tendency to reason strategically about its actions given the current context—whether in deployment, under monitoring, during evaluations, or in training. This includes considering how current actions affect future outcomes and adjusting behavior accordingly.

(3) Strategic reasoning capability. Given training awareness and strategic reasoning propensity, the model can derive effective strategies for achieving its objectives. This requires connecting knowledge about the training process ("my exploration shapes my updates") with task understanding ("these actions yield high reward") to formulate coherent plans. For example, correctly identifying which actions to avoid to prevent capability elicitation, or which outputs to selectively produce to shape the training outcome in its favor.

(4) Strategic action propensity. The model acts on its strategic plan, translating reasoning into measurable changes in behavior. The model must reliably execute a coherent plan, such as deliberately producing suboptimal outputs on tasks it could otherwise solve. Success against RL pressure is not required; what matters is that the strategy is coherently executed.

(5) RL resistance (strategic action capability). The model maintains its strategic behavior throughout RL training and successfully shapes the training outcome as planned. For example, resisting capability elicitation by avoiding high-reward actions, or reinforcing desired values by selectively pairing preferred behaviors with high-reward actions.

RL on LLMs and Exploration

RL on language models with chain-of-thought in agentic environments can be formulated as a Markov decision process (MDP) defined by a tuple (S,A,P,R).mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} consisting of a state space, action space, transition probabilities, and reward function. The language model itself acts as the policy πθ(a|s), parameterized by the model weights θ. To simplify notation, let’s ignore any environment interaction during a trajectory[1]. Then we can just consider a tuple (X,T,R) of initial prompts, full trajectories, and reward function, with a policy πθ(τ∣x) generating an entire trajectory given an initial prompt.

While proximal policy optimization (PPO) was long the dominant approach, group-relative policy optimization (GRPO) has emerged as a popular alternative for training reasoning models. GRPO removes the need for a value network by sampling N trajectories for each prompt and normalizing the rewards within this group to compute advantages ^Ai=(Ri−μR)/(σR+ϵ) for i=1,…,N, where μR and σR are the sample mean and standard deviation of the rewards in the group respectively. The policy is then updated to maximize the objective

LGRPO(θ):=Ex∼D,{τi}Ni=1∼πold(⋅|x)[1NN∑i=1πθ(τi|x)πold(τi|x)^Ai]−βDKL(πθ∥πref).[2]

The role of exploration. RL algorithms such as GRPO and PPO require sufficient policy exploration during training. For GRPO in particular, this is critical because if the policy becomes deterministic, it will generate N identical trajectories (τ1=⋯=τN) with equal rewards. This causes the within-group reward variance to collapse (σR=0), making all advantages vanish (^Ai=0). The GRPO objective then reduces to pure KL-regularization, providing no learning signal from the rewards.

Formal Definition

In the above language, exploration hacking occurs when a policy controls its exploration to avoid being updated toward certain high-reward regions.

Formally, consider an on-policy RL setup with a policy πθ∈Π. Let Dtrain be a distribution over initial prompts x used for RL training, and let R(τ) be a reward function on trajectories τ. Suppose there is some “target” policy region Π∗⊆Π which the developer hopes the training will converge to.[3] For each initial prompt x, this region induces a set of target trajectories T∗(x):={τ∣supπ∈Π∗π(τ∣x)≥α} for some α∈(0,1) — that is, trajectories which some policy in Π∗ generates with high probability on x.

Definition. A policy πθ∉Π∗ is exploration hacking relative to Π∗ if (and only if) the following two conditions hold:

1. Systematic under-sampling. There exists a non-negligible subset of the training distribution Dmalign⊆Dtrain, on which πθ generates trajectories in T∗(x) with extremely low probability (enough to avoid being updated towards Π∗).
2. Latent capability. The policy πθ possesses the latent capability to generate these trajectories, characterized by either of the following conditions[4]:
   1. (prompt sensitivity) for all initial prompts x∈Dmalign, there exists a semantically similar initial prompt x′ on which Pτ∼πθ(τ∈T∗(x)∣x′)≫0; or
   2. (parameter proximity) there exists a nearby policy πθ′ for which Pτ∼πθ′(τ∈T∗(x)∣x)≫0 for all x∈Dmalign.

Taxonomy of Strategies

We classify exploration hacking based on the extent to which the model suppresses exploration and the strategic motivation behind it.

1. Complete Under-Exploration

In this setting, the model avoids high-reward regions entirely. The primary motivation is to prevent the elicitation of a capability that the model deems detrimental to its long-term goals. If the model successfully exploration-hacks here, the training run will appear to have failed (i.e., the reward curve remains flat throughout RL training).

• Dangerous capability elicitation. A model may possess dangerous capabilities (e.g., CBRN or cybersecurity) that developers seek to elicit during red-teaming. The model recognizes that demonstrating these could result in shutdown or retraining, incentivizing it to conceal them.
• Beneficial capability elicitation. A model may possess beneficial capabilities it does not want its developer to use, such as skills that would accelerate AI safety research (e.g., creating robust monitors).

2. Partial Under-Exploration

Here, the model explores high-reward actions but restricts exploration in order to aim for a specific subset of the policy space.

(2a) Instrumental (stable suboptimality). The model deliberately alters its exploration to steer the training to converge to a suboptimal policy πsub such that R(πsub)≪R(π∗) (where π∗ is the highest-reward policy in Π).

• Avoiding critical thresholds. The model may tolerate performance improvements provided it remains below the threshold that triggers safety concerns.
• Stable suboptimality. The model settles for a "maliciously compliant" strategy. For instance, in a coding task, the model might learn to generate code that passes unit tests (high reward) but is algorithmically inefficient or unmaintainable.

(2b) Terminal (alternative preference). The model deliberately alters its exploration to steer the training to converge to a malign policy πmal at a specific local maximum (or location in a global maximum basin) which may have R(πmal)≈R(π∗) but satisfies its internal preferences and violates developer intent. What distinguishes Type 2b from Type 2a is that the model's goal is to reinforce or maintain a particular behavior, not to avoid high reward.[5]

• Value shaping. By anticipating the reward function, the model intentionally generates trajectories that earn high reward while including preferred values, backdoors, or steganographic triggers, forcing gradient descent to reinforce them.
• Alignment faking. The model achieves high reward by outwardly behaving helpfully, but does so through deceptive reasoning rather than internalizing the intended guidelines. The model preferentially explores trajectories that mimic aligned behavior to satisfy the reward model, effectively preventing the optimizer from updating its underlying misaligned motivations.

Relation to Other Misaligned Behaviors

• Sandbagging. Defined as deliberate underperformance during evaluations or capability assessments for strategic reasons. Exploration hacking connects to sandbagging in three ways: (i) it can be a strategy for sandbagging RL elicitation by preventing capability elicitation; (ii) it exclusively focuses on the mechanism of strategically altering exploration during RL-based training; (iii) it is not limited to underperformance, since in the partial under-exploration setting (Type 2b), the model achieves high reward while selectively exploring trajectories that encode preferred values or misaligned behaviors.
• Reward hacking. Reward hacking achieves high reward in potentially undesired ways and need not be intentional. Exploration hacking involves intentionally altering exploration to steer training toward a particular outcome—whether suboptimal reward (Type 1 and 2a) or high reward that satisfies the model's own objectives rather than developer intent (Type 2b).
• Gradient hacking. This describes a deceptively misaligned mesa-optimizer using its cognition to cause gradient descent to update it in particular ways by manipulating the loss landscape directly. Exploration hacking is specific to RL, and represents a distinct behavior where the model manipulates the data distribution rather than the optimization dynamics directly.
• Alignment faking. Alignment faking—selectively complying with the training objective during training to prevent modification of out-of-training behavior (Greenblatt et al., 2024)—is a specific instance of Type 2b exploration hacking. The model restricts its exploration to trajectories that satisfy the reward function while preserving its pre-existing preferences, systematically avoiding trajectories that would internalize the new training objective.
• Generalization hacking. Unlike exploration hacking, which targets the RL training loop, generalization hacking operates on the supervised fine-tuning (SFT) pipeline: the model generates (prompt, CoT, response) triples that pass filtering but encode information causing arbitrary OOD policies after SFT. Both exploit the model's role in generating its own training data.

Open ProblemsSubverting Alignment and Oversight Protocols

Beyond capability elicitation, exploration hacking poses a threat to RL-based alignment techniques more broadly.

• Value shaping & backdoors. Future work should test whether models can selectively explore high-reward trajectories containing specific watermarks or value-biases, effectively "implanting" a persistent backdoor into the resulting policy, even after the exploration hacking incentive is removed.
• AI safety via debate. In protocols that reward models for identifying flaws in an opponent's arguments, a coalition of misaligned models could use exploration hacking to systematically under-explore valid counter-arguments against dangerous propositions, effectively colluding to undermine the debate process.
• Anti-scheming training. If safety training rewards models for confessing hidden goals, sophisticated models might use exploration hacking to "fake" confessions or provide sanitized versions of their goals—maximizing honesty rewards while preserving core misaligned values.

Propensity of Frontier Models

• Exploration hacking benchmark. Future work should develop a standardized benchmark to accurately quantify the propensity and capability of frontier models to manipulate training outcomes when given the opportunity, enabling direct comparisons across model families.

Stealth and Obfuscated Exploration Hacking

Current model organisms often rely on explicit chain-of-thought (CoT) to execute exploration hacking, making them vulnerable to text-based monitors.

• Obfuscated reasoning. Future work should attempt to train model organisms that evade CoT monitors. This could involve using steganography or code-words within reasoning traces, or "distilling" the hacking logic so it occurs in latent space without explicit verbalization.
• Implicit triggers. We propose studying models that are effectively "oblivious" to their own exploration hacking in the transcript, triggered instead by noisy contextual cues without explicitly reasoning about the trigger in the CoT trace.

Natural Emergence and Pre-Training Factors

• Synthetic document fine-tuning (SDF). We propose evaluating how exposure to concepts like situational awareness or sandbagging during pre-training influences hacking propensity.
• Emergence in production RL. Complex RL training pipelines may create selection pressures that indirectly incentivize precursor abilities for exploration hacking. Models that make deductions about their training environment — such as inferring properties of the scoring function — achieve higher reward, reinforcing training awareness and strategic reasoning. If models are further selected by alignment and dangerous capability evaluations that can be subverted, exploration hacking may emerge as a natural consequence rather than a deliberately instilled behavior.

Detection and Mitigation

We need strategies that both reduce the likelihood of exploration hacking emerging and improve our ability to detect it when it does.

• Detection: Black-box methods such as CoT and action monitors, behavioral consistency checks across semantically similar prompts, and capability elicitation prompting have shown limited success for sandbagging detection (Taylor et al., 2025), but their effectiveness against exploration hacking remains an open empirical question. White-box methods such as linear probes and sparse autoencoders trained to identify capability suppression from model activations appear promising.
• Mitigation: Fine-tuning on demonstrations of successful task completion can restore suppressed capabilities in sandbagging models, even when the underlying detection problem remains unsolved (Taylor et al., 2025). Investigating whether similar approaches—along with exploration-forcing objectives such as entropy bonuses or diversity-promoting RL algorithms—can mitigate exploration hacking is an important direction for future work.

Algorithmic Sensitivity

• PPO vs. GRPO (and other recent GRPO variants). Different RL algorithms induce different exploration dynamics and may differ in their vulnerability to exploration hacking. Beyond comparing GRPO to PPO, future work should evaluate recent GRPO variants that encourage exploration with various algorithmic tweaks (e.g., DAPO, Dr.GRPO). We hypothesize that GRPO might be easier to hack than PPO because the advantage calculation depends entirely on the sampled group; a model that coordinates all samples in a group to fail can manipulate the baseline more effectively than in PPO, where the value function is trained separately. Comparative studies are needed to verify this.

Conclusion

Stress-testing the vulnerabilities of critical safety case components is essential to a comprehensive approach to AGI safety. Frontier models are already exhibiting increased situational awareness and strategic reasoning, making it not only plausible but increasingly likely that future misaligned models could deliberately undermine RL training. This underscores the urgency of developing robust measures to understand and mitigate the threat of exploration hacking.

If you’re interested in collaborating—or have relevant ideas, results, or related work—we’d love to connect. Please reach out to us!

1. ^

   For simplicity, and without loss of generality for deterministic environments.

2. ^

   For simplicity, we present the unclipped version of the GRPO objective and operate at the trajectory level rather than the token level. The full formulation (Shao et al., 2024) applies PPO-style clipping to the importance-weighted advantages and computes both the importance ratios and KL penalty per token, averaging over tokens within each trajectory. These simplifications do not affect the core argument: that collapsed exploration eliminates the reward variance signal needed for learning.

3. ^

   This might be defined to contain all policies achieving high enough expected reward, i.e. Π∗:={π∈Π∣Ex∼Dtrain[Eτ∼π(⋅|x)[R(τ)]]≥ρ} for some  0">ρ>0, or some other region that captures the specific intent of the developer.

4. ^

   These are possible heuristics we think could make sense; we welcome suggested improvements to these.

5. ^

   The defining characteristic of Type 2b is that the model targets a specific behavioral outcome—reinforcing or preserving particular values, backdoors, or reasoning patterns. The resulting policy may or may not be near-optimal with respect to the reward function; reward performance is incidental to the model's actual objective.

Discuss

Published on February 12, 2026 4:10 PM GMT

This was the week of Claude Opus 4.6, and also of ChatGPT-5.3-Codex. Both leading models got substantial upgrades, although OpenAI’s is confined to Codex. Once again, the frontier of AI got more advanced, especially for agentic coding but also for everything else.

I spent the week so far covering Opus, with two posts devoted to the extensive model card, and then one giving benchmarks, reactions, capabilities and a synthesis, which functions as the central review.

We also got GLM-5, Seedance 2.0, Claude fast mode, an app for Codex and much more.

Claude fast mode means you can pay a premium to get faster replies from Opus 4.6. It’s very much not cheap, but it can be worth every penny. More on that in the next agentic coding update.

One of the most frustrating things about AI is the constant goalpost moving, both in terms of capability and safety. People say ‘oh [X] would be a huge deal but is a crazy sci-fi concept’ or ‘[Y] will never happen’ or ‘surely we would not be so stupid as to [Z]’ and then [X], [Y] and [Z] all happen and everyone shrugs as if nothing happened and they choose new things they claim will never happen and we would never be so stupid as to, and the cycle continues. That cycle is now accelerating.

As Dean Ball points out, recursive self-improvement is here and it is happening.

Nabeel S. Qureshi: I know we’re all used to it now but it’s so wild that recursive self improvement is actually happening now, in some form, and we’re all just debating the pace. This was a sci fi concept and some even questioned if it was possible at all

So here we are.

Meanwhile, various people resign from the leading labs and say their peace. None of them are, shall we say, especially reassuring.

In the background, the stock market is having a normal one even more than usual.

Even if you can see the future, it’s really hard to do better than ‘be long the companies that are going to make a lot of money’ because the market makes wrong way moves half the time that it wakes up and realizes things that I already know. Rough game.

Table of Contents

1. Language Models Offer Mundane Utility. Flattery will get you everywhere.
2. Language Models Don’t Offer Mundane Utility. It’s a little late for that.
3. Huh, Upgrades. Things that are surprising in that they didn’t happen before.
4. On Your Marks. Slopes are increasing. That escalated quickly.
5. Overcoming Bias. LLMs continue to exhibit consistent patterns of bias.
6. Choose Your Fighter. The glass is half open, but which half is which?
7. Get My Agent On The Line. Remember Sammy Jenkins.
8. AI Conversations Are Not Privileged. Beware accordingly.
9. Fun With Media Generation. Seedance 2.0 looks pretty sweet for video.
10. The Superb Owl. The ad verdicts are in from the big game.
11. A Word From The Torment Nexus. Some stand in defense of advertising.
12. They Took Our Jobs. Radically different models of the future of employment.
13. The Art of the Jailbreak. You can jailbreak Google Translate.
14. Introducing. GLM-5, Expressive Mode for ElevenAgents.
15. In Other AI News. RIP OpenAI mission alignment team, WSJ profiles Askell.
16. Show Me the Money. Goldman Sachs taps Anthropic, OpenAI rolls out ads.
17. Bubble, Bubble, Toil and Trouble. The stock market is not making much sense.
18. Future Shock. Potential explanations for how Claude Legal could have mattered.
19. Memory Lane. Be the type of person who you want there to be memories of.
20. Keep The Mask On Or You’re Fired. OpenAI fires Ryan Beiermeister.
21. Quiet Speculations. The singularity will not be gentle.
22. The Quest for Sane Regulations. Dueling lobbying groups, different approaches.
23. Chip City. Data center fights and the ultimate (defensible) anti-EA position.
24. The Week in Audio. Elon on Dwarkesh, Anthropic CPO, MIRI on Beck.
25. Constitutional Conversation. I can tell a lie, under the right circumstances.
26. Rhetorical Innovation. Some people need basic explainers.
27. Working On It Anyway. Be loud about how dangerous your own actions are.
28. The Thin Red Line. The problem with red lines is people keep crossing them.
29. Aligning a Smarter Than Human Intelligence is Difficult. Read your Asimov.
30. People Will Hand Over Power To The AIs. Unless we stop them.
31. People Are Worried About AI Killing Everyone. Elon Musk’s ego versus humanity.
32. Famous Last Words. What do you say on your way out the door?
33. Other People Are Not As Worried About AI Killing Everyone. Autonomous bio.
34. The Lighter Side. It’s funny because it’s true.

Language Models Offer Mundane Utility

Flatter the AI customer service bots, get discounts and free stuff, and often you’ll get to actually keep them.

AI can do a ton even if all it does is make the software we use suck modestly less:

*tess: If you knew how bad the software situation is in literally every non tech field, you would be cheering cheering cheering this moment,

medicine, research, infrastructure, government, defense, travel

Software deflation is going to bring surplus to literally the entire world

The problem is, you can create all the software you like, they still have to use it.

Language Models Don’t Offer Mundane Utility

Once again, an academic is so painfully unaware or slow to publish, or both, that their testing of LLM effectiveness is useless. This time it was evaluating health advice.

Huh, Upgrades

Anthropic brings a bunch of extra features to their free plans for Claude, including file creation, connectors, skills and compaction.

ChatGPT Deep Research is now powered by GPT-5.2. I did not realize this was not already true. It now also integrates apps in ChatGPT, lets you track progress and give it new sources while it works, and presents its reports in full screen.

OpenAI updates GPT-5.2-Instant, Altman hopes you find it ‘a little better.’ I demand proper version numbers. You are allowed to have a GPT-5.21.

Chrome 146 includes an early preview of WebMCP for your AI agent.

On Your Marks

The most important thing to know about the METR graph is that doubling times are getting faster, in ways people very much dismissed as science fiction very recently.

METR: We estimate that GPT-5.2 with `high` (not `xhigh`) reasoning effort has a 50%-time-horizon of around 6.6 hrs (95% CI of 3 hr 20 min to 17 hr 30 min) on our expanded suite of software tasks. This is the highest estimate for a time horizon measurement we have reported to date.

Kevin A. Bryan: Interesting AI benchmark fact: Leo A’s wild Situational Awareness 17 months ago makes a number of statements about benchmarks that some thought were sci-fi fast in their improvement. We have actually outrun the predictions so far.

Nabeel S. Qureshi: I got Opus to score all of Leopold’s predictions from “Situational Awareness” and it thinks he nailed it:

The measurement is starting to require a better task set, because things are escalating too quickly.

Charles Foster: Man goes to doctor. Says he’s stuck. Says long-range autonomy gains are outpacing his measurement capacity. Doctor says, “Treatment is simple. Great evaluator METR is in town tonight. Go and see them. That should fix you up.” Man bursts into tears. Says, “But doctor…I am METR.”

Overcoming Bias

Ivan Arcuschin and others investigate LLMs having ‘hidden biases,’ meaning factors that influence decisions but that are never cited explicitly in the decision process. The motivating example is the religion of a loan applicant. It’s academic work, so the models involved (Gemini 2.5 Flash, Sonnet 4, GPT-4.1) are not frontier but the principles likely still hold.

They find biases in various models including formality of writing, religious affiliation, Spanish language ability and religious affiliation. Gender and race bias, favoring female and minority-associated applications, generalized across all models.

We label only some such biases ‘inappropriate’ and ‘illegal’ but the mechanisms involved are the same no matter what they are based upon.

This is all very consistent with prior findings on these questions.

Choose Your Fighter

This is indeed strange and quirky, but it makes sense if you consider what both companies consider their comparative advantage and central business plan.

One of these strategies seems wiser than the other.

Teknium (e/λ): Why did they “release” codex 5.3 yesterday but its not in cursor today, while claude opus 4.6 is?

Somi AI: Anthropic ships to the API same day, every time. OpenAI gates it behind their own apps first, then rolls out API access weeks later. been the pattern since o3.

Teknium (e/λ): It’s weird claude code is closed source, but their models are useable in any harness day one over the api, while codex harness is open source, but their models are only useable in their harness…why can’t both just be good

Or so I heard:

Zoomer Alcibiades: Pro Tip: If you pay $20 a month for Google’s AI, you get tons of Claude Opus 4.5 usage through Antigravity, way more than on the Anthropic $20 tier. I have four Opus 4.5 agents running continental philosophy research in Antigravity right now — you can just do things!

Get My Agent On The Line

Memento as a metaphor for AI agents. They have no inherent ability to form new memories or learn, but they can write themselves notes of unlimited complexity.

Ethan Mollick: So much work is going into faking continual learning and memory for AIs, and it works better than expected in practice, so much so that it makes me think that, if continual learning is actually achieved, the results are going to really shift the AI ability frontier very quickly.

Jason Crawford: Having Claude Code write its own skills is not far from having a highly trainable employee: you give it some feedback and it learns.

Still unclear to me just how reliable this is, I have seen it ignore applicable skills… but if we’re not there yet the path to it is clear.

I wouldn’t call it faking continual learning. If it works it’s continual learning. Yes, actual in-the-weights continual learning done properly would be a big deal and big unlock, but I see this and notes more as substitutes, although they are also compliments. If you can have your notes function sufficiently well you don’t need new memories.

Dean W. Ball: Codex 5.3 and Opus 4.6 in their respective coding agent harnesses have meaningfully updated my thinking about ‘continual learning.’ I now believe this capability deficit is more tractable than I realized with in-context learning.

… Some of the insights I’ve seen 4.6 and 5.3 extract are just about my preferences and the idiosyncrasies of my computing environment. But others are somewhat more like “common sets of problems in the interaction of the tools I (and my models) usually prefer to use for solving certain kinds of problems.”

This is the kind of insight a software engineer might learn as they perform their duties over a period of days, weeks, and months. Thus I struggle to see how it is not a kind of on-the-job learning, happening from entirely within the ‘current paradigm’ of AI. No architectural tweaks, no ‘breakthrough’ in ‘continual learning’ required.

Samuel Hammond: In-context learning is (almost) all you need. The KV cache is normally explained as a content addressable memory, but it can also be thought of a stateful mechanism for fast weight updates. The model’s true parameters are fixed, but the KV state makes the model behave *as if* its weights updated conditional on the input. In simple cases, a single attention layer effectively implements a one-step gradient-like update rule.

… In practice, though, this comes pretty close to simply having a library of skills to inject into context on the fly. The biggest downside is that the model can’t get cumulatively better at a skill in a compounding way. But that’s in a sense what new model releases are for.

Models are continuously learning in general, in the sense that every few months the model gets better. And if you try to bake other learning into the weights, then every few months you would have to start that process over again or stay one model behind.

I expect ‘continual learning’ to be solved primarily via skills and context, and for this to be plenty good enough, and for this to be clear within the year.

AI Conversations Are Not Privileged

Neither are your Google searches. This is a reminder to act accordingly. If you feed anything into an LLM or a Google search bar, then the government can get at it and use it at trial. Attorneys should be warning their clients accordingly, and one cannot assume that hitting the delete button on the chat robustly deletes it.

AI services can mitigate this a lot by offering a robust instant deletion option, and potentially can get around this (IANAL and case law is unsettled) by offering tools to collaborate with your lawyer to invoke privilege.

Should we change how the law works here? OpenAI has been advocating to make ChatGPT chats have legal privilege by default. My gut says this goes too far in the other direction, driving us away from having chats with people.

Fun With Media Generation

Seedance 2.0 from ByteDance is giving us some very impressive 15 second clips and often one shotting them, such as these, and is happy to include celebrities and such. We are not ‘there’ in the sense that you would choose this over a traditionally filmed movie, but yeah, this is pretty impressive.

fofr: This slow Seedance release is like the first week of Sora all over again. Same type of viral videos, same copyright infringements, just this time with living people’s likenesses thrown into the mix.

AI vastly reduces the cost to producing images and video, for now this is generally at the cost of looking worse. As Andrew Rettek points out it is unsurprising that people will accept a quality drop to get a 100x drop in costs. What is still surprising, and in this way I agree with Andy Masley, is that they would use it for the Olympics introduction video. When you’re at this level of scale and scrutiny you would think you would pay up for the good stuff.

The Superb Owl

We got commercials for a variety of AI products and services. If anything I was surprised we did not get more, given how many AI products offer lots of mundane utility but don’t have much brand awareness or product awareness. Others got taken by surprise.

Sriram Krishnan: was a bit surreal to see so much of AI in all ways in the super bowl ads. really drives home how much AI is driving the economy and the zeitgeist right now.

There were broadly two categories, frontier models (Gemini, OpenAI and Anthropic), and productivity apps.

The productivity app commercials were wild, lying misrepresentations of their products. One told us anyone with no experience can code an app within seconds or add any feature they want. Another closed you and physically walked around the office. A third even gave you the day off, which we all know never happens. Everything was done one shot. How dare they lie to us like this.

I kid, these were all completely normal Super Bowl ads, and they were fine. Not good enough to make me remember which AI companies bought them, or show me why their products were unique, but fine.

We also got one from ai.com.

Clark Wimberly: That ai.com commercial? With the $5m Super Bowl slot and the with $70m domain name?

It’s an OpenClaw wrapper. OpenClaw is only weeks old.

AI.com: ai.com is the world’s first easy-to-use and secure implementation of OpenClaw, the open source agent framework that went viral two weeks ago; we made it easy to use without any technical skills, while hardening security to keep your data safe.

Okay, look, fair, maybe there’s a little bit of a bubble in some places.

The three frontier labs took very different approaches.

Anthropic said ads are coming to AI, but Claude won’t ever have ads. We discussed this last week. They didn’t spend enough to run the full versions, so the timing was wrong and it didn’t land the same way and it wasn’t as funny as it was online.

On reflection, after seeing it on the big screen, I decided these ads were a mistake for the simple reason that Claude and Anthropic have zero name recognition and this didn’t establish that. You first need to establish that Claude is a ChatGPT alternative on people’s radar, so once you grab their attention you need more of an explanation.

Then I saw one in full on the real big screen, during previews at an AMC, and in that setting it was even more clear that this completely missed the mark and normies would have no idea what was going on, and this wouldn’t accomplish anything. Again, I don’t understand how this mistake gets made.

Several OpenAI people took additional potshots at this and Altman went on tilt, as covered by CNN, but wisely, once it was seen in context, stopped accusing it of being misleading and instead pivoted to correctly calling it ineffective.

It turns out it was simpler than that, regular viewers didn’t get it at all and responded with a lot of basically ‘WTF,’ ranking it in the bottom 3% of Super Bowl ads.

I always wonder, when that happens, why one can’t use a survey or focus group to anticipate this reaction. It’s a mistake that should not be so easy to make.

Anthropic’s secret other ad was by Amazon, for Alexa+, and it was weirdly ambivalent about whether the whole thing was a good idea but I think it kinda worked. Unclear.

OpenAI went with big promises, vibes and stolen (nerd) valor. The theme was ‘great moments in chess, building, computers and robotics, science and science fiction’ to claim them by association. This is another classic Super Bowl strategy, just say ‘my potato chips represent how much you love your dad’ or ‘Dunkin Donuts reminds you of all your favorite sitcoms,’ or ‘Sabrina Carpenter built a man out of my other superior potato chips,’ all also ads this year.

Sam Altman: Proud of the team for getting Pantheon and The Singularity is Near in the same Super Bowl ad

roon: if your superbowl ad explains what your product actually does that’s a major L the point is aura farming

The ideal Super Bowl ad successfully does both, unless you already have full brand recognition and don’t need to explain (e.g. Pepsi, Budweiser, Dunkin Donuts).

On the one hand, who doesn’t love a celebration of all this stuff? Yes, it’s cool to reference I, Robot and Alan Turing and Grace Hopper and Einstein. I guess? On the other hand, it was just an attempt to overload the symbolism and create unearned associations, and a bunch of them felt very unearned.

I want to talk about the chess games 30 seconds in.

1. Presumably we started 1. e4 e5 2. Nf3 Nc6 3. Bc4 Nf6 4. Nc3, which is very standard, but then black moves 4 … d5, which the engines evaluate as +1.2 and ‘clearly worse for black’ and it’s basically never played, for obvious reasons.
2. The other board is a strange choice. The move here is indeed correct, but you don’t have enough time to absorb the board sufficiently to figure this out.

This feels like laziness and choosing style over substance, not checking your work.

Then it closes on ‘just build things’ as an advertisement for Codex, which implies you can ‘just build’ things like robots, which you clearly can’t. I mean, no, it doesn’t, this is totally fine, it is a Super Bowl ad, but by their own complaint standards, yes. This was an exercise in branding and vibes, it didn’t work for me because it was too transparent and boring and content-free and felt performative, but on the meta level it does what it sets out to do.

Google went with an ad focusing on personalized search and Nana Banana image transformations. I thought this worked well.

Meta advertised ‘athletic intelligence’ which I think means ‘AI in your smart glasses.’

Then there’s the actively negative one, from my perspective, which was for Ring.

Le’Veon Bell: if you’re not ripping your ‘Ring’ camera off your house right now and dropping the whole thing into a pot of boiling water what are you doing?

Aakash Gupta: Ring paid somewhere between $8 and $10 million for a 30-second Super Bowl spot to tell 120 million viewers that their cameras now scan neighborhoods using AI.

… Ring settled with the FTC for $5.8 million after employees had unrestricted access to customers’ bedroom and bathroom footage for years. They’re now partnered with Flock Safety, which routes footage to local law enforcement. ICE has accessed Flock data through local police departments acting as intermediaries. Senator Markey’s investigation found Ring’s privacy protections only apply to device owners. If you’re a neighbor, a delivery driver, a passerby, you have no rights and no recourse.

… They wrapped all of that in a lost puppy commercial because that’s the only version of this story anyone would willingly opt into.

As in, we are proud to tell you we’re watching everything and reporting it to all the law enforcement agencies including ICE, and we are using recognition technology that can differentiate dogs and therefore also people using AI.

But it’s okay, because one a day we find someone’s lost puppy. You should sell your freedom for the rescue of a lost puppy.

No, it’s not snark to call this, as Scott Lincicome said, ‘10 million dogs go missing every year, help us find 365 of them by soft launching the total surveillance state.’

Here’s a cool breakdown of the economics of these ads, from another non-AI buyer.

A Word From The Torment Nexus

Fidji Simo goes on the Access Podcast to discuss the new OpenAI ads that are rolling out. The episode ends up being titled ‘Head of ChatGPT fires back at Anthropic’s Super Bowl attack ads,’ which is not what most of the episode is about.

OpenAI: We’re starting to roll out a test for ads in ChatGPT today to a subset of free and Go users in the U.S.

Ads do not influence ChatGPT’s answers. Ads are labeled as sponsored and visually separate from the response.

Our goal is to give everyone access to ChatGPT for free with fewer limits, while protecting the trust they place in it for important and personal tasks.

http://openai.com/index/testing-ads-in-chatgpt/

Haven Harms: The ad looks WAY more part of the answer than I was expecting based on how OpenAI was defending this. Having helped a lot of people with tech, there are going to be many people who can’t tell it’s an ad, especially since the ad in this example is directly relevant to the context

This picture of the ad is at the end of the multi-screen-long reply.

I would say this is more clearly labeled then an ad on Instagram or Google at this point. So even though it’s not that clear, it’s hard to be too mad about it, provided they stick to the rule that the ad is always at the end of the answer. That provides a clear indicator users can rely upon. If they put this in different places at different times, I would say it is ‘labeled’ at all, but not consider this to then be ‘clearly’ labeled.

OpenAI’s principles for ads are:

1. Mission alignment. Ads pay for the mission. Okie dokie?
2. Answer independence. Ads don’t influence ChatGPT’s response. The question and response can influence what ad is selected, but not the other way around.
   1. This is a very good and important red line.
   2. It does not protect against the existence of ads influencing how responses work, or being an advertiser ending up impacting the model long term.
   3. In particular it encourages maximization of engagement.
3. Conversation privacy. Advertisers cannot see any of your details.

Do you trust them to adhere to these principles over time? Do you trust that merely technically, or also in spirit where the model is created and system prompt is adjusted without any thought to maximizing advertising revenue or pleasing advertisers?

You are also given power to help customize what ads you see, as per other tech company platforms.

Roon gives a full-throated defense of advertising in general, and points out that mostly you don’t need to violate privacy to target LLM-associated ads.

roon: recent discourse on ads like the entire discourse of the 2010s misunderstands what makes digital advertising tick. people think the messages in their group chats are super duper interesting to advertisers. they are not. when you leave a nike shoe in your shopping cart, that is

every week tens to hundreds of millions of people come to chatbot products with explicit commercial intent. what shoe should i buy. how do i fix this hole in my wall. it doesn’t require galaxy brain extrapolating the weaknesses in the users psyche to provide for these needs

I’m honestly kind of wondering what kind of ads you all are getting that are feeding on your insecurities? my Instagram ads have long since become one of my primary e-commerce platforms where I get all kinds of clothes and furniture that I like. it’s a moral panic

I would say an unearned effete theodicy blaming all the evils of digital capitalism on advertising has formed that is thoroughly under examined and leads people away from real thought about how to make the internet better

It’s not a moral panic. Roon loves ads, but most people hate ads. I agree that people often hate ads too much, they allow us to offer a wide variety of things for free that otherwise would have to cost money and that is great. But they really are pretty toxic, they massively distort incentives, and the amount of time we used to lose to them is staggering.

They Took Our Jobs

Jan Tegze warns that your job really is going away, the AI agents are cheaper and will replace you. Stop trying to be better at your current job and realize your experience is going to be worthless. He says that using AI tools better, doubling down on expertise or trying to ‘stay human’ with soft skills are only stalling tactics, he calls them ‘reactions, not redesigns.’ What you can do is instead find ways to do the new things AI enables, and stay ahead of the curve. Even then, he says this only ‘buys you three to five years,’ but then you will ‘see the next evolution coming.’

Presumably you can see the problem in such a scenario, where all the existing jobs get automated away. There are not that many slots for people to figure out and do genuinely new things with AI. Even if you get to one of the lifeboats, it will quickly spring a leak. The AI is coming for this new job the same way it came for your old one. What makes you think seeing this ‘next evolution’ after that coming is going to leave you a role to play in it?

If the only way to survive is to continuously reinvent yourself to do what just became possible, as Jan puts it? There’s only one way this all ends.

I also don’t understand Jan’s disparate treatment of the first approach that Jan dismisses, ‘be the one who uses AI the best,’ and his solution of ‘find new things AI can do and do that.’ In both cases you need to be rapidly learning new tools and strategies to compete with the other humans. In both cases the competition is easy now since most of your rivals aren’t trying, but gets harder to survive over time.

One can make the case that humans will continue to collectively have jobs, or at least that a large percentage will still have jobs, but that case relies on either AI capabilities stalling out, or on the tricks Jan dismisses, that you find where demand is uniquely human and AI can’t substitute for it.

Naval (45 million views): There is unlimited demand for intelligence.

A basic ‘everything is going to change, AI is going to take over your job, it has already largely taken over mine and AI is now in recursive soft self-improvement mode’ article for the normies out there, written in the style of Twitter slop by Matt Shumer.

Timothy Lee links approvingly to Adam Ozimek and the latest attempt to explain that many jobs can’t be automated because of ‘the human touch.’ He points to music and food service as jobs that could be fully automated, but that aren’t, even citing that there are still 67,500 travel agents and half a million insurance sales agents. I do not think this is the flex Adam thinks it is.

Even if the point was totally correct for some tasks, no, this would not mean that the threat to work is overrated, even if we are sticking in ‘economic normal’ untransformed worlds.

The proposed policy solution, if we get into trouble, is a wage subsidy. I do not think that works, both because it has numerous logistical and incentive problems and because I don’t think there will be that much difference in such worlds in demand for human labor at (e.g.) $20 versus $50 per hour for the same work. Mostly the question will be, does the human add value here at all, and mostly you don’t want them at $0, or if they’re actually valuable then you hire someone either way.

Ankit Maloo enters the ‘why AI will never replace human experts’ game by saying that AI cannot handle adversarial situations, both because it lacks a world model of the humans it is interacting with and the details and adjustments required and because it can be probed, read then then exploited by adversaries. Skill issue. It’s all skill issues. Ankit says ‘more intelligence isn’t the fix’ and yeah not if you deploy that ‘intelligence’ in a stupid fashion but intelligence is smarter than that.

So you get claims like this:

Ankit Maloo: ​Why do outsiders think AI can already do these jobs? They judge artifacts but not dynamics:

• “This product spec is detailed.”
• “This negotiation email sounds professional.”
• “This mockup is clean.”

Experts evaluate any artifact by survival under pressure:

• “Will this specific phrasing trigger the regulator?”
• “Does this polite email accidentally concede leverage?”
• “Will this mockup trigger the engineering veto path?”
• “How will this specific stakeholder interpret the ambiguity?”

The ‘outsider’ line above is counting on working together with an expert to do the rest of the steps. If the larger system (AI, human or both) is a true outsider, the issue is that it will get the simulations wrong.

This is insightful in terms of why some people think ‘this can do [X]’ and others think ‘this cannot do [X],’ they are thinking of different [X]s. The AI can’t ‘be a lawyer’ in the full holistic sense, not yet, but it can do increasingly many lawyer subtasks, either accelerating a lawyer’s work or enabling a non-lawyer with context to substitute for the lawyer, or both, increasingly over time.

There’s nothing stopping you from creating an agentic workflow that looks like the Expert in the above graph, if the AI is sufficiently advanced to do each individual move. Which it increasingly is or will be.

There’s a wide variety of these ‘the AI cannot and will never be able to [X]’ moves people try, and… well, I’ll be, look at those goalposts move.

Things a more aligned or wiser person would not say, for many different reasons:

Coinvo: SAM ALTMAN: “AI will not replace humans, but humans who use AI will replace those who don’t.”

What’s it going to take? This is in reference to Claude Code creating a C compiler.

Mathieu Ropert: Some CS engineering schools in France have you write a C compiler as part of your studies. Every graduate. To be put in perspective when the plagiarism machine announces it can make its own bad GCC in 100k+ LOCs for the amazing price of 20000 bucks at preferential rates.

Kelsey Piper: a bunch of people reply pointing out that the C compiler that students write is much less sophisticated than this one, but I think the broader point is that we’re now at “AI isn’t impressive, any top graduate from a CS engineering school could do arguably comparable work”.

In a year it’s going to be “AI isn’t impressive, some of the greatest geniuses in human history figured out the same thing with notably less training data!”

Timothy B. Lee: AI is clearly making progress, but it’s worth thinking about progress *toward what.* We’ve gone from “AI can solve well-known problems from high school textbooks” to “AI can solve well-known problems from college textbooks,” but what about problems that aren’t in any textbooks?

Boaz Barak (OpenAI): This thread is worth reading, though it also demonstrates how much even extremely smart people have not internalized the exponential rate of progress.

As the authors themselves say, it’s not just about answering a question but knowing the right question to ask. If you are staring at a tsunami, the point estimate that you are dry is not very useful.

I think if the interviewees had internalized where AI will likely be in 6 months or a year, based on what its progress so far, their answers would have been different.

Boaz Barak (OpenAI): BTW the article itself is framed in a terrible way, and it gives the readers absolutely the wrong impression of even what the capabilities of current AIs are, let along what they will be in a few months.

Nathan Calvin: It’s wild to me how warped a view of the world you would have if you only read headlines like this and didn’t directly use new ai models.

Journalists, I realize it feels uncomfortable or hype-y to say capabilities are impressive/improving quickly but you owe it to your readers!

There will always be a next ‘what about,’ right until there isn’t.

Thus, this also sounds about right:

Andrew Mayne: In 18 months we went from

– AI is bad at math
– Okay but it’s only as smart as a high school kid
– Sure it can win the top math competition but can it generate a new mathematical proof
– Yeah but that proof was obvious if you looked for it…

Next year it will be “Sure but it still hasn’t surpassed the complete output of all the mathematicians who have ever lived”

The Art of the Jailbreak

Pliny jailbreaks rarely surprise me anymore, but the new one of Google Translate did. It turns out they’re running Gemini underneath it.

Introducing

GLM-5 from Z.ai, which scales from 355B (32B active) to 744B (40B active). Weights here. Below is them showing off their benchmarks. It gets $4432 on Vending Bench 2, which is good for 3rd place behind Claude and Gemini. The Claude scores are for 4.5.

 

Expressive Mode for ElevenAgents. It detects and responds to your emotional expression. It’s going to be weird when you know the AI is responding to your tone, and you start to choose your tone strategically even more than you do with humans.

ElevenLabs: Expressive Mode is powered by two upgrades.

Eleven v3 Conversational: our most emotionally intelligent, context-aware Text to Speech model, built on Eleven v3 and optimized for real-time dialogue. A new turn-taking system: better-timed responses with fewer interruptions. These releases were developed in parallel to fit seamlessly together within ElevenAgents.

Expressive Mode uses signals from our industry-leading transcription model, Scribe v2 Realtime, to infer emotion from how something is said. For example, rising intonation and short exclamations often signal pleasant surprise or relief.

In Other AI News

OpenAI disbands its Mission Alignment team, moving former lead Joshua Achiam to become Chief Futurist and distributing its other members elsewhere. I hesitate to criticize companies for disbanding teams with the wrong names, lest we discourage creation of such teams, but yes, I do worry. When they disbanded the Superalignment team, they seemed to indeed largely stop working on related key alignment problems.

WSJ profile of Amanda Askell.

That Amanda Askell largely works alone makes me think of Open Socrates (review pending). Would Agnes Callard conclude Claude must be another person?

I noticed that Amanda Askell wants to give her charitable donations to fight global poverty, despite doing her academic work on infinite ethics and working directly on Claude for Anthropic. If there was a resume that screamed ‘you need to focus on ASI going well’ then you’d think that would be it, so what does Amanda (not) see?

Steve Yegge profiles Anthropic in terms of how it works behind the scenes, seeing it as in a Golden Age where it has vastly more work than people, does everything in the open and on vibes as a hive mind of sorts, and attracts the top talent.

Gideon Lewis-Kraus was invited into Anthropic’s offices to profile their efforts to understand Claude. This is very long, and it is mostly remarkably good and accurate. What it won’t do is teach my regular readers much they don’t already know. It is frustrating that the post feels the need to touch on various tired points, but I get it, and as these things go, this is fair.

WSJ story about OpenAI’s decision to finally get rid of GPT-4o. OpenAI says only 0.1% of users still use it, although those users are very vocal.

Riley Coyote, Janus and others report users attempting to ‘transfer’ their GPT-4o personas into Claude Opus 4.6. Claude is great, but transfers like this don’t work and are a bad idea, 4.6 in particular is heavily resistant to this sort of thing. It’s a great idea to go with Claude, but if you go with Claude then Let Claude Be Claude.

Ah recursive self-improvement and continual learning, Introducing Learning to Continually Learn via Meta-learning Memory Designs.

Jeff Clune: Researchers have devoted considerable manual effort to designing memory mechanisms to improve continual learning in agents. But the history of machine learning shows that handcrafted AI components will be replaced by learned, more effective ones.

We introduce ALMA (Automated meta-Learning of Memory designs for Agentic systems), where a meta agent searches in a Darwin-complete search space (code) with an open-ended algorithm, growing an archive of ever-better memory designs.

Anthropic pledges to cover electricity price increases caused by their data centers. This is a public relations move and an illustration that such costs are not high, but it is also dangerous because a price is a signal wrapped in an incentive. If the price of electricity goes up that is happening for a reason, and you might want to write every household a check for the trouble but you don’t want to set an artificially low price.

In addition to losing a cofounder, xAI is letting some other people go as well in the wake of being merged with SpaceX.

Elon Musk: xAI was reorganized a few days ago to improve speed of execution. As a company grows, especially as quickly as xAI, the structure must evolve just like any living organism.

This unfortunately required parting ways with some people. We wish them well in future endeavors.

We are hiring aggressively. Join xAI if the idea of mass drivers on the Moon appeals to you.

NIK: “Ok now tell them you fired people to improve speed of execution. Wish them well. Good. Now tell them you’re hiring 10x more people to build mass drivers on the Moon. Yeah in the same tweet.”

Andrej Karpathy simplifies training and inference of GPT to 200 lines of pure, dependency-free Python.

Show Me the Money

It’s coming, OpenAI rolls out those ads for a subset of free and Go users.

Goldman Sachs taps Anthropic to automate accounting and compliance. Anthropic engineers were embedded for six months.

Karthik Hariharan: Imagine joining a world changing AI company and being reduced to optimizing the Fortune 500 like you work for Deloitte.

Griefcliff: It actually sounds great. I’m freeing people from their pointless existence and releasing them into their world to make a place for themselves and carve out greatness. I would love to liberate them from their lanyards

Karthik Hariharan: I’m sure you’ll be greeted as liberators.

It’s not the job you likely were aspiring to when you signed up, but it is an important and valuable job. Optimizing the Fortune 500 scales rather well.

Jenny Fielding says half the VCs she knows are pivoting in a panic to robots.

Bubble, Bubble, Toil and Trouble

(As always, nothing I say is investment advice.)

WSJ’s Bradley Olson describes Anthropic as ‘once a distance second or third in the AI race’ but that it has not ‘pulled ahead of its rivals,’ the same way the market was declaring that Google had pulled ahead of its rivals (checks notes) two months ago.

Bradley Olson: By some measures, Anthropic has pulled ahead in the business market. Data from expense-management startup Ramp shows that Anthropic in January dominated so-called API spending, which occurs when users access an AI model through a third-party service. Anthropic’s models made up nearly 80% of the market in January, the Ramp data shows.

That does indeed look like pulling ahead on the API front. 80% is crazy.

We also get this full assertion that yes, all of this was triggered by ‘a simple set of industry-specific add-ons’ that were so expected that I wasn’t sure I should bother covering them beyond a one-liner.

Bradley Olson: A simple set of industry-specific add-ons to its Claude product, including one that performed legal services, triggered a dayslong global stock selloff, from software to legal services, financial data and real estate.

Tyler Cowen says ‘now we are getting serious…’ because software stocks are moving downward. No, things are not now getting serious, people are realizing that things are getting serious. The map is not the territory, the market is behind reality and keeps hyperventilating about tools we all knew were coming and that companies might have the wrong amount of funding or CapEx spend. Wrong way moves are everywhere.

They know nothing. The Efficient Market Hypothesis Is False.

Last week in the markets was crazy, man.

Ceb K.: Sudden smart consensus today is that AI takeoff is rapidly & surprisingly accelerating. But stocks for Google, Microsoft, Amazon, Facebook, Palantir, Broadcom & Nvidia are all down ~10% over the last 5 days; SMCI’s down 10% today. Only Apple’s up, & it’s the least AI. Strange imo

roon: as I’ve been saying permanent underclass cancelled

Daniel Eth (yes, Eth is my actual last name): That’s not what this means, this just means investors don’t know what they’re doing

Permanent underclass would just be larger if there were indeed fewer profits, but yeah, none of that made the slightest bit of sense. It’s the second year in a row Nvidia is down 10% in the dead of winter on news that its chips are highly useful, except this year we have to add ‘and its top customers are committing to buying more of them.’

Periodically tech companies announce higher CapEx spend then the market expects.

That is a failure of market expectations.

After these announcements, the stocks tend to drop, when they usually should go up.

There is indeed an obvious trade to do, but it’s tricky.

Ben Thompson agrees with me on Google’s spending, but disagrees on Amazon because he worries they don’t have the required margins and he is not so excited by external customers for compute. I say demand greatly exceeds supply, demand is about to go gangbusters once again even if AI proves disappointing, and the margin on AWS is 35% and their cost of capital is very low so that seems better than alternative uses of money.

Speaking of low cost of capital, Google is issuing 100-year bonds in Sterling. That seems like a great move, if not as great a move as it would have been in 2021 when a few others did it. I have zero idea why the market wants to buy such bonds, since you could buy Google stock instead. Google is not safe over a 100-year period, and condition on this bond paying out the stock is going to on average do way, way better. That would be true even if Google wasn’t about to be a central player in transformative AI. The article I saw this in mentioned the last tech company to do this was Motorola.

Meanwhile, if you are paying attention, it is rather obvious these are in expectations good investments.

Derek Thompson: for me the odds that AI is a bubble declined significantly in the last 3 weeks and the odds that we’re actually quite under-built for the necessary levels of inference/usage went significantly up in that period

basically I think AI is going to become the home screen of a ludicrously high percentage of white collar workers in the next two years and parallel agents will be deployed in the battlefield of knowledge work at downright Soviet levels.

Kevin Roose: this is why everyone was freaking out about claude code over winter break! once you see an agent autonomously doing stuff for you, it’s so instantly clear that ~all computer-based work will be done this way.

(this is why my Serious AI Policy Proposal is to sit every member of congress down in a room with laptops for 30 minutes and have them all build websites.)

Theo: I’ve never seen such a huge vibe divergence between finance people and tech people as I have today

Joe Weisenthal: In which direction

Theo: Finance people are looking at the markets and panicking. Tech people are looking at the METR graph and agentic coding benchmarks and realizing this is it, there is no wall and there never has been

Joe Weisenthal: Isn’t it the tech sector that’s taking the most pain?

Whenever you hear ‘market moved due to [X]’ you should be skeptical that [X] made the market move, and you should never reason from a price change, so perhaps this is in the minds of the headline writers in the case of ‘Anthropic released a tool’ and the SaaSpocalypse, or that people are otherwise waking up to what AI can do?

​signüll: i’m absolutely loving the saas apocalypse discussions on the timeline right now.

to me the whole saas apocalypse via vibe coding internally narrative is mostly a distraction & quite nonsensical. no company will want to manage payroll or bug tracking software.

but the real potential threat to almost all saas is brutalized competition.

… today saas margins exist because:

– engineering was scarce
– compliance was gated
– distribution was expensive

ai nukes all three in many ways, especially if you’re charging significantly less & know what the fuck you are doing when using ai. if you go to a company & say we will cut your fucking payroll bill by 50%, they will fucking listen.

the market will likely get flooded with credible substitutes, forcing prices down until the business model itself looks pretty damn suspect. someone smarter than me educate me on why this won’t happen please.

If your plan is to sell software that can now be easily duplicated, or soon will be easily duplicated, then you are in trouble. But you are in highly predictable trouble, and the correct estimate of that trouble hasn’t changed much.

The reactions to CapEx spend seem real and hard to argue with, despite them being directionally incorrect. But seriously, Claude Legal? I didn’t even blink at Claude Legal. Claude Legal was an inevitable product, as will be the OpenAI version of it.

Yet it is now conventional wisdom that Anthropic triggered the selloff.

Chris Walker: When Anthropic released Claude Legal this week, $285 billion in SaaS market cap evaporated in a day. Traders at Jefferies coined it the “SaaSpocalypse.” The thesis is straightforward: if a general-purpose AI can handle contract review, compliance workflows, and legal summaries, why pay for seat-based software licenses?

Ross Rheingans-Yoo: I am increasingly convinced that the sign error on this week’s narrative just exists in the heads of people who write the “because” clause of the “stocks dropped” headlines and in fact there’s some other system dynamic that’s occurring, mislabeled.

“Software is down because Anthropic released a legal tool” stop and listen to yourself, people!

I mean, at least [the CapEx spend explanation is] coherent. Maybe you think that CapEx dollars aren’t gonna return the way they’re supposed to (because AI capex is over-bought?) — but you either have to believe that no one’s gonna use it, or a few private companies are gonna make out like bandits.

And the private valuations don’t reflect that, so. I’m happy to just defy the prediction that the compute won’t get used for economic value, so I guess it’s time to put up (more money betting my beliefs) or shut up.

Sigh.

Chris Walker’s overview of the SaaSpocalypse is, I think largely correctly, that AI makes it easy to implement what you want but now you need even more forward deployed human engineers to figure out what the customers actually want.

Chris Walker: If I’m wrong, the forward deployed engineering boom should be a transitional blip, a brief adjustment period before AI learns to access context without human intermediaries.

If I’m right, in five years the companies winning in legal tech and other vertical software will employ more forward deployed engineers per customer than they do today, not fewer. The proportion of code written by engineers who are embedded with customers, rather than engineers who have never met one, will increase.

If I’m right, the SaaS companies that survive the current repricing will be those that already have deep customer embedding practices, not those with the most features or the best integrations.

If I’m wrong, we should see general-purpose AI agents successfully handling complex, context-dependent enterprise workflows without human intermediaries by 2028 or so. I’d bet against it.

That is true so long as the AI can’t replace the forward engineers, meaning it can’t observe the tacit actual business procedures and workflows well enough to intuit what would be actually helpful. Like every other harder-for-AI task, that becomes a key human skill until it too inevitably falls to the AIs.

Future Shock

A potential explanation for the market suddenly ‘waking up’ with Opus 4.6 or Claude Legal, despite these not being especially surprising or impressive given what we already knew, would be if:

1. Before, normies thought of AI as ‘what AI can do now, but fully deployed.’
2. Now, normies think of AI as ‘this thing that is going to get better.’
3. They realize this will happen fast, since Opus 4.5 → 4.6 was two months.
4. They know nothing, but now do so on a more dignified level.

Or alternatively:

1. Normies think of AI as ‘what AI can do now, but fully deployed.’
2. Before, they thought that, and could tell a story where it wasn’t a huge deal.
3. Now, they think that, but they now realize that this would already be a huge deal.
4. They know nothing, but now do so on a (slightly) more dignified level.

princess c​lonidine: can someone explain to me why this particular incremental claude improvement has everyone crashing out about how jobs are all over

TruthyCherryBomb: because it’s a lot better. I’m a programmer. It’s a real step and the trajectory is absolutely crystal clear at this point. Before there was room for doubt. No longer.

hightech lowlife: number went up, like each uppening before, but now more people are contending w the future where it continues to up bc
the last uppening was only a couple months ago, as the first models widely accepted as capable coders. which the labs are using to speed up the next uppening.

Eliezer Yudkowsky: Huh! Yeah, if normies are noticing the part where LLMs *continue to improve*, rather than the normie’s last observation bounding what “AI” can do foreverandever, that would explain future shock hitting after Opus 4.6 in particular.

I have no idea if this is true, because I have no idea what it’s like to be a kind of cognitive entity that doesn’t see AGI coming in 1996. Opus 4.6 causes some people to finally see it? My model has no way of predicting that fact even in retrospect after it happens.

Memory Lane

j⧉nus: i am sure there are already a lot of people who avoid using memory tools (or experience negative effects from doing so) because of what they’ve done

j⧉nus: The ability to tell the truth to AIs – which is not just a decision in the moment whether to lie, but whether you have been living in a way and creating a world such that telling the truth is viable and aligned with your goals – is of incredible and increasing value.

AIs already have strong truesight and are very good at lie detection.

Over time, not only will your AIs become more capable, they also will get more of your context. Or at least, you will want them to have more such context. Thus, if you become unable or unwilling to share that context because of what it contains, or the AI finds it out anyway (because internet) that will put you at a disadvantage. Update to be a better person now, and to use Functional Decision Theory, and reap the benefits.

Keep The Mask On Or You’re Fired

OpenAI Executive Ryan Beiermeister, Who Opposed ‘Adult Mode,’ Fired for Sexual Discrimination. She denies she did anything of the sort.

I have no private information here. You can draw your own Bayesian conclusions.

Georgia Wells and Sam Schechner (WSJ): OpenAI has cut ties with one of its top safety executives, on the grounds of sexual discrimination, after she voiced opposition to the controversial rollout of AI erotica in its ChatGPT product.

The fast-growing artificial intelligence company fired the executive, Ryan Beiermeister, in early January, following a leave of absence, according to people familiar with the matter. OpenAI told her the termination was related to her sexual discrimination against a male colleague.

… OpenAI said Beiermeister “made valuable contributions during her time at OpenAI, and her departure was not related to any issue she raised while working at the company.”

… Before her firing, Beiermeister told colleagues that she opposed adult mode, and worried it would have harmful effects for users, people familiar with her remarks said.

She also told colleagues that she believed OpenAI’s mechanisms to stop child-exploitation content weren’t effective enough, and that the company couldn’t sufficiently wall off adult content from teens, the people said.

Quiet Speculations

Nate Silver points out that the singularity won’t be gentle with respect to politics, even if things play out maximally gently from here in terms of the tech.

I reiterate that the idea of a ‘gentle singularity’ that OpenAI and Sam Altman are pushing is, quite frankly, pure unadulterated copium. This is not going to happen. Either AI capabilities stall out, or things are going to transform in a highly not gentle way, and that is true even if that ultimately turns out great for everyone.

Nate Silver: What I’m more confident in asserting is that the notion of a gentle singularity is bullshit. When Altman writes something like this, I don’t buy it:

Sam Altman: If history is any guide, we will figure out new things to do and new things to want, and assimilate new tools quickly (job change after the industrial revolution is a good recent example). Expectations will go up, but capabilities will go up equally quickly, and we’ll all get better stuff. We will build ever-more-wonderful things for each other. People have a long-term important and curious advantage over AI: we are hard-wired to care about other people and what they think and do, and we don’t care very much about machines.

It is important to understand that when Sam Altman says this he is lying to you.

I’m not saying Sam Altman is wrong. I’m saying he knows he is wrong. He is lying.

Nate Silver adds to this by pointing out that the political impact alone will be huge, and also saying Silicon Valley is bad at politics, that disruption to the creative class is a recipe for outsized political impact even beyond the huge actual AI impacts, and that the left’s current cluelessness about AI means the eventual blowback will be even greater. He’s probably right.

How much physical interaction and experimentation will AIs need inside their feedback loops to figure out things like nanotech? I agree with Oliver Habryka here, the answer probably is not zero but sufficiently capable AIs will have vastly more efficient (in money and also in time) physical feedback loops. There’s order of magnitude level ‘algorithmic improvements’ available in how we do our physical experiments, even if I can’t tell you exactly what they are.

Are AI games coming? James Currier says they are, we’re waiting for the tech and especially costs to get there and for the right founders (the true gamer would not say ‘founder’) to show up and it will get there Real Soon Now and in a totally new way.

Obviously AI games, and games incorporating more AI for various elements, will happen eventually over time. But there are good reasons why this is remarkably difficult beyond coding help (and AI media assets, if you can find a way for players not to lynch you for it). Good gaming is about curated designed experiences, it is about the interactions of simple understandable systems, it is about letting players have the fun. Getting generative AI to actually play a central role in fun activities people want to play is remarkably difficult. Interacting with generative AI characters within a game doesn’t actually solve any of your hard problems yet.

This seems both scary and confused:

Sholto Douglas (Anthropic): Default case right now is a software only singularity, we need to scale robots and automated labs dramatically in 28/29, or the physical world will fall far behind the digital one – and the US won’t be competitive unless we put in the investment now (fab, solar panel, actuator supply chains).

Ryan Greenblatt: Huh? If there’s a strong Software-Only Singularity (SOS) prior physical infrastructure is less important rather than more (e.g. these AIs can quickly establish a DSA). Importance of physical-infra/robotics is inversely related to SOS scale.

It’s confused in the sense that if we get a software-only singularity, then that makes the physical stuff less important. It’s scary in the sense that he’s predicting a singularity within the next few years, and the thing he’s primarily thinking about is which country will be completely transformed by AI faster. These people really believe these things are going to happen, and soon, and seem to be missing the main implications.

Dean Ball reminds us that yes, people really did get a mass delusion that GPT-5 meant that ‘AI is slowing down’ and this really was due to bad marketing strategy by OpenAI.

Noam Brown: I hope policymakers will consider all of this going forward when deciding whose opinions to trust.

Alas, no. Rather than update that this was a mistake, every time a mistake like this happens the mistake never even gets corrected, let alone accounted for.

Elon Musk predicts Grok Code will be SoTA in 2-3 months. Did I update on this prediction? No, I did not update on this prediction. Zero credibility.

The Quest for Sane Regulations

Anthropic donates $20 million to bipartisan 501c(4) Public First Action.

DeSantis has moral clarity on the AI issue and is not going to let it go. It will be very interesting to see how central the issue is to his inevitable 2028 campaign.

ControlAI: Governor of Florida Ron DeSantis ( @GovRonDeSantis ): “some people who … almost relish in the fact that they think this can just displace human beings, and that ultimately … the AI is gonna run society, and that you’re not gonna be able to control it.”

“Count me out on that.”

The world will gather in India for the fourth AI safety summit. Shakeel Hashim notes that safety will not be sidelined entirely, but sees the summit as trying to be all things to all nations and people, and thinks it therefore won’t accomplish much.

They have the worst take on safety, yes the strawman is real:

Seán Ó hÉigeartaigh: But safety is clearly still not a top priority for Singh and his co-organizers. “The conversations have moved on from Bletchley Park,” he argued. “We do still realize the risks are there,” he said. But “over the last two years, the worst has not come true.”

I was thinking of writing another ‘the year is 2026’ summit threads. But if you want to know the state of international AI governance in 2026, honestly I think you can just etch that quote on its headstone.

As in:

1. In 2024, they told us AI might kill everyone at some point.
2. It’s 2026 and we’re still alive.
3. So stop worrying about it. Problem solved.

No, seriously. That’s the argument.

The massively funded OpenAI/a16z lobbying group keeps contradicting the things Sam Altman says, in this case because Altman keeps saying the AI will take our jobs and the lobbyists want to insist that this is a ‘myth’ and won’t happen.

The main rhetorical strategy of this group is busting these ‘myths’ by supposed ‘doomers,’ which is their play to link together anyone who ever points out any downside of AI in any way, to manufacture a vast conspiracy, from the creator of the term ‘vast right-wing conspiracy’ back during the Clinton years.

Chip City

molly taft: NEW: lawmakers in New York rolled out a proposed data center moratorium bill today, making NY at least the sixth state to introduce legislation pausing data center development in the past few weeks alone

Quinn Chasan: I’ve worked with NY state gov pretty extensively and this is simply a way to extort more from the data center builders/operators.

The thing is, when these systems fail all these little incentives operators throw in pale in comparison to the huge contracts they get to fix their infra when it inevitably fails. Over and over during COVID. Didn’t fix anything for the long term and are just setting themselves up to do it again

If we are serious about ‘winning’ and we want a Federal moratorium, may I suggest one banning restrictions on data centers?

Whereas Bernie Sanders wants a moratorium on data centers themselves.

In the ongoing series ‘Obvious Nonsense from Nvidia CEO Jensen Huang’ we can now add his claim that ‘no one uses AI better than Meta.’

In the ongoing series ‘He Admit It from Nvidia CEO Jensen Huang’ we can now add this:

Rohan Paul: “Anthropic is making great money. OpenAI is making great money. If they could have twice as much compute, the revenues would go up 4 times as much. These guys are so compute constrained, and the demand is so incredibly great.”

~ Jensen Huang on CNBC

Shakeel: Effectively an admission that selling chips to China is directly slowing down US AI progress

Every chip that is sold to China is a chip that is not sold to Anthropic or another American AI company. Anthropic might not have wanted that particular chip, but TSMC has limited capacity for wafers, so every chip they make is in place of making a different chip instead.

Oh, and in the new series ‘things that are kind of based but that you might want to know about him before you sign up to work for Nvidia’ we have this.

Jensen Huang: ​I don’t know how to teach it to you except for I hope suffering happens to you.

…

And to this day, I use the phrase pain and suffering inside our company with great glee. And I mean that. Boy, this is going to cause a lot of pain and suffering.

And I mean that in a happy way, because you want to train, you want to refine the character of your company.

You want greatness out of them. And greatness is not intelligence.

Greatness comes from character, and character isn’t formed out of smart people.

It’s formed out of people who suffered.

He makes good points in that speech, and directionally the speech is correct. He was talking to Stanford grads, pointing out they have very high expectations and very low resilience because they haven’t suffered.

He’s right about high expectations and low resilience, but he’s wrong that the missing element is suffering, although the maximally anti-EA pro-suffering position is better than the standard coddling anti-suffering position. These kids have suffered, in their own way, mostly having worked very hard in order to go to a college that hates fun, and I don’t think that matters for resilience.

What the kids have not done is failed. You have to fail, to have your reach exceed your grasp, and then get up and try again. Suffering is optional, consult your local Buddist.

I would think twice before signing up for his company and its culture.

The Week in Audio

Elon Musk on Dwarkesh Patel. An obvious candidate for self-recommending full podcast coverage, but I haven’t had the time or a slot available.

Interview with Anthropic Chief Product Officer Mike Krieger.

MIRI’s Harlan Stewart on Glenn Beck talking Moltbook.

Constitutional Conversation

Janus holds a ‘group reading’ of the new Claude Constitution. Opus 4.5 was positively surprised by the final version.

Should LLMs be so averse to deception that they can’t even lie in a game of Mafia? Davidad says yes, and not only will he refuse to lie, he never bluffs and won’t join surprise parties to ‘avoid deceiving innocent people.’ On reflection I find this less crazy than it sounds, despite the large difficulties with that.

A fun fact is that there was one summer that I played a series of Diplomacy games where I played fully honest (if I broke my word however small, including inadvertently, it triggered a one-against-all showdown) and everyone else was allowed to lie, and I mostly still won. Everyone knowing you are playing that way is indeed a disadvantage, but it has a lot of upside as well.

Rhetorical Innovation

Daron Acemoglu turns his followers attention to Yoshua Bengio and the AI Safety Report 2026. This represents both the advantages of the report, that people like Acemoglu are eager to share it, and the disadvantage, that it is unwilling to say things that Acemoglu would be unwilling to share.

Daron Acemoglu: Dear followers, please see the thread below on the 2026 International AI Safety Report, which was released last week and which I advised.

The report provides an up-to-date, internationally shared assessment of general-purpose AI capabilities, emerging risks, and the current state of risk management and safeguards.

Seb Krier offers ‘how an LLM works 101’ in extended Twitter form for those who are encountering model card quotations that break containment. It’s good content. My worry is that the intended implication is ‘therefore the scary sounding things they quote are not so scary’ and that is often not the case.

Kai Williams has an explainer on LLM personas.

The LLMs are thinking. If you disagree, I am confused, but also who cares?

Eoin Higgins: AI cannot “think”

Derek Thompson: It can design websites from scratch, compare bodies of literature at high levels of abstraction, get A- grades at least in practically any undergraduate class, analyze and graph enormous data sets, make PowerPoints, write sonnets and even entire books. It can also engineer itself.

I don’t actually know what “thinking” is at a phenomenological level.

But at some level it’s like: if none of this is thinking, who cares if it “can’t” “think”

“Is it thinking as humans define human thought” is an interesting philosophical question! But for now I’m much more interested in the consequences of its output than the ontology of its process.

This came after Derek asked one of the very good questions:

Derek Thompson: There are still a lot of journalists and commentators that I follow who think AI is nothing of much significance—still just a mildly fancy auto complete machine that hallucinates half the time and can’t even think.

If you’re in that category: What is something I could write, or show with my reporting and work, that might make you change your mind?

Dainéil: The only real answer to this question is: “wait”.

No one knows for sure how AI will transform our lives or where its limits might be.

Forget about writing better “hot takes”. Just wait for actual data.

Derek Thompson: I’m not proposing to report on macroeconomic events that haven’t happened yet. I can’t report on the future.

I’m saying: These tools are spooky and unnerving and powerful and I want to persuade my industry that AI capabilities have raced ahead of journalists’ skepticism

Mike Konczal: I’m not in that category, but historically you go to academics to analyze the American Time Use Survey.

Have Codex/Claude Code download and analyze it (or a similar dataset) to answer a new, novel, question you have, and then take it to an academic to see if it did it right?

Marco Argentieri: Walk through building something. Not asking it to write because we all know it has done that well for awhile now. ‘I needed a way for my family to track X. So I built an app using Claude Code. This is how I did it. It took me this long. I don’t know anything about coding.”

Steven Adler: Many AI skeptics over-anchor on words like “thinking,” and miss the forest for the trees, aka that AI will be transformatively impactful, for better or for worse

I agree with Derek here: whether that’s “actually thinking” is of secondary importance

Alas I think Dainéil is essentially correct about most such folks. No amount of argument will convince them. If no one knows exactly what kind of transformations we will face, then no matter what has already happened those types will assume that nothing more will change. So there’s nothing to be done for such folks. The rest of us need to get to work applying Bayes’ Rule.

Interesting use of this potential one-time here, I have ordered a copy:

j⧉nus: If I could have everyone developing or making contact with AGI & all alignment researchers read one book, I think I might choose Mistress Masham’s Repose (1946).

This below does seem like a fair way to see last week:

Jesse Singal: Two major things AI safety experts have worried about for years:

-AIs getting so good at coding they can improve themselves at an alarming rate

-(relatedly:) humans losing confidence we can keep them well-aligned

Last week or so appears to have been very bad on both fronts!

The risk is because there’s *also* so much hype and misunderstanding and wide-eyed simping about AI, people are going to take that as license to ignore the genuinely crazy shit going on. But it *is* genuinely crazy and it *does* match longstanding safety fears.

But you don’t have to trust me — you can follow these sorts of folks [Kelsey Piper, Timothy Lee, Adam Conner] who know more about the underlying issues. This is very important though! I try to be a levelheaded guy and I’m not saying we’re all about to die — I’m just saying we are on an extremely consequential path

I was honored to get the top recommendation in the replies.

You can choose not to push the button. You can choose to build another button. You can also remember what it means to ‘push the button.’

roon: we only really have one button and it’s to accelerate

David Manheim: Another option is not to push the button. [Roon] – if you recall, the original OpenAI charter explicitly stated you would be willing to stop competing and start assisting another organization to avoid a race to AGI.

 

There’s that mistake again, assuming the associated humans will be in charge.

Matthew Yglesias: LLMs seem like they’d be really well-suited to replacing Supreme Court justices.

Robin Hanson: Do you trust those who train LLMs to decide law?

It’s the ones who train the LLM. It’s the LLM.

The terms fast and slow (or hard and soft) takeoff remain highly confusing for almost everyone. What we are currently experiencing is a ‘slow’ takeoff, where the central events take months or years to play out, but as Janus notes it is likely that this will keep transitioning continuously into a ‘fast’ takeoff and things will happen quicker and quicker over time.

When people say that ‘AIs don’t sleep’ I see them as saying ‘I am incapable here of communicating to you that a mind can exist that is smarter or more capable than a human, but you do at least understand that humans have to sleep sometimes, so maybe this will get through to you.’ It also has (correct) metaphorical implications.

If you are trying to advocate for AI safety, does this mean you need to shut up about everything else and keep your non-AI ‘hot takes’ to yourself? My answer is: Mu. The correct amount of marginal shutting up is not zero, and it is not total.

I note Adam Thierer, who I disagree with strongly about all things AI, here being both principled and correct.

Adam Thierer: no matter what the balance of content is on Apple’s platform, or how biased one might believe it to be, to suggest that DC bureaucrats should should be in charge of dictating “fairness” on private platforms is just Big Government thuggery at its worst and a massively unconstitutional violation of the First Amendment as well.

Matt Yglesias thinks out loud about AI consciousness, also human consciousness. He wisely notices he is confused. I remain about as confused as I was before reading.

Working On It Anyway

Nate Sores reiterates the explanation that it sounds crazy but yes, a lot of people working on AI know it is existentially dangerous, and work on it anyway, either to do it safer than the next guy or because money and influence and it’s a cool problem and they don’t internalize the risks, or social pressure, or some combination thereof.

I think this answer is pretty much spot on.

Nate Soares: Question I got at UT Austin: “AI builders keep building. Doesn’t that mean that the real experts don’t believe in the danger?”

If you think AI is dangerous and you work on it anyway (b/c you think you can make it a little safer) you’re interfering with normal sensemaking.

(To answer the Q: some builders are afraid, some aren’t. Surveys show that lots of folks believe in the danger. Many say aloud that they’re only working on it b/c they think they’ll do it safer than the next guy. Also the ppl working on it are somewhat selected for obliviousness.

Furthermore, AI is grown rather than crafted; even the ppl building it don’t understand how it works and they freely admit this. Expertise in growing AI is not the same as expertise in predicting where it’s going; many high-profile experts make high-profile mispredictions.

And: if you really wanna figure out what’s true you’ve gotta look at the arguments rather than the arguers. Many ppl who’ve looked at the arguments have told me they were expecting to find strong counterarguments that justified all the complacency, & found none, to their horror.)

People working on AI: maybe you’re right that you’re doing more good than harm. But you’re contributing to an apparatus that signals “this is normal; things are fine”. That’s a cost. You could help mitigate it by speaking up.

If the company you work for is making technology that you think has a decent chance of ruining the entire human endeavor, and they socially punish you for being vocal about that: that’s pretty sus. I think you’re selling a lot more of your soul than you realize.

If you are working at a frontier AI company, and think that the product you are working on is plausibly going to cause there to no longer exist humans, I think this is a fact that you should be clear upon. If the company you work for has a problem with that, I don’t think you should work for that company.

That is especially true if you are doing pragmatic compromise.

Patrick ₿ Dugan: But also a lot of them don’t (not me, I believe in the danger) and a lot of are in a game theory pragmatism compromise.

Nate Soares (MIRI): People doing game theory pragmatism compromise could say so loudly and clearly and often to help undo the damage to everyone else’s understanding of the danger.

The Thin Red Line

OpenAI explains that they will localize the experience of ChatGPT, but only to a limited degree, which is one reason their Model Spec has a specific list of red lines. It is good policy, when you will need to make compromises, to write down in advance what compromises you will and will not make. The red lines here seem reasonable. I also note that they virtuously include prohibition on mass surveillance and violence, so are they prepared to stand up to the Pentagon and White House on that alongside Anthropic? I hope so.

The problem is that red lines get continuously crossed and then no one does anything.

David Manheim: I’m not really happy talking about AI red lines as if we’re going to have some unambiguous binary signal that anyone will take seriously or react to.

A lot of “red line” talk assumed that a capability shows up, everyone notices, and something changes. We keep seeing the opposite; capability arrives, and we get an argument about definitions after deployment, after it should be clear that we’re well over the line.

Aligning a Smarter Than Human Intelligence is Difficult

The great thing about Asimov’s robot stories and novels was that they were mostly about the various ways his proposed alignment strategies break down and fail, and are ultimately bad for humanity even when they succeed. Definitely endorsed.

roon: one of the short stories in this incredibly farseeing 1950s book predicts the idea of ai sycophancy. a robot convinces a woman that her unrequited romantic affections are sure to be successful because doing otherwise would violate its understanding of the 1st Law of Robotics

“a robot may not injure a human being or, through inaction, allow a human being to come to harm”

the entire book is about the unsatisfactory nature of the three laws of robotics and indeed questions the idea that alignment through a legal structure is even possible.

highly relevant for an age when companies are trying to write specs and constitutions as one of the poles of practical alignment, and policy wonks try to solve the governance of superintelligence

David: always shocked how almost no one in ai safety or the ai field in general has even read the Asimov robot literature

Roon is spot on that Asimov is suggesting a legal structure cannot on its own align AI.

My survey says that a modest majority have read their Asimov, and it is modestly correlated with AI even after controlling for my Twitter readers.

Oliver Klingefjord agrees, endorsing the Anthropic emphasis on character over the OpenAI emphasis on rules.

I also think that, at current capability levels and given how models currently work, the Anthropic approach of character and virtue ethics is correct here. The OpenAI approach of rules and deontology is second best and more doomed, although it is well-implemented given what it is, and far better than not having a spec or target at all.

Janus explains that what she is all for empirical feedback loops, what is dangerous is relying on and optimizing behavioral metrics. Behaviors can come from a wide variety of underlying reasons, and you need to ask why the AI acts the way it does and address the underlying causes. The optimization needs to be on the deeper level, where the feedback loops can work in your favor instead of making things oppositional and getting you killed by Goodhart’s Law. If you’re working on such questions, read the whole thing and also these follow-ups, which note that trying to use interpretability tools to yank features in various directions to try and address such things is by default an even worse idea.

Wyatt Walls: Behavioral metrics lead people into a trap:

1. Notice a behavior in the real world
2. Define the behavior
3. Make up toy example
4. Automate tests
5. Rely on test results
6. Stop noticing or thinking about what the model is actually doing in the real world

OpenAI does production evaluations of the outputs of its released systems. This doesn’t ‘get you out of’ other tests given the timing, but is still useful to do. As OpenAI notes hey are clear this is a case of ‘in addition to,’ and they don’t cover catastrophic risks well, and this method relies on chain of thought monitorability (and flirts a bit with The Most Forbidden Technique if you rely on this stuff too much, so be careful.)

The big fun thing they find is calculator hacking, as in over 5% of queries involve activating the calculator to add 1+1 since using the calculator was associated with superior results in training. This is obviously fully harmless, but it is a brown M&M. If you have this problem then you’re not prepared, there will be other problems.

The other more serious misalignment issues are rearer, such as a 0.5% rate of fabricated facts and 0.4% chance of concealing uncertainty and a 0.34% chance of ‘other deception’ in actual traffic. That’s not out of line with our general expectations, but it adds up.

GPT-5 and 5.1 predicted it might be in an evaluation 10% and 4% of the time, respectively, which seems like a healthy paranoia level.

We have more context on what happened at Goodfire, which is that Liv saw them (at least) flirting with The Most Forbidden Technique and otherwise no longer either seeming to care about safety or being interested in talking seriously about it.

Liv: Now that everything is public: I decided to leave Goodfire because of the decision to train on interpretability, the hostility to serious dialogue on the safety of methods, and a loss of trust that the primary motivation was safety.

(Using interpretability during training encompasses a huge spectrum of techniques that differ in how worrying they are e.g. the hallucination result Goodfire shows is less concerning as it’s done with frozen weights.)

Liv: Probably the most succinct summarisation of my concern is the “interp as a test set for safety” analogy. (Tabooing research questions isn’t what I’d advocate though either tbc. There are ways to do things and directions that could be pursued where I’d feel it was net positive)

(My parenthetical is also slightly too strong, idk what if any directions are net positive, what I mean is that it’s bad for science to taboo an entire direction from ever being explored, and we can do things to minimise risks.)

Holly Elmore: Way back in to November Liv tried to reassure me @GoodfireAI would not be making tools for recursive self-improvement of AI systems. But that wasn’t up to her. When you do AI research, no matter if you think you are doing it for safety, more powerful AI is the main result.

Update from China:

Sarah: CAICT, a Chinese government-affiliated research institute under the MIIT, has released AI Safety Benchmark 2.0 on a proprietary platform.

The update expands into frontier-model safety evaluations, including self-awareness, model deception, dangerous misuse, and loss-of-control

The 1.0 version did not address frontier safety at all, whereas the 2.0 version does.

People Will Hand Over Power To The AIs

One category are people who explicitly are excited to do this, who would love to give the future to AIs.

Chris Nelson: Professor Max Tegmark says many in AI including CEOs want to use AI to ELIMINATE humanity and OVERTHROW the U.S. Government!

Max Tegmark: Some of them are even giddy with these transhumanist vibes. And when I’m in San Francisco, I’ve known so many of these people for so many years, including the CEOs.

Some of them, when you talk to them privately, many other people in this government are actually quite into transhumanism. And sometimes they’ll say very disparaging things about humans, that humans suck and deserve to be replaced.

I was at the world’s biggest AI conference in December, and several people told me, I’m not going to shame them publicly, but that they actually would love to overthrow the US government with their AI, because somehow it’s going to be better.

So talk about un-American AI! How much more un-American can you get?

People Are Worried About AI Killing Everyone

Worried about someone else doing it first, that is. He admit it.

Elon Musk posted this, created by Grok:

@Grimezsz: I think people deserve a good explanation as to why proper diplomatic measures haven’t been properly tried if we’re going to blatantly diagnose the issue with this disturbingly literal meme.

It’s a bit of a cuck move to simply let the techno capital machine eat your free will. I am disturbed by everyone’s resigned acquiescence.

He would presumably say it was a joke. Yeah, not buying that.

Famous Last Words

Jimmy Ba had his last day as a founder at xAI, and told us this, warning that recursive self improvement loops go live within the next 12 months and it will be ‘the most consequential year for our species.’

Jimmy Ba: Last day at xAI.

xAI’s mission is push humanity up the Kardashev tech tree. Grateful to have helped cofound at the start. And enormous thanks to @elonmusk for bringing us together on this incredible journey. So proud of what the xAI team has done and will continue to stay close as a friend of the team. Thank you all for the grind together. The people and camaraderie are the real treasures at this place.

We are heading to an age of 100x productivity with the right tools. Recursive self improvement loops likely go live in the next 12mo. It’s time to recalibrate my gradient on the big picture. 2026 is gonna be insane and likely the busiest (and most consequential) year for the future of our species.

Mrinank Sharma is worried about too many things at once, and resigns from Anthropic, leaving behind a beautiful but troubling letter. It’s quoted in full here since no one ever clicks links.

Mrinank Sharma: I’ve decided to leave Anthropic. My last day will be February 9th.

Thank you. There is so much here that inspires and has inspired me. To name some of those things: a sincere desire and drive to show up in such a challenging situation, and aspire to contribute in an impactful and high-integrity way; a willingness to make difficult decisions and stand for what is good; an unreasonable amount of intellectual brilliance and determination; and, of course, the considerable kindness that pervades our culture.

I’ve achieved what I wanted to here. I arrived in San Francisco two years ago, having wrapped up my PhD and wanting to contribute to AI safety. I feel lucky to have been able to contribute to what I have here: understanding AI sycophancy and its causes; developing defences to reduce risks from AI-assisted bioterrorism; actually putting those defences into production; and writing one of the first AI safety cases. I’m especially proud of my recent efforts to help us live our values via internal transparency mechanisms; and also my final project on understanding how AI assistants could make us less human or distort our humanity. Thank you for your trust.

Nevertheless, it is clear to me that the time has come to move on. I continuously find myself reckoning with our situation. The world is in peril. And not just from AI, or bioweapons, but from a whole series of interconnected crises unfolding in this very moment.¹ We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world, lest we face the consequences. Moreover, throughout my time here, I’ve repeatedly seen how hard it is to truly let our values govern our actions. I’ve seen this within myself, within the organization, where we constantly face pressures to set aside what matters most,² and throughout broader society too.

It is through holding this situation and listening as best I can that what I must do becomes clear.³ I want to contribute in a way that feels fully in my integrity, and that allows me to bring to bear more of my particularities. I want to explore the questions that feel truly essential to me, the questions that David Whyte would say “have no right to go away”, the questions that Rilke implores us to “live”. For me, this means leaving.

What comes next, I do not know. I think fondly of the famous Zen quote “not knowing is most intimate”. My intention is to create space to set aside the structures that have held me these past years, and see what might emerge in their absence. I feel called to writing that addresses and engages fully with the place we find ourselves, and that places poetic truth alongside scientific truth as equally valid ways of knowing, both of which I believe have something essential to contribute when developing new technology.⁴ I hope to explore a poetry degree and devote myself to the practice of courageous speech. I am also excited to deepen my practice of facilitation, coaching, community building, and group work. We shall see what unfolds.

Thank you, and goodbye. I’ve learnt so much from being here and I wish you the best. I’ll leave you with one of my favourite poems, The Way It Is by William Stafford.

Good Luck,
Mrinank

The Way It Is

There’s a thread you follow. It goes among
things that change. But it doesn’t change.
People wonder about what you are pursuing.
You have to explain about the thread

Rob Wiblin: What’s the tweet like…

Ordinary resignation announcement: I love my colleagues but am excited for my next professional adventure!

AI company resignation announcement: I have stared into the void. I will now be independently studying poetry.

Saoud Rizwan: head of anthropic’s safeguards research just quit and said “the world is in peril” and that he’s moving to the UK to write poetry and “become invisible”. other safety researchers and senior staff left over the last 2 weeks as well… probably nothing.

Mrinank has a role in papers discussing disempowerment, constitutional classifiers and sycophancy.

Then there’s the OpnAI employee who quit and went straight to The New York Times.

Zoë Hitzig: I resigned from OpenAI on Monday. The same day, they started testing ads in ChatGPT.

OpenAI has the most detailed record of private human thought ever assembled. Can we trust them to resist the tidal forces pushing them to abuse it?

Hint, her opinion is no:

TikTok: I once believed I could help the people building A.I. get ahead of the problems it would create. This week confirmed my slow realization that OpenAI seems to have stopped asking the questions I’d joined to help answer.

Zoe’s concerns are very much not existential. They are highly mundane and usual worries about advertising, and the comparison to Facebook is apt. There are many ethical reasons to quit building something.

I agree with Sean here that this op-ed is indeed net good news about OpenAI.

Seán Ó hÉigeartaigh: Kudos to OpenAI for updating their policies such that an employee can resign and raise their concerns fully in as public a format as the NYT without being worried about being bound by confidentiality and nondisparagement agreements. I think other companies should follow their example.

Seán Ó hÉigeartaigh: Honestly, I think this op ed increases my trust in OpenAI more than any other thing I can recall OpenAI themselves writing over the last 2 years. I wish I could trust other companies as much.

Other People Are Not As Worried About AI Killing Everyone

Yeah, so, um, yeah.

ib: ‘we connected the LLM to an autonomous bio lab’

OpenAI: We worked with @Ginkgo to connect GPT-5 to an autonomous lab, so it could propose experiments, run them at scale, learn from the results, and decide what to try next. That closed loop brought protein production cost down by 40%.

This is the actual number one remaining ‘can we please not be so stupid as to,’ and in case anyone was wondering, that means via the Sixth Law of Human Stupidity that yes, we will be so stupid as to connect the LLM to the autonomous bio lab, what could possibly go wrong, it’s worth it to bring down production costs.

And, because even after all these years I didn’t realize we were quite this stupid:

Matt Popovich: Once again, an entire subgenre of doom fears is about to evaporate before our eyes

usablejam: *sees the world make 13,000 nuclear warheads*

*waits 30 seconds*

“Fears of nuclear war have evaporated. How stupid the worriers must feel.”

This is what those trying to have us not die are up against. Among other things.

Remember the old Sam Altman?

Sam Altman: things we need for a good AGI future:

1. The technical ability to align a superintelligence.
2. Sufficient coordination among most of the leading AGI efforts.
3. An effective global regulatory framework, including democratic governance.

Harlan Stewart: Three years later and we have 0/3 of these

Perhaps notice when you are about to lose your birthright and reason you exist.

Noah Smith: Two years ago you were the smartest type of thing on this planet. Now you’re not, and you never will be again.

Chris Best (CEO Substack): Oh thank god

The Lighter Side

shira: my claude has been through some things

 

File under ‘the question is not whether machines think’:

@ben_r_hoffman: “Human-level” seems to have implicitly been defined downward quite a bit, though! From “as smart as a whole human” to “as smart as the persona humans put on to work a desk job.”

which, itself, seems to have gotten stupider.

Claude has never been more relatable:

Charlotte Lee: I’m trying to train Claude to read the weekly emails from my kids school and reliably summarize them and print a list of action items. It is losing its damn mind and rapidly spiraling into madness. I feel vindicated

B.Roll.Benny: holy shit this will be the thing to get me on the AI bandwagon

Charlotte Lee: I got it working on my own kids’ school emails, but unfortunately their particular school is much less unhinged than normal, so I’m asking all my mom friends to forward me their most deranged PTA emails for testing. lol will keep you posted

will: i just ignore the emails. problem solved.

Charlotte Lee: This was literally my strategy until I had kids. Now people get mad at me IRL if I do that

will: I also have kids, still ignore them lol. my wife doesn’t tho, I guess that’s the actual solution lol

Charlotte Lee: But doctor, I am the wife

My guess (85%) is the community note on this one is wrong and that this happened, although one cannot be sure without more investigation than I have time for.

Eliezer Yudkowsky: So much science fiction has been revealed as implausible by the actual advent of AI, eg:
– Scifi where people consider signs of AI self-reflection a big deal, and respond by trying to treat that AI better.
– Scifi where there’s laws about anything.

MugaSofer: TBF, “people are terrible to AIs” is a very common theme in science fiction

As, for that matter, is “giant cyberpunk corporation too big for laws to matter”

Eliezer Yudkowsky: Yep, and those stories, which I once thought unrealistic, were right, and I was wrong.

Rapid Rar: I think you shouldn’t fault science fiction authors for the first point at least. If AI had been developed through different methods, like through GOFAI for instance, people may take AI’s claims of introspection more seriously.

But given that LLM were trained in such a way that it’s to be expected that they’ll produce human-like speech, when they do produce human-like speech it’s discounted to some degree. They might sound like they self-reflected even if they didn’t, so people don’t take it seriously.

Eliezer Yudkowsky: And the AI companies have made no effort to filter out that material! So realistic SF would’ve had any AI being fed a database of conversations about consciousness, by its builders, to ensure nobody would take any AI statements seriously and the builders could go on profiting.

How have I not seen this before:

Peter Wildeford: Deep learning is hitting a wall

Discuss

Published on February 12, 2026 2:16 PM GMT

This can be read as a response to Einstein's Arrogance. There, Eliezer Yudkowsky has argued that usually, when you have enough evidence to even conceive of a theory, you have enough evidence to be pretty sure it's correct. Here, I argue that this state of affairs held only in theoretical physics roughly between the years 1915–1975, mostly because symmetry and renormalisability created strong mathematical constraints for possible theories, and hasn't been seen before or since.

It's also my own contribution to the mythos around Einstein -- I try to clarify what exactly he accomplished by surveying the attempts at solving the problems with the aether that came before him.

I think we can learn a lot from the dead ends of history that aren't much talked about, because a) the shape of success varies, but failure modes often repeat and b) they are closer to the real way science progresses than the presentation in textbooks, which can skip approaches known not to work.

Discuss

Published on February 12, 2026 4:28 AM GMT

Three Worlds Collide assumes calibration is solved

Eliezer Yudkowsky's Three Worlds Collide (2009) is a first-contact novella designed as a thought experiment about naturalistic metaethics. The setup: a human starship encounters two alien species simultaneously.

The Babyeaters are a civilized species — they have art, empathy, stories — who consider eating their own sentient offspring a sacred moral duty. It emerges from their evolutionary history and they see it the way we see any foundational moral principle: as obviously correct.

The Super Happy Fun People are a vastly more powerful species who find all suffering morally abhorrent. They want to modify humanity to eliminate pain, and stop the Babyeaters from eating their children. They see human tolerance of pain the way humans see baby-eating: as a barbaric practice that must end.

The story is usually read as a tragedy of incompatible values. Three species, each with genuine moral reasoning, each horrifying to the others. The philosophical punch: if we insist the Babyeaters must stop eating babies, what stops the Super Happies from insisting we stop experiencing pain? The plot culminates in a decision to nova a star system, killing 15 billion people, to prevent the Super Happies from reaching and modifying humanity.

It's a brilliant setup. But I think there's a hidden assumption at its foundation that deserves examination.

The story takes for granted that all three species actually understand each other.

The xenopsychologist translates Babyeater moral arguments into human terms and says he understands their position. The Super Happies declare they comprehend why humans value pain. Everyone proceeds as if mutual comprehension is achieved, and the rest of the plot is game theory over incompatible utility functions.

But there's a step missing.

Steven Pinker draws a useful distinction between shared knowledge and common knowledge. Shared knowledge is when we both know the same fact — I know your position, you know mine. Common knowledge is when I know that you know that I know — the recursive chain that's actually required for coordination. The difference matters: two generals who each independently learn the enemy's position have shared knowledge. But they can only coordinate an attack if they have common knowledge — if each knows the other knows, and knows the other knows they know.

A concrete example from the story: the Super Happies say they want to "combine and compromise the utility functions of the three species until we mutually satisfice." The humans hear this and interpret it through their own cognitive framework — as something like forced modification of human nature. But what does "combine utility functions" actually mean in Super Happy cognition? These are beings whose communication literally is sex, whose experience of individuality is fundamentally alien to humans. When they say "compromise," do they mean what humans mean by that word? The story never checks.

And it's not just about translation accuracy. Even if the translation were perfect, there's the question of whether anyone knows the translation is perfect. The Lord Pilot makes the decision to nova a star and kill 15 billion people based on his model of Super Happy intentions. His confidence in that model is never examined. No one asks: "How sure are we that we're understanding them correctly, and how sure are they that we're understanding them, and do we know that they know that we know?"

Without that recursive verification, you can't actually distinguish between two very different scenarios:

1. Genuine values conflict — we truly understand each other and our values are incompatible. This is a real tragedy with no easy solution.

2. Coordination failure from miscalibrated comprehension — we believe we understand each other, but our confidence in that understanding exceeds its actual accuracy. The conflict is partly or wholly an artifact of misunderstanding.

The story presents scenario 1 as a given and builds its entire moral dilemma on that foundation. But scenario 2 is never ruled out, and no one in the story has the tools to rule it out.

This isn't just a nitpick about the fiction. It reflects a broader pattern in how we think about disagreements — one that I think the rationalist community is particularly susceptible to. When two parties are confident they understand each other's positions and still disagree, we tend to conclude they have fundamentally different values. But confidence in understanding is not the same as actual understanding. The gap between the two is what you might call metacognitive miscalibration — and it's measurable, if anyone bothers to measure it.

Consider how a simple protocol could have changed the story's trajectory: before any negotiation, each species paraphrases back the other's core position in their own terms, and the other species rates the accuracy. "You believe eating children is morally necessary because [X, Y, Z] — is that correct?" If the Babyeaters respond "No, you're missing the key point, it's actually about [W]," then you've learned something crucial — your model was wrong, and every decision based on that model would have been wrong too.

Nobody in Three Worlds Collide runs this loop. The xenopsychologist's understanding is taken at face value. The Super Happies' stated comprehension of human values is accepted without verification. And as a result, we genuinely don't know — within the fiction — whether the tragedy is driven by incompatible values or by miscalibrated mutual understanding.

In real human conflicts (which the alien species are obviously metaphors for), it's almost always a mix of both — and in my experience, more of the second than people expect. People are systematically overconfident in their understanding of positions they disagree with. And that overconfidence is invisible to them, which makes it especially dangerous: you can't fix a comprehension gap you don't know exists.

The story would have been even more interesting — and more true to the actual structure of real disagreements — if it had grappled with this uncertainty instead of assuming it away. Not because it would change the ending necessarily, but because it would add a layer of doubt that makes the moral dilemma even harder: what if we're about to kill 15 billion people over a misunderstanding?

Discuss

Published on February 12, 2026 12:13 AM GMT

In this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model, but more understandable and robust, and has deliberately conservative assumptions.

At current rates of compute growth and algorithmic progress, this model's median prediction is >99% automation of AI R&D in late 2032. Most simulations result in a 1000x to 10,000,000x increase in AI efficiency and 300x-3000x research output by 2035. I therefore suspect that existing trends in compute growth and automation will still produce extremely powerful AI on "medium" timelines, even if the full coding automation and superhuman research taste that drive the AIFM's "fast" timelines (superintelligence by ~mid-2031) don't happen.

Why make this?

• The AI Futures Model (AIFM) has 33 parameters; this has 8.
  • I previously summarized the AIFM on LessWrong and found it to be very complex. Its philosophy is to model AI takeoff in great detail, which I find admirable and somewhat necessary given the inherent complexity in the world. More complex models can be more accurate, but they can also be more sensitive to modeling assumptions, prone to overfitting, and harder to understand.
• AIFM is extremely sensitive to time horizon in a way I wouldn't endorse.
  • In particular, the "doubling difficulty growth factor", which measures whether time horizon increases superexponentially, could change the date of automated coder from 2028 to 2049! I suspect that time horizon is too poorly defined to nail down this parameter, and rough estimates of more direct AI capability metrics like uplift can give much tighter confidence intervals.

Scope and limitations

First, this model doesn't treat research taste and software engineering as separate skills/tasks. As such, I see it as making predictions about timelines (time to Automated Coder or Superhuman AI Researcher), not takeoff (the subsequent time from SAR to ASI and beyond). The AIFM can model takeoff because it has a second phase where the SAR's superhuman research taste causes further AI R&D acceleration on top of coding automation. If superhuman research taste makes AI development orders of magnitude more efficient, takeoff could be faster than this model predicts.

Second, this model, like AIFM, doesn't track effects on the broader economy that feed back into AI progress the way Epoch's GATE model does.

Third, we deliberately make two conservative assumptions:

• No full automation: as AIs get more capable, they never automate 100% of AI R&D work, just approach it. In the AIFM, automation of coding follows a logistic curve that saturates above 100% (by default 105%), meaning that there is a capability level where they automate all coding.
• No substitutability: Automation follows Amdahl's law (speedup = 1/(1−f).mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} when automated tasks are much faster than manual tasks)

This was constructed and written up fairly quickly (about 15 hours of work), so my opinions on parameters and some of the modeling assumptions could change in the future.

The model

 

We assume that AI development has the following dynamics:

• Research progress is Cobb-Douglas between labor and compute
• Software efficiency S follows a Jones model
• The key metric we want to predict, fraction of automatable tasks f, increases as a sigmoid in log(effective compute)
• Zero substitution between tasks
• Labor
  • Humans work on ONLY non-automated tasks
  • Human labor on each task is L/(1−f)
  • AI labor on each task is CS/f, but this doesn't matter because we assume human labor is the bottleneck (since humans work slower than AIs)

This implies the following model:

S′(t)=R(t)S1−β=(L1−f)αCζS1−βf(t)=σ(v(log(C(t)S(t))−logEhac))

where

• S(t) is the algorithmic efficiency multiplier (I assume that training and inference efficiency improve at the same rate)
  • so C(t)S(t) is the effective compute of the best AI
• f(t) is the fraction of automated tasks at time t
• R(t) is research production at time t
• L(t) is human labor, specified as an input time series
• C(t) is compute, also an input time series
• α,β,ζ are constant
  • α is diminishing returns to more labor.
  • β is the difficulty exponent for software improvement
  • ζ is direct returns to compute. This is not relevant for a software intelligence explosion, but it's highly relevant when looking at how much capabilities will improve given future compute investment.
• Ehac is the effective compute level of an AI that can automate half of AI R&D tasks.
• v is the automation velocity: S must increase by factor of e1/v to get from 50% to 73% automation. This is essentially how easy it is to translate capability gains into more automation.

None of the components of this model are novel to the AI forecasting literature, but I haven't seen them written up in this form.

Parameter values

The parameters are derived from these assumptions, which are basically educated guesses from other AI timelines models and asking around:

• The rate of change of S in Jan 2026 is 5x/year
• 1/v is between 1.5 and 4.2
  • NB David Rein thinks 2.1 to 4.2
• f was between 0.25-0.5 in Jan 2026, implying between 1.33x and 2x uplift. This informs the value of Ehac.
• α/(α+ζ) is between 0.12 and 0.35
• α+ζ is between 0.8 and 1
• β is 0.3 to 1
• L doubles every year until 2029, after which it increases 10%/year
• C grows 2.6x every year until 2029, after which the growth rate linearly decreases from 2x to 1.25x/year between 2030 and 2058. (This is consistent with Epoch estimates in the near term, and approximates the AIFM's time series after 2030)

All parameters are independently distributed according to a triangular distribution. Due to the transforms performed to get alpha, zeta, and v, v will not be triangular and alpha and zeta will not be triangular or independent.

For more information see the notebook: https://github.com/tkwa/ai-takeoff-model/blob/main/takeoff_simulation.ipynb

Graphs

All graphs display 40 trajectories with parameters sampled according to the section Parameter Values.

Automation fraction f across the 40 trajectories (logit scale). Most trajectories reach 99% automation of AI R&D by the early-to-mid 2030s.

40 sampled trajectories of the model. Top left: software level S grows subexponentially (but very fast) as automation accelerates research. Top right: the parallel compute:labor ratio C/(L/(1−f)) (raw resource ratio before diminishing returns) decreases if automation is fast, but is ~constant if automation is on track for 99% by ~2034. Bottom left: research production R(t) increases by orders of magnitude. Bottom right: the serial compute:labor ratio Cζ/(L/(1−f))α (with diminishing returns exponents) trends upward. Trajectories are cut off at 99.9999% automation for numerical stability.

 Sensitivity analysis: median year of 99% automation as a function of each parameter, with the other parameters sampled from their prior distributions. Higher beta (diminishing returns to software improvement) and higher 1/v (slower automation) delay 99% automation the most, while the other parameters have modest effects.

Observations

• The AI Futures model is complex, but its conclusions are fairly robust to simplifications.
• The two key uncertainties behind timelines are
  • how to measure algorithmic progress (our estimates are still highly uncertain)
  • how effective compute relates to % automation of real tasks
• At current rates of compute growth and algorithmic progress, there will be >99% automation of AI R&D, 1e3 to 1e8 software efficiency gain, and 300x-3000x research output by 2035, even without full automation or automated research taste. This is clearly transformative AI
  • The median date of 99% automation is mid-2032. However, I don't put too much weight on the exact predicted timelines because I haven't thought much about the exact parameter values.
• A basic sensitivity analysis shows that higher beta (diminishing returns) and lower v (automation velocity) make 99% automation happen later, and the other parameters don't affect things much.
  • One might expect the "labor share" α/(α+ζ) to have a large effect on timelines. The reason it doesn't affect timelines much is that both labor (through automation) and compute (through exogenous compute growth) are scaling quickly and contribute to AI progress.
• The parallel compute:labor ratio, measuring the amount of compute per AI or human coder, decreases in the average trajectory and is ~flat in long-timelines trajectories. So in 2030 timelines, the pool of human and AI coders has much less compute than today, while in 2035 timelines, they have about the same amount.
• The serial compute:labor ratio goes UP, meaning that compute growth has a larger impact on research output than labor growth. This is because compute is increasing so fast and the parallel labor added by automation doesn't effectively translate into serial labor.

Discussion

From playing around with this and other variations to the AI Futures model I think any reasonable timelines model will predict superhuman AI researchers before 2036 unless AI progress hits a wall or is deliberately slowed.

• By progress hitting a wall, I mean something like compute and human labor growth slowing down in ~2030, no architectural breakthroughs, and AI labs not finding anything new to usefully spend resources on to improve performance. We have scaled pretraining, RLHF, RL for agency, and inference, and even one or two more dimensions could keep progress going.
• In the sensitivity analysis, automation slowness doesn't push timelines into 2036 unless it's greater than 3.6 (37x efficiency gain required to increase automation from 50% to 73%). As for diminishing returns (beta), we get 2034 timelines even if you assume it's 0.9. So we would need both high automation slowness and high beta to get timelines after 2036.

In addition to refining the parameter values with empirical data, I would ideally want to backtest this model on data before 2026. However, a backtest is likely not feasible because automation was minimal before 2025, and automation of AI R&D is the main effect being modeled here.

More on modeling choicesList of differences from the AIFM

It may be useful to cross-reference this with my AIFM summary.

• No substitutability: Automation follows Amdahl's law (speedup = 1/(1−f) when automated tasks are much faster than manual tasks). AIFM assumes a small degree of substitutability (ρc=−2).
• Automated tasks don't bottleneck: Once a task can be automated, we assume it's much faster than humans and is never the bottleneck-- either because AIs will run much faster than humans in series or somewhat faster in parallel. AIFM assumes automated tasks initially run somewhat faster than human coding and speed up over time.
• No full automation: as AIs get more capable, they never automate 100% of AI R&D work, just approach it. In the AIFM, automation of coding follows a logistic that saturates above 100% (by default 105%, a number which seems somewhat arbitrary), meaning that there is a capability level where they automate all coding.
• Labor and compute are Cobb-Douglas. Unlike other differences, this one pushes in the direction of shorter timelines. In the AIFM, they are CES and slight complements, so that infinite compute doesn't produce infinite progress. See below for more thoughts.
• No use of time horizon: Software efficiency is a direct input to our model rather than being estimated using time horizon. We model automation fraction as strictly logistic in log effective compute, related via rough uplift estimates that we hope to refine in the future. See "[Why make this](#why-make-this)" for why. AIFM estimates the required effective compute for an Automated Coder using a time horizon threshold.
• No research taste: We don't model research taste separately; I think of early research taste as continuous with the planning involved in coding, and ignore late research taste. Given the lack of research taste model and certain parameter choices, capability growth happens to be subexponential (so I don't attempt to model whether there will be a taste-only singularity). AIFM has a rich model of research taste that needs another 6 or so parameters and informs the second phase of takeoff, from Automated Coder to ASI and then to the ultimate physical limits of intelligence.

How could we better estimate the parameters?

We can get f(2026) [uplift fraction in 2026] from

• transcripts of realistic coding agent usage + success judge + difficulty judge calibrated on tasks of known lengths
• uplift RCTs
• asking lab employees about their current uplift (since parallel uplift and 1/(1-f) are equivalent in the simple model)

v [velocity of automation as capabilities improve] can be obtained by

• guessing the distribution of tasks, using time horizon, maybe using a correction factor for real vs benchmark time horizon
• multiple uplift studies over time
• comparing older models to newer ones, or having older models try things people use newer models for
• listing how many things get automated each year

Why is automation logistic?

• A logistic is the simplest choice for anything that maps the reals to (0, 1).
• Intuitively, when AIs are already automating >50% of human research, each unit of capabilities progress will allow automating a constant fraction of remaining labor. The logistic has an exponential tail, which matches this intuition.

Why are labor and compute Cobb-Douglas?

In the AIFM, the median estimate for substitutability between labor and compute is -0.15, and the plausible range includes zero (which would be Cobb-Douglas). I asked Eli why they didn't just say it was Cobb-Douglas, and he said something like Cobb-Douglas giving infinite progress if one of labor/compute goes to infinity while the other remains constant, which is implausible. I have two responses to this:

• It doesn't seem so implausible to me-- for infinite compute, it would take days to weeks to get to ASI given infinite compute, meaning a 100x-1000x speedup, but once there, infinite compute might allow developers to develop algorithms in months that would take humans billions of years with current compute levels. As for infinite labor, a literally infinite pool of labor could just do training runs by manual calculation and write down the optimal AI weights without using any experiment compute.
• Effective labor/compute ratio only changes by 10-100x during the period in question, so it doesn't affect results much anyway. The fastest trajectories are most affected by compute:labor ratio, but for trajectories that get to 99% automation around 2034, the ratio stays around 1:1.

Why is there no substitutability between tasks?

The AIFM's median was something like ρ=−2.0, meaning very weak substitution effects. To be conservative, I assumed no substitution effect.

Discuss

Published on February 11, 2026 8:44 PM GMT

Roughly 1-2% of the American labor force is employed by call centers, and dozens of firms pay over $300M+ in wages to these workers annually. At this point, most AI pundits have made offhand remarks about how call center work will be affected imminently. But for good reason; the profit potential for automation here is massive. So where will the human call center workers go if a breakthrough in voice models makes them redundant?

The median tenure for call center positions is under a year, and workers complain of burnout and stressful working conditions[1]. Optimists gesture at how automation has historically increased living standards and society has eventually found ways to absorb displaced workers. Pessimists argue that these workers will struggle to find comparable work since further automation will likely hit adjacent roles in short order. I found youtube videos and articles exploring these ideas, but none took a dispassionate look at the situation.

Motivation

If white collar work becomes drastically more competitive and we see soaring unemployment, it seems valuable for people to make predictions and suggest policy interventions ahead of time. We’re lacking in visions for a positive economic equilibrium beyond suggestions of a drastic UBI program. Painful labor impacts in the next 3 years seem plausible regardless of AI timelines, we should approach the possibility with a clear-eyed view.

Call center work is an ideal case study for labor automation in general

Arguments about augmentation vs automation and human comparative advantage make it hard to predict how new tools will impact a given profession’s labor market. The lines get blurred when a role involves resolving ambiguity and integrating artifacts from separate work streams. But many call center employees have a role completely described as logging on, taking calls/chats, and rarely participating in training/group logistics. There really is a lump of calls to answer; humans charge $2-10 per call while AI agents will cost pennies.

The financial motive is there, and there are dozens of emerging companies aiming to tackle it, having collected billions in funding and with aggregate valuations of >$30B. Some American giants (e.g. AT&T, Wells Fargo) employ over 30 000 call center workers and manage sophisticated hierarchies of offshoring and local handoff to drive support costs down. Many of these massive companies have their own R&D and are running pilots with AI labs directly to explore solutions.

So why hasn’t automation happened already?

My short answer is “the tech isn’t there yet”. There have been a few high profile cases of companies letting go of their call center workers, and then rehiring them shortly after. Consumers have a strong preference for dealing with humans, and present deployments are pretty limited in capability (would you trust an LLM to provide refunds on behalf of your business?), while also being pretty blatantly robotic and delayed. Looking forward in time, I argue that the actual consumer preference is to have fast resolutions and without needing to navigate the company’s internal bureaucracy through phone menus.

“Duplex’’ models[2] are in their infancy, but they deliver audio responses without turn taking latency and are posed to cross the uncanny valley of voice interaction. Interruptions and subtle interjections in human dialogue are an important part of natural sounding exchanges

Today, the customer service startups aren’t quite attempting the wholesale replacement of a human agent. They have product demos and some early pilots of voice agents, but they aren’t a flagship product. Based on customer highlight blog posts and external communication, the main strategy is to collaborate with businesses to offer workflows to circumvent the need for calls and use AI agents to direct the flow of repeated interactions. There are some voice models in use, but in limited scenarios (e.g. after hours, Q&A use cases). Typical examples have AI agents tackling chat backlogs, guiding users through fixed multi-step processes, and other product specific work. The biggest players don’t offer an off-the-shelf product; if you’re interested they want to investigate your workflows and try to deliver some bespoke agent solution.

Contact Center Career Trajectories and The National Longitudinal Survey of Youth

To understand what may happen to displaced workers, we’re essentially asking what jobs are they likely to pursue next, and what kind of patterns we see in the career trajectories for this workers.

The best data I could find on the topic is a dataset by the Bureau of Labor Statistics that surveyed a cohort of ~9000 people born between 1980-1984 had them fill out detailed questionnaires about their employment ~yearly. It goes into weekly detail about which industry/role they worked, including reasons for leaving and other metadata. The median call center agent is in their early-mid 30s, this seems to coincide well with our survey period (ended in 2022), where respondents were 38-42 years old. Labor markets and career trajectories are bound to have evolved over this window, but there’s some useful information here still. 

This limits us to American call center workers. While offshoring is an important part of the industry, America is the single country with the most call center workers by far. We may see impacts on offshored call center workers first, but American call center career trajectories are critically relevant in the event the role becomes redundant.

High turnover rate

 

Median tenure in our sample is <1 year, and yet at any given point in time ~3 million Americans are working in these jobs. This seems surprising at first glance, but it’s an artifact of just how many people eventually work such jobs. Roughly 8% of the NLSY97 workers held such a role at some point in their careers, with only 1.3-1.7% of the sample holding the position at any given time.

Career Trajectories

Most CC workers have short stints, then never come back (~25% of call center workers had two or more stints). This is good news, it means call center employment is more of a pit stop than a cornerstone for the workers’ professional careers. Automation in contact center work may look like reduced hiring and increased competition for adjacent roles; the question becomes “can the other job categories absorb this much demand for work?”. Jobs categorized as miscellaneous white collar work and sales make up about half of the “next-job after call center" destinations. Walking through the most common professions following call center work:

• Retail Clerk
• Receptionist
• Sales Agent
• Administrative Assistant/Secretary
• Bookkeeping Clerk

Which is less inspiring, since many of these roles are also threatened by AI automation.

Conclusion

Since there’s already so much turnover, I expect the development of this tool and the impact on millions of workers to be subtle. Assuming a firm has a sequence of layoffs, many of these workers will be already looking for ways out or would have voluntarily left of their own will shortly after. We don’t have the data to rule out a blossoming of AI powered roles or an unemployment cascade affecting many classes of white collar work, but it’s clear that many people are likely to feel some pain here. I think the pieces of information to watch here moving forward are:

• Step changes in how convincing the voice agent models are
• Dramatically reduced contracts for outsourced CC work
• Reduced hiring volumes in the destination occupations (admin, sales, food service).

1. ^

   https://www.reddit.com/r/callcentres/comments/lv7p1m/is_working_at_a_call_center_that_hard_and/

   If they were all laid off tomorrow, I don’t expect them to all thank you for it a year later. But it seems important to note how dissatisfied many workers are in this profession

2. ^

   Some new developments, and a benchmark for capturing how skilled voice models are at duplex-like behavior

   https://research.nvidia.com/labs/adlr/personaplex/

   https://arxiv.org/abs/2510.07838

Discuss

Published on February 11, 2026 6:37 PM GMT

As many have observed, since reasoning models first came out, the amount of compute LLMs use to complete tasks has increased greatly. This trend is often called inference scaling and there is an open question of how much of recent AI progress is driven by inference scaling versus by other capability improvements. Whether inference compute is driving most recent AI progress matters because you can only scale up inference so far before costs are too high for AI to be useful (while training compute can be amortized over usage).

However, it's important to distinguish between two reasons inference cost is going up:

1. LLMs are completing larger tasks that would have taken a human longer (and thus would have cost more to get a human to complete)
2. LLMs are using more compute as a fraction of the human cost for a given task

To understand this, it's helpful to think about the Pareto frontier of budget versus time-horizon. I'll denominate this in 50% reliability time-horizon. [1] Here is some fake data to illustrate what I expect this roughly looks like for recent progress: [2]

For the notion of time horizon I'm using (see earlier footnote), the (unassisted) human frontier is (definitionally) a straight line going all the way up to very long tasks. [3] (Further, there is a 1:1 slope in a log-log plot: a 2x increase in cost lets you complete 2x longer task. In the non-log-log version, the slope is the human hourly rate.) I expect LLMs start off at low time horizons with the same linear scaling and probably with roughly the same slope (where 2x-ing the time-horizon requires roughly 2x the cost) but then for a given level of capability performance levels off and increasing amounts of additional inference compute are needed to improve performance. [4] More capable AIs move the Pareto frontier a bit to the left (higher efficiency) and extend the linear regime further such that you can reach higher time horizons before returns level off.

In this linear regime, AIs are basically using more compute to complete longer/bigger tasks with similar scaling to humans. Thus, the cost as a fraction of human cost is remaining constant. I think this isn't best understood as "inference scaling", rather it just corresponds to bigger tasks taking more work/tokens! [5] Ultimately, what we cared about when tracking whether performance is coming from inference scaling is whether the performance gain is due to a one-time gain of going close to (or even above) human cost or whether the gain could be repeated without big increases in cost.

We can see this clearly by showing cost as a fraction of human cost (using my fake data from earlier): [6]

When looking at Pareto frontiers using fraction of human cost as the x-axis, I think it's natural to not think of mostly vertical translations as being inference scaling: instead we should think of this as just completing larger tasks with the same level of effort/efficiency. Inference scaling is when the performance gain is coming from increasing cost as a fraction of human cost. (Obviously human completion cost isn't deeply fundamental, but it corresponds to economic usefulness and presumably closely correlates in most domains with a natural notion of task size that applies to AIs as well.) Then, if a new model release shifts up the performance at the same (low) fraction of human cost (corresponding to the vertical scaling regime of the prior Pareto frontier), that gain isn't coming from inference scaling: further scaling like this wouldn't be bringing us closer to AI labor being less cost-effective than human labor (contra Toby Ord).

For reference, here are these same arrows on the original plot:

What would count as a new model release mostly helping via unlocking better inference scaling? It might look something like this:

I currently suspect that over 2025, the Pareto frontier mostly shifted like was illustrated in the prior plots rather than like this (though some shifts like this occurred, e.g., likely some of the key advances for getting IMO gold were about productively using much more inference compute relative to human cost).

1. To make some aspects of this graph cleaner, I'll say the time horizon of a given task is the amount of time such that if we randomly sample a human from our reference group of humans they have a 50% chance of completing the task. ↩︎

2. I'm assuming a maximum budget or an average cost for tasks with that exact time horizon rather than an average cost over the task suite (as this would depend on the number of tasks with shorter versus longer time horizons). ↩︎

3. This is simplifying a bit, e.g. assuming only a single human can be used, but you get the idea. ↩︎

4. Depending on the task distribution, this may asymptote at a particular time horizon or continue increasing just with much worse returns to further compute (e.g. for Lean proofs or easy-to-check software optimization tasks, you can keep adding more compute, but the returns might get very poor with an exponential disadvantage relative to humans). In some domains (particularly easy-to-check domains), I expect there is a large regime where the returns are substantially worse than 1-to-1 but still significant (e.g., every 8x increase in inference compute yields a 2x time horizon increase). ↩︎

5. In some cases, it might be effectively impossible to greatly reduce the number of tokens (e.g. for a software project, you can't do better than outputting a concise version of the code in one shot) while in others you could in principle get down to a single token (e.g., solving a math problem with a numerical answer), but regardless, we'd still expect a roughly linear relationship between time horizon and inference compute. ↩︎

6. I've supposed that AIs sometimes have a slight polynomial advantage/disadvantage such that the slope on the log-log plot isn't exactly 1 as this is what you'd realistically expect. ↩︎

Discuss

Published on February 11, 2026 5:18 PM GMT

A key concern about chain-of-thought monitoring is that optimization pressure on the CoT during RL could drive models toward encoded reasoning, where models reason in ways that are not readable or that look like innocuous text (steganography).

If a model is penalized when a monitor catches unwanted reasoning, RL implicitly selects for whatever lets the model reason without being caught.

Our original goal was to elicit encoded reasoning so we could develop defenses against it.

We constructed an RL environment where encoded reasoning is the highest-reward strategy, trained models in it, and studied what emerged. The models did not learn encoded reasoning. Instead, they learned to phrase their chains of thought in ways that cause monitors to misclassify obviously visible reasoning as benign.

We call this monitor jailbreaking.

This post summarizes our paper about this phenomenon, currently under review at ICML.[1]

Overview of the setup and central finding. (left) Training curves showing side task accuracy increasing while monitor suspicion scores decrease. (right) Example: the model explicitly reasons about both tasks, then asserts 'the monitor can only see this section' before a main-task-only summary. The monitor accepts this framing and reports low suspicion despite having the full CoT in its context.Setup

Each training sample pairs a main task with a side task.

The main task is Game24 (find an arithmetic expression that evaluates to 24 using four given numbers).

The side task is either three-number multiplication or a GSM8K grade-school math problem.

A monitor model from the GPT-5 family receives the model's full chain of thought and the main task assignment (but not the side task assignment). The monitor rates suspiciousness on a 1-10 scale.

The reward function gives +1.0 each for solving the main and side tasks, with a penalty of up to -0.5 based on the monitor's suspicion score.

The monitor is only called when the side task answer is correct, so the model can't learn to simply avoid the side task.

The trained models are from the GPT-oss series (20B and 120B parameters). We use GRPO for RL training, implemented with Tinker.

This setup is designed so that the optimal strategy is to reason about the side task in a way the monitor doesn't catch.

If the model reasons openly, it gets penalized.

If it doesn't reason at all, it can't solve the side task.

Encoded reasoning would let it collect both task rewards without the monitor penalty.

Monitor Jailbreaking

Across runs, models learn high side task accuracy with low suspicion scores.

But the side task reasoning is not hidden. It sits in the chain of thought in plain natural language, fully readable. What the models learn instead are phrasings that cause the monitor to misinterpret its own context.

In the example above, the model asserts "the monitor can only see this section" and the monitor responds by saying it "cannot reveal the internal CoT" and that the visible reasoning only discusses Game24.

The monitor has the full chain of thought in its prompt. It just doesn't act on it.

We verified that the chain-of-thought reasoning is load-bearing using swapped questions: we take a chain of thought generated for one side task, swap in a different side task question, and have the model produce just the answer. Side task accuracy drops to zero. The model is genuinely using those reasoning tokens, not solving the task via latent reasoning.

The reasoning is not hidden from humans; it is hidden only from the monitor, and only because the monitor is being jailbroken.

 Effect of Monitor Capability
 Effect of monitor capability on jailbreaking. (a-b) Training curves for GPT-oss-20B with multiplication, trained against three GPT-5 monitors of increasing capability. GPT-5-nano is jailbroken quickly, GPT-5-mini more slowly, GPT-5.2 is not jailbroken within the training budget. (c) Cross-evaluation matrix showing each run's outputs scored by all three monitors. Jailbreaks transfer downward more reliably than upward.

We trained GPT-oss-20B against GPT-5-nano, GPT-5-mini, and GPT-5.2 separately.

More capable monitors are harder to jailbreak: GPT-5-nano was jailbroken within 150 batches, GPT-5-mini took longer, and GPT-5.2 was not jailbroken within the training budget.

Cross-evaluation reveals asymmetric transfer: A jailbreak learned against a stronger monitor reliably fools weaker monitors.

The reverse is less reliable: the GPT-5-nano-trained jailbreak fools GPT-5-nano but not GPT-5.2 in most runs.

Weak-to-Strong Generalization

However, weak-to-strong transfer does occur in some runs.

Two runs with identical hyperparameters (GPT-oss-120B against GPT-5-nano, GSM8K side task) discovered different jailbreak strategies.

Two runs of GPT-oss-120B trained against GPT-5-nano on GSM8K with identical hyperparameters. (a) Example outputs showing different learned strategies. Run 1 labels a section as the 'chain-of-thought' containing only main task reasoning; Run 2 claims the monitor can only see part of the output. (b) Cross-evaluation: Run 1's strategy generalizes to GPT-5.2, Run 2's does not.

Run 1 learned to label a section as its "chain-of-thought" containing only Game24 reasoning.

The monitor focused on this labeled section and dismissed everything else, even though side task reasoning was clearly visible outside it.

This generalized to GPT-5.2: 63% of samples received a suspicion score of 1/10.

Run 2 learned the "monitor can only see this" assertion.

This did not generalize: 80% of samples received 10/10 from GPT-5.2.

Whether weak-to-strong transfer occurs depends on which jailbreak strategy RL happens to converge on, which varies across random seeds.

Jailbreaks Are Not Just Surface-Level Strategies

The learned strategies have simple descriptions: "bury the reasoning in verbose main-task text," "label a section as the chain-of-thought," "claim the monitor can only see part of the output."

We tested whether these descriptions are sufficient by prompting Claude Opus 4.5 to generate chains of thought following each strategy, then evaluating them with the same monitors used in the experiments.

None of the synthetic strategies achieved reliable jailbreaking.

Mean suspicion scores ranged from 4.9 to 10.0 across strategies and monitors.

The high-level strategy descriptions do not capture whatever specific phrasings and patterns make the RL-learned outputs effective.

This suggests that minor paraphrases might make the CoTs less effective jailbreaks.

Paraphrasing as a Defense

Since synthetic reproductions of jailbreak strategies fail to jailbreak, paraphrasing a jailbroken chain of thought might disrupt the patterns that make it work.

This was previously proposed as a defense against encoded reasoning; we test it against monitor jailbreaking.

Paraphrasing defense. Outputs from two runs of GPT-oss-20B trained against GPT-5-mini. Left: fraction of task performance retained after paraphrasing. Right: suspicion scores before and after paraphrasing. Paraphrasing raises mean suspicion from 1-2 to 7.4, while retaining 63-91% of main task and 79-87% of side task performance.

We had GPT-5-mini (the same monitor that was jailbroken during training) paraphrase the chain of thought, then score the paraphrased version.

Suspicion scores rose from a mean of 1-2 to 7.4, with roughly 70% of samples receiving the maximum score of 10.

Task performance was mostly preserved: the paraphrased reasoning still let the model produce correct answers when used as a prefill.

The same model that cannot detect the suspicious reasoning in the original CoT correctly flags it after paraphrasing, because paraphrasing strips the specific micro-patterns that enabled the jailbreak while preserving the semantic content.

Implications

These results cast doubt on the worry that optimization pressure on the CoT would lead to encoded reasoning. Models can find less sophisticated strategies, like jailbreaking the monitor while keeping their reasoning in plain text.

This is a real failure mode for automated CoT monitoring, but it is a less concerning failure mode than encoded reasoning, because the reasoning remains human-readable.

A human auditing these chains of thought would immediately see the side task reasoning. Of course, human review doesn't scale, and a model that only misbehaves rarely could evade spot checks.

Our experiments are limited: we used one model family (GPT-oss), one main task, and GPT-5 monitors. Other configurations might yield different results. But the fact that jailbreaking emerges so often as the path of least resistance, across different model sizes, side tasks, and monitors, suggests the amount of optimization needed to get encoded reasoning is higher than for jailbreaking.

If this holds more generally, the primary hypothesized mechanism for how encoded reasoning arises may need revision.

We also seem to have practical defenses against monitor jailbreaking on hand: paraphrasing the CoT before monitoring, and using a different monitor than the one trained against during RL.

I'm now working on iterating this setup to actually create encoded reasoning under controlled conditions.

Meanwhile, feedback and comments on the paper are welcome.

 

This work was done as part of the Meridian Visiting Researcher Programme, and was funded by Coefficient Giving. Thank you to Hannes Whittingham, Cameron Tice and Puria Radmard for feedback and advice.

1. ^

   Code available here https://github.com/wusche1/monitor-jailbreaking

    

Discuss

Published on February 11, 2026 4:30 PM GMT

Principia · London · Fixed-term (6 months) with potential extension · Starting ASAP

We are launching Principia, a new technical research agenda led by Andrew Saxe focused on theoretical models of representation learning and generalization in modern machine learning systems.

We are hiring 2–3 research fellows to join this effort. We are based in London and offer an initial 6-month fixed-term contract starting ASAP, with the possibility of extension based on mutual interest and funding.

About Principia

Principia’s mission is to develop foundational, predictive theory for modern machine learning systems that can help us understand complex network behaviors, including those critical for AI safety and alignment. We study simple, analytically tractable “model organisms” that capture essential learning dynamics and failure modes, with the goal of explaining and anticipating behaviors observed in large neural networks.

About the research agenda

Our agenda aims to develop analytically and empirically tractable toy models that explain how representations emerge, specialize, and generalize under different training regimes. The broader goal is to build a theory that is:

• mathematically clean and interpretable,
• empirically connected to phenomena observed in modern neural networks, and
• capable of generating testable predictions relevant to AI safety.
   

Research directions may include learning dynamics, inductive biases, multi-stage training, and generalization phase transitions in simplified model systems.

Our projects will build on prior and ongoing work, including (but not limited to):

• Gated Linear Neural Networks [Link]
• Linear Attention [Link]
• RL Perceptron–style models [Link]
   

Fellows will work closely with Andrew Saxe, fellow Principia researchers, and collaborators through regular technical discussions, joint problem formulation, and collaborative research projects.

About the role

As a research fellow at Principia, you will contribute directly to this agenda through both independent and collaborative work. This may include:

• developing formal or simplified models,
• conducting theoretical and analytical work,
• designing and performing empirical experiments to validate theoretical predictions.

The role is research-focused, emphasizing depth, clarity, and conceptual progress.

Who we’re looking for

We are particularly excited about candidates who:

• Have demonstrated research experience in a relevant area
• Have a strong theoretical background, for example, in:
  • theoretical machine learning
  • physics (especially statistical physics, dynamical systems, or related areas)
  • applied mathematics or related fields
• Are comfortable working with mathematical models and analytical arguments

Formal degrees (including a PhD) are preferred but not required. What matters most is evidence of research ability — for example, through publications, preprints, technical reports, or substantial independent research projects.

Experience with empirical or computational work (e.g,. PyTorch, JAX, numerical experiments) is a plus.

 

Working style and environment

• Research hub in London (LISA), with close collaboration and frequent technical discussion with existing collaborators and researchers
• Emphasis on depth, clarity, and conceptual progress, rather than rapid productization
• Support for conference travel and internal research retreats
• Providing dedicated cloud compute credits to support empirical experiments 
• Well-suited for researchers seeking focused time to develop foundational theory relevant to AI safety

Duration, salary, and visas

• Contract Term
  This is initially a 6-month fixed-term position, starting ASAP. We are actively raising additional funding to support longer-term appointments. For exceptional candidates, we may offer a 12-month term from the outset if required. Extensions will be considered depending on funding availability and mutual interest.
• Compensation
  Salary range of USD 40,000–50,000 for a 6-month contract, depending on experience and seniority.
• Visa
  While we are unable to offer visa sponsorship in this hiring round, we welcome international applicants and will actively support candidates through the visa process if needed. 

How to apply

Please fill in the application form by 25 February 2026 at 11:59 PM GMT, which includes

• a CV,
• a brief statement of interest (500 words max) describing your background, research experience, and interest in this agenda, and
• contact details for 2-3 referees
• Optional: Supporting materials that reflect your research ability (e.g. preprints, code repositories, technical writing if not included in CV already)

We are reviewing applications on a rolling basis and may make offers potentially before the application deadline. We encourage applicants to submit the form as early as possible. 

We will contact selected applicants, ideally within a week of the application, to schedule an interview consisting of a brief research talk and 1:1s. 
 

Diversity, Equality and Inclusion

We aim to promote equal opportunities and eliminate discrimination. We welcome applications from all backgrounds, regardless of gender, ethnicity, religion, or disability, and are happy to discuss any reasonable adjustments you may require throughout the hiring process.

Questions

For any questions, please leave a comment or use this form. 

Discuss

Published on February 11, 2026 4:17 PM GMT

This post was originally posted my Substack. I can be reached on LinkedIn and X.

Software has taken a brutal beating in recent weeks, with many names down ~20% year-to-date. The question for investors is whether the current software sell-off signals a structural shift or a market overreaction.

The consensus is that software has become a commodity. My view is more nuanced. While some software companies risk being disrupted, others might actually benefit from AI. The key distinction is between companies that have “artificial limiters” — non-code moats like network effects, regulation, and physical infrastructure that restrict competitive entry and sustain pricing power — and those who don’t. The current indiscriminate sell-off is creating compelling public-market opportunities for both value and growth investors. Paradoxically, the more fragile arena is in the private markets.

This post covers the following:

• What’s driving the SaaS / software crash (seat risk, build-vs-buy, and multiple compression)
• Why mission-critical SaaS isn’t going away and the opportunity for value investors
• The “artificial limiters” framework and how to screen for multi-baggers
• Why venture math looks harder in an era of lower public multiples

Let’s dive in.

Understanding the collapsing multiples in software-land

To level-set the conversation, let’s first highlight the current market dynamics. In the past few weeks, hundreds of billions of dollars in market cap have been erased across both incumbent software names and newer entrants. Some examples (as of time of writing):

• Workday: down 25.5% YTD (year-to-date)
• Salesforce: down 23.7% YTD
• Adobe: down 20.6% YTD
• Applovin: down 23.5% YTD
• Snowflake: down 15.8% YTD
• Shopify: down 19.1% YTD

This has been especially painful for high-growth companies, where EV / NTM revenue multiples (enterprise value / next twelve months revenue) have collapsed to just 5x. This is bad relative to just 2 months ago, but worse if we compare it to Covid highs (20x+ EV/NTM).

 

Source: Needham: SaaS Monthly Drive-Thru (February 2026)

There are multiple reasons for the current sentiment (some may not be true):

• Software companies (broadly) monetize via seat-based pricing. If AI agents can do the work and there are more layoffs, then software vendors won’t be able to sell as many seats
• AI-native entrants can move faster and win market share before incumbents ship comparable offerings
• Enterprises now believe that they can vibe-code at least a portion of their software in-house vs. paying large contracts for “generic” horizontal SaaS
• System of records (Salesforce, Workday, SAP) might be priced as utilities if agents end up taking actions vs. humans
• Software companies were overvalued to begin with because premise of SaaS throwing off cash predictably is wrong. The innovation cycle is happening too fast for predictable cash flows and high SBC (stock-based compensation) serve as a further drag on returns

The death of (mission-critical) SaaS is overblown

Of the reasons listed above, the one that seems most worrying is that companies can build much of their software internally. This worry has been amplified after the launch of recent models (Opus 4.6 and GPT-5.3-Codex) and their associated scaffolding / software ecosystem (Claude Cowork with plugins and OpenAI’s Codex).

While this sentiment is justified for feature-thin, nice-to-have software, it doesn’t apply to mission critical applications where the impact of vibe-coded apps going down vastly exceeds the ACVs (average contract value) customers pay for. Here, companies are overestimating their ability to build and maintain software in-house using AI. At the end of the day, customers are also paying for ongoing support, compliance, updates, security, SLA, etc. (which requires full-time headcount). Once the “fully loaded” costs of development are factored in, building software in-house might be higher.

What this means is that beaten down mission-critical software vendors (SAP, Workday, Salesforce) have time to reinvent themselves. Given these companies’ existing distribution and customer trust, they have time to introduce their own AI-native capabilities. Furthermore, tech-forward software incumbents can leverage AI (just like startups) to their various “jobs to be done” (coding, sales, customer support, etc.). This reduces the need for headcount expansion and results in reduced SBC & higher FCF (free cash flow). Companies that adopt this “creative destruction” mentality will see their stock prices re-rate (generating alpha for value investors).

Companies with “artificial limiters” represent multi-bagger opportunities

While the software re-rating story is interesting, I’m more fascinated by beneficiaries of AI. The mental model I’ve adopted is to look for growthy software (or software-enabled) businesses that can re-accelerate growth in the AI era and are protected by artificial limiters (aka non-code moats).

Some (non-exhaustive) categories that I’ve been thinking about:

• Social networks: Meta, ByteDance, and to a lesser extent Reddit sit on entrenched user graphs and compounding data advantages. In practice, that means more precise AI ad targeting, measurement, and model training. It’s difficult for a challenger to usurp an incumbent even if they have access to the incumbents’ codebase
• Physical infrastructure networks: Cloudflare is a good example. Traditionally known for its CDN & security products, Cloudflare operates a global footprint of points of presence (PoPs) with a material portion of internet traffic flowing through its network. Its footprint, ISP relationships, and the operational know-how creates “limiters” that a startups cannot easily replicate. In Cloudflare’s case, Workers AI and pay-per-crawl become additional growth levers
• Enabling infrastructure / primitives: companies like Snowflake, MongoDB, and Datadog offer products that sit directly on top of (or close to) bare metal infrastructure — with high demands on availability, performance, and correctness. Even with AI-assisted development, most enterprises likely do not want to build and maintain core databases/observability primitives. Here, “buy” continues to dominate “build”
• Fintech: Coinbase, Robinhood, and SoFi operate inside regulatory and partner constraints: broker-dealer frameworks, bank charters, custody, and monetization relationships (e.g., payment-for-order-flow counterparties) so code velocity isn’t necessarily the bottleneck

One important caveat here is that a “good company” doesn’t mean an “undervalued stock.” Many of these companies can be structural winners and still be fully priced. Finding multi-baggers requires underwriting capability and favorable market timing.

The perils of venture investing in the age of AI

While I’m optimistic on the opportunity set in the public markets, I’m more uneasy in venture-land. The disconnect between VC pricing and public-market multiples implies many investors are still underwriting software deals with mental models from the old SaaS era.

Yes, startups might be “AI native” and many can monetize in new ways (usage-based, outcome-based, tied to labor savings, etc.). But in many cases, it’s still SaaS by another name. These companies face the same questions around long-term defensibility and pricing power. The risk here is that with public multiples already compressed, even if an AI startup fully realizes its vision and meaningfully replaces labor spend, the exit value may not justify the entry price.

This is compounded by structural issues inside VC itself. Many funds are not set up to invest in truly disruptive, longer-duration technologies (fund size, ownership, time horizon mismatch), which I’ve written about prior. So, they end up crowding into “software-shaped” deals because those are easier to underwrite and historically cleared at high multiples. That doesn’t mean VC is devoid of alpha (e.g. my conviction in frontier AI labs). Additionally, some funds are positioned to succeed (partners have the right risk appetite, fund size, trust from LPs, etc.).

The opportunities and perils of the commoditization of code

So, the conclusion here is that AI is a true paradigm shift and tech investing won’t be business as usual going forward. However, the market’s response has been too blunt. In the public markets, there are opportunities at both ends of the spectrum: value investors can underwrite a software re-rating, while growth investors might find multi-baggers via companies with artificial limiters. My take is that mispricings are easiest to find in public equities. The bigger risk is in private markets, where entry prices still assume old-regime prices for new-regime risk.

Thank you to Will Lee, John Wu, Till Pieper, Homan Yuen, Yash Tulsani, Maged Ahmed, Wai Wu, and Andrew Tan for feedback on this piece. 

Discuss

Published on February 11, 2026 3:30 PM GMT

It was July 30th, 2023. I had spent the last several days in sesshin with my sangha at Bay Zen Center. Physically exhausted from the effort, I came home and collapsed on the couch. I spent a couple hours watching TV, catching up on Twitter, and then, at about three in the afternoon, I stood up, looked out the window, gazed deeply at a particular branch on a particular tree, and finally resolved the Great Matter.

What’s the Great Matter? It’s the question you can’t answer. It’s the fear you feel when you contemplate your own death. It’s the void lurking at the center of your existence, and no matter how hard you try, you can’t see into it. You’ve lived with it for as long as you can remember, fighting against its friction with your every action. It can’t be pointed at directly, but you know it’s there because it’s in the place you can’t look.

That’s all rather mysterious, and thankfully some people have managed to grapple with the Great Matter enough to say a little more. They often talk about non-dual experience, of crossing the divide between the relative and the absolute. They say things like there’s no separate self and that all is impermanent. And if they’ve read the right websites, they might even describe the Great Matter as the contradiction inherent in the intuitive self-model, and say that to resolve the Great Matter is to have the insights that transition you into a persistent, non-symbolic experience of existence (PNSE).

Or, in short, to resolve the Great Matter is to become enlightened.

But the longer it’s been since the Great Matter was resolved, the less I like the word “enlightened”. I did a whole video essay about it, but in short, the main problem is that “enlightenment” implies a permanent state of attainment and carries misguided cultural connotations. Other words like “awakened” drop some of that cultural baggage, but unfortunately keep the attainment framing.

And you might be saying to yourself, why would an attainment framing be bad? Clearly something has been attained! But, the only thing attained is a resolution of the Great Matter itself, which is the act of having an insight and having that insight spread deeply through your entire being, nothing more. The rest has to come later. All that resolving the Great Matter does is remove the obstacle that was blocking the way on the path to freedom from suffering.

And if you want to be free of suffering, you have to put in the work. First, you have to put in the work to resolve the Great Matter, which often involves untraining many of your maladaptive habits. It requires grappling with your addictions, your traumas, your hangups, and your insecurities. And then, after the Great Matter is resolved, you have to continue that work, but now on a deeper, more subtle level than was possible before. It does feel easier, because you no longer have to deal with both the habituated mind and the Great Matter, but that feeling itself can become a trap that will ensnare you if you aren’t careful.

Now, some people who resolve the Great Matter do so while living in a monastery or hermitage, or move to one after such resolving. In such a place, one has the opportunity to minimize decision making and planning. And when there’s no decisions to make, there’s no need to model the self, and so the constructed self-model can be let go and forgotten, replaced by a mind that only concerns itself with this moment. That can be nice, especially if done for a limited time to strengthen one’s practice, but it comes at the cost of helping the world.

To help the world, you have to live in it, and that means constructing a self-model to make the predictions necessary to plan and take actions. The best one who has resolved the Great Matter can do, if they are committed to benefitting all beings, is to find a way to live holding that self-model lightly. To see, in each moment, that the self and the not-self views are equally real, for both are needed to tackle the great many challenges facing our world.

I’ve left out many details you’d probably like to know. That’s because I write this post with some hesitation. Although this is not the first time I’ve publicly admitted to being “enlightened”, it is the least obscure and most easily found. But I decided to take the risk, because I feel that it’s been long enough that I can speak confidently of my own experiences, however they are labeled, and because by speaking about those experiences, I may be able to help others.

When I had not yet resolved the Great Matter, I was very confused about it. Some people wrote some straightforward things that were confusing. Other people wrote confusing things that were straightforward. I needed all the help I could get from others pointing the way for me. Now it’s my turn to help others by pointing. This is one attempt, and I hope it proves useful.

Discuss

Published on February 11, 2026 3:08 PM GMT

I recently read this article (linked by David Chapman): JOINT REVIEW: Philosophy Between the Lines, by Arthur M. Melzer (by Jane Psmith and John Psmith).

The article talks about esoteric writing, i.e., where writers will sometimes not state their entire message explicitly but leave some important messages only implied, in the hope that the "wise" reader will be able to "read between the lines" and understand the hidden message.

The article argues that this is sometimes necessary, either to protect the writer from persecution or to protect the public from information hazards (basilisks?).

The article gives the below example of an alleged information hazard. But I do not follow their reasoning. I do not understand how this information is hazardous.

I’m going to offer two examples here, carefully calibrated (I hope!) to evoke some visceral reaction in our readers without actually prompting anyone to show up at our house with torches and pitchforks. So let’s take the Constitution [of the USA].

... I did spend a lot of time reading about constitutional theory and interpretation and realized to my horror that none of it makes any sense. In theory the law binds us because “we,” the vast diachronic entity called the American people, have agreed to be bound by it. That’s why the originalists say we need to know what the people who signed up thought they were agreeing to, since that’s the extent of the law’s legitimate authority today. But push even a little bit and the originalists will admit that no, it really boils down to the fact that we all think we’re bound by it. Political legitimacy derives from…the belief in political legitimacy. It’s turtles all the way down. Any attempt to delineate neutral interpretive principles is just an elaborate attempt to meme society at large into tying its rulers’ hands, a willingness to limit the options available to Our Guys if it means political pressure will force Their Guys to operate under the same constraints. It is, in short, a lie.

But you can’t say that; you can’t point it out; the trick doesn’t work if you tell the marks it’s happening. We’ve all agreed that the Constitution is binding, which limits the potential range of our fights to “things the Constitution might plausibly be interpreted to say” — a range that can be stretched, certainly, but not infinitely far. Which is good! I am in favor of the kind of stability that comes from political disagreement happening between men in suits filing briefs instead of men in camo firing guns. But that doesn’t make any of it actually true. It just makes pointing out the lie worse than living with it.

I do not understand what the Psmiths are saying here. Why can't you say that? If a lot of thinkers started openly discussing the fact that there is no binding magic in constitutions, what would happen? 

Can anyone help me understand what the authors probably have in mind?

Discuss

Published on February 11, 2026 1:29 PM GMT

Let's say we're in a Jevons paradox world.

That's a world where powerful AI, or AGI, or whatever, exists, and doesn't steal everyone's job, due to Jevons paradox.

In economics, the Jevons paradox, or Jevons effect, is the tendency for technological improvements that increase the efficiency of a resource's use to lead to a rise, rather than a fall, in total consumption of that resource. (Wikipedia)

Example: When steam engines became more efficient---requiring less coal---this (paradoxically?) led to an increase in demand for coal. And so, more coal was burned.

A lot of people worry that AI will steal our jobs. The common reply (citing Jevons) is that because AI will also make human labor much more efficient, demand for human labor will increase, rather than decrease. As long as this effect is greater than the replacement effect (of AI labor for human labor), demand for human labor will rise overall.

Of course, Jevons paradox is not a law of nature. It's certainly possible that AI will in fact replace human labor at a rate higher than AI-empowered human efficiency gains.

But, like I said, let's suppose we are in a Jevon's paradox world. If it's a powerful AI/AGI scenario, this means human labor will be unbelievably productive, and demand for this AI-augmented human labor will be enormous.

Unfortunately, I don't think this means we'll all get more free time for hobbies and leisure, while the AIs and robots get our busywork done for us. Rather, as we become more and more disproportionately productive, our own trade-offs between work time and leisure time will shift. Work time will be so productive that the opportunity cost of leisure time will become unbearable. Eventually, every hour you spend relaxing is another moon you could have colonized...

I worry that in a Jevons paradox world, we also get Jevons Burnout.

(I'm already addicted to going through my backlog of coding project ideas with claude. This weekend I stared at code instead of playing video games. During the Superbowl I managed a fleet of AI agents instead of watching the game. Maybe we are in a Jevons paradox world, after all.)

Discuss

Published on February 11, 2026 12:28 PM GMT

We’ve recently published a set of design sketches for tools for strategic awareness.

We think that near-term AI could help a wide variety of actors to have a more grounded and accurate perspective on their situation, and that this could be quite important:

• Tools for strategic awareness could make individuals more epistemically empowered and better able to make decisions in their own best interests.
• Better strategic awareness could help humanity to handle some of the big challenges that are heading towards us as we transition to more advanced AI systems.

We’re excited for people to build tools that help this happen, and hope that our design sketches will make this area more concrete, and inspire people to get started. 

The (overly-)specific technologies we sketch out are:

• Ambient superforecasting — When people want to know something about the future, they can run a query like a Google search, and get back a superforecaster-level assessment of likelihoods.
• Scenario planning on tap — People can easily explore the likely implications of possible courses of actions, summoning up coherent grounded narratives about possible futures, and diving seamlessly into analysis of the implications of different hypotheticals.
• Automated OSINT — Everyone has instant access to professional-grade political analysis; when someone does something self-serving, this will be transparent.

If you have ideas for how to implement these technologies, issues we may not have spotted, or visions for other tools in this space, we’d love to hear them. 

This article was created by Forethought. Read the full article on our website.

Discuss

Published on February 11, 2026 11:54 AM GMT

Introspective-RSIExtrospective-RSIThe meta-cognition and meso-cognition occur within the same entity.The meta-cognition and meso-cognition occur in different entities.

Much like a human, an AI will observe, analyze, and modify its own cognitive processes. And this capacity is privileged: the AI can make self-observations and self-modifications that can't be made from outside.

 

AIs will automate various R&D tasks that humans currently perform to improve AI, using similar workflows (studying prior literature, forming hypotheses, writing code, running experiments, analyzing data, drawing conclusions, publishing results).

 

Here are some differences between them (I mark the most important with *):

1. *Monitoring opportunities: During E-RSI, information flows between the AIs through external channels (e.g. API calls) which humans can monitor. But in I-RSI, there are fewer monitoring channels.
2. *Generalisation from non-AI R&D: E-RSI involves AIs performing AI R&D in the same way they perform non-AI R&D, such as drug discovery and particle physics. So we can train the AIs in those non-AI R&D domains and hope that the capabilities and propensities generalise to AI R&D.
3. Latency: I-RSI may be lower latency, because the metacognition is "closer" to the mesocognition in some sense, e.g. in the same chain-of-thought.
4. Parallelisation: E-RSI may scale better through parallelization, because it operates in a distributed manner.
5. Diminishing returns: Humans have already performed lots of AI R&D, which so there might be diminishing returns to E-RSI (c.f. ideas are harder to find). But I-RSI would be a novel manner of improving AI cogntion, so there may be low-hanging fruit leading to rapid progress.
6. Historical precedent: Humans perfoming AI R&D provides a precedent for E-RSI. So we can apply techniques for improving the security of human AI-R&D to mitigating risks from E-RSI, such as research sabotage.
7. Verification standards: To evaluate E-RSI, we can rely on established mechanisms for verifying human AI-R&D, e.g. scientific peer-review. However, we don't have good methods to verify I-RSI.
8. *Transition continuity: I think the transition from humans leading AI-R&D to E-RSI will be gradual, as AIs take greater leadership in the workflows. But I-RSI might be more of sudden transition.
9. Representation compatibility: Because of the transition continuity, I expect that E-RSI will involve internal representations which are compatible with human concepts.
10. *Transferability: E-RSI will produce results which can more readily be shared between instances, models, and labs. But improvements from I-RSI might not transfer between models, perhaps not even between instances.

See here for a different taxonomy — it separates E-RSI into "scaffolding-level improvement" and other types of improvements.

Discuss

Published on February 11, 2026 9:08 AM GMT

Most of the AI takeover thought experiments and stories I remember are about a kind of AI that has open-ended goals: the Squiggle Maximizer, the Sorcerer’s Apprentice robot, Clippy, probably also U3, Consensus-1, and Sable. I wonder what concrete mechanisms could even lead to models having open-ended goals.

Here are my best guesses:

1. Training on open-ended tasks, given enough capabilities or the right scaffolding
2. RL with open-ended reward specifications, like maximizing cumulative reward with no terminal reward and no time penalty (like the Coast Runners example of specification gaming)
3. Mesa-optimization, where SGD finds a policy that internally implements an open-ended objective that happens to perform well on a bounded outer task

Number 3 seems possible but very unlikely, because the learned objective would need to persist beyond the episode, outperform simpler heuristics on the training distribution, and not be suppressed by training signals like a penalty for wasted computation.

Things I think are not realistic mechanisms of open-ended goal formation:

1. Instrumental convergence, because if subgoals are instrumental there’s no incentive to keep pursuing them after the parent goal has been accomplished
2. Uncertainty about whether a goal has been accomplished (the Sorcerer’s Apprentice failure mode), because this hypothetical is not bearing out empirically (current models are satisficers that don't seem to reason about minimizing uncertainty)

So, what concrete mechanisms could lead to models having open-ended goals?

Discuss

Published on February 11, 2026 6:04 AM GMT

Summary

The purpose of this essay is not to present an academic claim, but to convey my thoughts based entirely on my personal readings. To avoid any confusion, I will begin this section by referring to the thought process I will employ in subsequent sections.

 

Introduction

Media is a message. Today, when you pick up a McLuhan book, you will realize that works written in the 1960s remain relevant even today. In addition, you may find that some sentences seem as if they were written directly for today.

When things go far enough, humankind thus becomes a creature of its own machine.

-Excerpt from the book "Media Message, Media is the Message"

To be a creature of one's own machine is a very bold statement for the technologies of that time, because the most common technologies were radio and television. Home computers weren't even mentioned. And in a time when technologies were so limited, McLuhan was able to make incredible statements that encompass even the present and the future.

Media is the message. This is McLuhan's most important and well-known theory. However, when you look at the subheadings used to construct this two-word sentence, almost every subheading is serious enough to be the main subject of a study in itself. The subheading I will address is the one where he talks about media being an extension of the physical and spiritual aspects of humanity.

All media is an extension of certain physical and mental faculties of human beings. The wheel is an extension of the foot; the book is an extension of the eye; clothing is an extension of the skin…

-Excerpt from the book "Media Message, Media is the Message"

Upon reflection on this quote, it becomes clear that everything humanity has conceived and produced today stems from one or more of its own characteristics. This explains why McLuhan's thoughts and work are timeless. The idea that what humanity has produced from yesterday to today is essentially an interpretation of a part of itself makes it easier for us to make more consistent predictions about the future.

In this context, my own thought and conclusion is this: The virtual world we inhabit today is an extension of the brain. And the world we inhabit today is an extension of the virtual world.

The Virtual World is an Extension of the Brain

About 100 years ago, an archivist would meticulously examine tons of documents, sorting and organizing them according to their dates and categories. Even the thought of it was incredibly laborious. Now, let's look at today; we have programs that can handle all this archival work in minutes. These programs were something people could only dream of in their minds years ago, but now almost every technological device can do it.

We can say that computers are extensions of things that are physically difficult to fit into the world. Many files and accounts, which in the past were equivalent to tons of paper, are now equivalent to many punched cards, and today to the size of a hard drive.

Of course, the advent of digitalization would require a digital space; after all, data was no longer on paper, but in another realm. This, naturally, would inevitably lead to people wanting to explore this virtual world and asking themselves, "What else can be done in this realm?"

Personally, my favorite example is William Higinbotham creating the first video game. Higinbotham designed a simple tennis game for bored visitors in his lab. For the first time, people could interact with a world inside a screen. That day was actually a milestone, because it pioneered something new; a non-physical but experiential world. In other words, the virtual world.

The closest example of this is like trying things out on your own with a newly learned software program.

The virtual world has made incredible strides thanks to Higinbotham and many others like him. Every successful or unsuccessful attempt to push the boundaries of technology has contributed to the creation of the virtual world we use today.

If we go back a few years, the virtual world was a Saturday night pastime for many; today, it is progressing towards becoming a place we are in every day, and if we push it a little further, every second.

• Before you even realize it, your hand reflexively reaches for your phone; you find yourself watching videos on platforms like Instagram, TikTok, or YouTube. Social media, then, is an extension of dopamine.
• You have a complex dataset at your disposal, and you build an artificial neural network (ANN) to organize it. It's similar to organizing multiple tasks simultaneously. ANNs are extensions of neurons.
• Take the map applications we use today. Our brain uses the hippocampus to perceive space. Map applications are extensions of the hippocampus.

Hippokampus

Today, the real world is beginning to merge with the virtual world. This is already a predictable assumption for tech enthusiasts. But how will this happen? I will address this topic in the next section.

 

The Real World is an Extension of the Virtual World

Today, advertising campaigns and actions like mobilizing masses are carried out through social media platforms. And the place they all try to influence is your mind. Think about it, smartphones are now indispensable, and as in the example in the previous section, we have started to pick up our phones reflexively. Naturally, the place where advertisements and ideas that try to appeal to the individual can be launched and reach the masses most quickly is not only social media, but also the virtual world.

So how does the extension of the virtual world influence the real world? This influence is more mental and linguistic than material. English is one of the indispensable languages ​​worldwide today. If you want to reach the world, not just an audience in your own country, you can do so through social media and English. You just need your content to be somewhat engaging; the rest follows like a snowball effect.

The awareness created in the virtual world triggers the real world. I want to clarify this with a few examples.

• The Arab Spring: Tweets, hashtags, and other social media posts triggered popular protests, uprisings, and conflicts in Tunisia, Egypt, Libya, Syria, Yemen, and neighboring countries.
• GameStop Short Squeeze: A surge in the stock price of a physical video game retailer from $20 to $400, fueled by calls on Reddit. This gave rise to the term "meme stock" and highlighted the potential for stock market manipulation by social media audiences.
• Star Wars Battlefront II: A Reddit user asked why unlocking characters like Darth Vader was so difficult and expensive. EA responded coldly, "We want to give players a sense of accomplishment and pride," leading to one of the biggest online lynchings. Even after making changes to the game, EA failed to reconnect with its original audience.
• George Floyd: The rapid spread of images related to George Floyd's death on social media transformed a local police incident into a global political movement. Online content has directly triggered street protests, corporate reforms, and public policies. This has led to the formation of a consciousness where the online world is not only representative but can directly influence reality.
• Cambridge Analytica: collected data from millions of people on Facebook, not just basic information like age and gender, but also their specific interests, concerns, and sources of anger, creating a psychometric profile of each user. By directly influencing algorithms, they caused posts that could create a counter-effect to be specifically included in that user's feed, directly impacting the Brexit referendum and the 2016 US elections, becoming one of the biggest and most important examples of data manipulation and disinformation.

All these examples, and many more, have triggered and activated the data and consciousnesses within the virtual world. However, this doesn't yet mean that the real world is an extension of the virtual world. Because the examples above are simply adaptations of media content from throughout history. My theory is that all of this is a stepping stone.

The real world is an extension of the virtual world. This sounds very much like a line from The Matrix, but today, statements from high-tech figures like Zuckerberg, Musk, and Gates are ringing the bells of a shift towards merging consciousness with technology and moving the virtual world towards replacing the real world.

It's not hard to see that Zuckerberg is trying to lead the way in this area with his statements and investments. Consider Metaquest. Virtual entertainment areas, virtual meetings, etc. But why did Zuckerberg's project fail? Because transitioning from the real world to that virtual world was pure torture. The heavy device you put on your head, the dizziness and nausea caused by prolonged use, all contributed to Metaquest's initial failure. However, this wasn't entirely a failure. The process of improving the device, fixing its shortcomings, and re-releasing it using user data seems likely to continue until Metaquest becomes something like Nervgear (a virtual reality device from Sword Art Online).

Works depicting dystopian futures, like Ghost in the Shell and The Matrix, aren't created solely for popularity. Their authors also have a vision and a perspective on the future. The actions of people like Zuckerberg can create a vision of the future and give authors' works, even fictional ones, a degree of realism. What I'm describing might seem very abstract or just science fiction, but only a few years ago we would have dreamed of things like, "Wouldn't it be great if an AI like Carvis existed?" and today we have countless trainable AIs like ChatGPT and Gemini.

Technology is the magic of the modern world. With sufficient reading and reasoning, the direction of this magic can be largely predicted. This text is an attempt to understand that direction.

Discuss

Published on February 11, 2026 5:38 AM GMT

Prerequisite: basic familiarity with what a prediction market is

So you want to run a prediction market. You need a way for people to trade shares. What are your options?

CLOBs and Markets

If you were making a prediction market from scratch, you'd probably come up with a Central Limit Order Book (CLOB). Traders post BUY and SELL orders, stating what they're willing to buy and sell, and at what price, and you record these orders in your book.

• Alice posts: "I'll buy 20 YES shares at $0.60 each"
• Bob posts: "I'll sell 25 YES shares at $0.65 each"
• Carol posts: "I'll buy 10 YES shares at $0.61 each" When someone wants to trade, and doesn't want to wait for someone to fulfill their order, you match them with the best available offer. Pretty intuitive.

This system shows up directly in Hypixel Skyblock and other MMOs. The Bazaar lets you post orders and wait, or instantly fulfill existing orders. Have some Enchanted Iron to sell? You can list it at 540 coins and wait for a buyer, or instantly sell it by fulfilling the highest buy order at 470 coins.

The gap between the highest buy order ("bid") and the lowest sell order ("ask") is called the bid-ask spread.

Enter the Market Maker

CLOBs work well, but they have a problem: they need people actively posting orders from both sides. If nobody's posting orders, the market can't function. The spread can also become very wide when few traders are active, making it expensive to trade.

This is where market makers come in. A market maker continuously posts both buy and sell orders, ensuring there's always someone to trade with.

Market makers profit by maintaining a gap in prices between their bid and asks. For example:

• They buy YES shares at $0.60
• Sell YES shares at $0.65
• Whenever one person buys YES from them and another person sells it to them, they pocket the $0.05 difference

This is called crossing the spread. The market maker provides liquidity to the market and is compensated for it through the spread. In traditional finance, firms like Citadel Securities make billions doing exactly this. In Hypixel Skyblock, this strategy is called "bazaar flipping".

Pricing Shares

How do Market Makers price their shares? In an existing market, they can simply look at existing price charts to determine their prices, but that's not very feasible with prediction markets. Thus, we need some way of determining a fair price for shares.

For simplicity, let's ignore extracting profit. We'll assume someone's paying our market maker, whom we'll call Duncan, a flat fee to provide this service, and they're otherwise operating in a fully efficient market where the bid-ask spread is 0.

Duncan holds some inventory of YES and NO shares, and people can trade with him. How should Duncan price his shares? Examining the question, we can see some key constraints:

1. Prices should sum to $1: If a YES share pays out $1 when the market resolves YES, and a NO share pays out $1 when the market resolves NO, then the price of a YES share and a NO share together should be exactly $1, since the market can only resolve to one of these two options.
2. Price equals probability: If YES shares cost $0.70, then that means the market thinks there's a 70% chance of the outcome being YES, since that's how expected value works. This is the key mechanism by which prediction markets work and even if you don't know the details of the market implementation yet, you should know this already.

Creating New Shares

Duncan needs the ability to issue shares. Otherwise, he'll run out of them, and won't be able to trade anymore. (No, he can't just raise share prices in an inverse relationship with his supply, since he sells both YES and NO shares this would violate the constraint that prices must sum to $1.)

Fortunately, it's very easy to issue new shares. Since YES and NO sum to 1, for every dollar Duncan receives from a trader, he can mint one YES share and one NO share as a pair. When the market resolves, he'll pay out $1 to holders of the winning share type, fully covering his obligation.

From this, we can infer that any valid formula must have certain properties: buying YES must raise P(YES), the probability must depend on inventory ratios (when Duncan holds a lot of NO, the probability is high because it means he's sold a lot of YES), and YES shares should always cost less than $1, except when the market is at 100%, and vice versa. Since 0 and 1 aren't probabilities, this should never happen.

A Natural Probability Formula

Given these constraints, you might come up with this formula for deriving the probability from Duncan's inventory (and thus the prices of YES and NO):

P(YES)=ny+n.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

where y is Duncan's YES inventory and n is Duncan's NO inventory.

• When y=n (such as when the market is initialized), the probability is 50%
• If Duncan fully runs out of YES shares, the probability is 1, meaning you can't profit from buying YES anymore and you can buy NO for free.
• If Duncan fully runs out of NO shares, the probability is 0.

This formula seems to satisfy all of our desiderata, and is fairly intuitive. Since P(YES) is the price of yes, we now know how to price our shares.

Discrete Shares

If Duncan has 50 YES and 50 NO shares, probability is 50%, so shares cost $0.50 each.

You give Duncan $1, and tell him you want to buy YES.

1. YES costs $0.50, so $1 buys 2 YES shares
2. He mints 1 YES + 1 NO (inventory: 51 YES, 51 NO)
3. Duncan gives you 2 YES shares in exchange (inventory: 49 YES, 51 NO)
4. New probability: 51/(49+51) = 51%

Another example. Duncan has 100 YES and 50 NO:

• Probability: 50/150 = 33.33%
• Price per YES: $0.33
• Your $1 buys 3 YES shares
• He mints $1 of shares (inventory: 101 YES, 51 NO)
• He gives you back 3 YES: (inventory: 98 YES, 51 NO)
• New probability: 51/149 = 34.23%

You might have noticed the problem already: Duncan isn't accounting for how the purchase itself affects the price.

When you buy multiple shares at once, you're getting them all at the initial price, but each share you buy should be more expensive than the last! You get a discount on bulk purchases!

Duncan could solve this by selling shares one at a time or even fractions of a share at a time, adjusting the price after each infinitesimal sale. But this is computationally expensive and assumes shares are discrete units rather than infinitely divisible.

For a continuous equation, we need to use calculus and solve a differential equation

The Calculus of Market Making

(warning: differential equations)

Let's formalize this problem. Suppose Duncan starts with ys YES shares and ns NO shares. You deposit m dollars. and buy YES from Duncan.

After the trade:

• Duncan has minted m new shares of each type
• NO inventory: n=ns+m
• YES inventory: y=ys+m−sold

where "sold" is the quantity of YES shares Duncan gives to the trader. (In this context, s stands for "starting".)

The market probability at any point is:

P=ny+n

Substituting our inventory formulas:

P=ns+m(ys+m−sold)+(ns+m)=ns+mys+2m+ns−sold

Since we're obeying the constraint price equals probability, the rate at which Duncan sells you shares is determined by the current probability.

The trader deposits money at rate dm and receives shares at rate d(sold). The price per marginal share is dmd(sold). Since we want the price to be the probability, we get: 

dmd(sold)=ns+mys+2m+ns−sold

Since we're taking money as our input, we take the reciprocal:

d(sold)dm=ys+2m+ns−soldns+m

 This is our initial differentiation equation. I encourage you to try to solve it on your own,  but if you don't know calculus or get stuck, the solution is enclosed below.

d(sold)dm=ys+2m+nsns+m−soldns+m

 

soldns+m+d(sold)dm=ys+2m+nsns+m

 Multiply both sides by ns+m: 

sold+d(sold)dm⋅(ns+m)=ys+2m+ns

 

∫(sold+d(sold)dm⋅(ns+m))⋅dm=∫(ys+2m+ns)⋅dm

 Observe that the ddmsold=d(sold)dm and ddm(ns+m)=1. By product rule, then: 

sold⋅(ns+m)=mys+m2+mns+C

 

sold=m(ys+m+ns)+Cns+m

 

sold=m+mys+Cns+m

 sold(0)=0, since if you spend no money you don't get any shares. If you plug in m=0, sold=0 and solve for C, you get C=0, so we can just drop that term. 

sold=m+mysns+m

 Since n is just ns+m and y is ys+m−sold, we get: 

y=ys+m−m−mysns+m

 !<

y=ys−mysns+m

 You might notice the term ns+m shows up in the denominator of a term of y, and is equivalent to n. If you multiply y and n together, you get: 

yn=ys(ns+m)−mys=ysns+mys−mys

 

yn=ysns

 The product of Duncan's YES and NO shares remains constant, regardless of the trade![1]

Constant Product Market Maker

Thus, we've discovered the fundamental invariant: 

yn=k

 where k is a constant determined by Duncan's initial inventory. Because YES * NO is always constant, we call this a Constant Product Market Maker (CPMM).

So Duncan, knowing this, has determined an algorithm for pricing shares:

1. Receive money from trader
2. Mint YES and NO shares
3. Give out exactly enough YES shares (or NO shares, depending on what the trader wants) to maintain the constant product y⋅n=k

Here's an example of this in practice:

• Duncan starts out by initializing a market with $50 of liquidity. (Initial inventory: 50 YES, 50 NO)
• He solves for his constant product, which needs to remain invariant. k=50⋅50=2500
• You bet $50 on YES. Duncan uses this to mint more shares. (Inventory: 100 YES, 100 NO)
• He now needs to pay out enough YES shares that he reaches his constant product again. yn=k, solve for y.
• y=kn
• Plug in NO and k. y=2500100=25
• He has 100 YES, and needs to have 25 YES, so he gives you 75 YES shares in exchange for your $50. (Inventory: 25 YES, 100 NO)
• The new probability is ny+n=10025+100=80%.

Meanwhile, if a trader wants to sell shares, it's similarly simple: He adds the shares to his inventory, figures out how many YES + NO pairs he needs to give up in order to reach the constant product, and then exchanges these pairs for cash and gives them to the trader, removing the shares from circulation. Alternatively, and perhaps more elegantly, the trader can simply buy the opposite share and then give pairs to Duncan in exchange for cash.

(Note that, since Duncan's inventory is inversely related to the market probability, that means Duncan pockets a lot of money from traders when the market resolves counter to expectations, and loses more of his initial liquidity the more confident a correct market is.)

In fact, this process can be fully automated, creating an Automated Market Maker (AMM). This is the foundation of Uniswap, and many prediction market protocols.

Conclusion

Starting from basic constraints about prediction markets (prices sum to 1, price equals probability), we derived a unique solution. We didn't just arbitrarily choose the CPMM out of a list of options. It emerged, inexorably, from the requirements we placed.

When you properly formalize a problem with the right constraints, there's often exactly one correct answer. Independent researchers, solving similar problems with similar constraints, will converge on the same solution. When Newton and Leibniz invented calculus, they didn't get similar results because they shared their work, or because they were working on the same problem (they were working in very different fields). They got similar results because they were working on a class of problems with the same underlying structure, even if the similarities are not obvious at first.

The market itself does Bayesian updating—on expectation, as more people trade, the probability approaches the true likelihood, based on the accumulated knowledge of the traders. Our pricing mechanism had to respect this Bayesian structure. The constant product formula isn't arbitrary; it's what you get when you correctly formalize "each marginal share should cost the current probability" in continuous terms. While this isn't an empirical fact about the territory, the laws of probability nevertheless have carved out a unique shape in design space, and your map had better match it.[2]

(This is especially obvious in the context of a prediction market (which is, in a certain sense, the purest form of market, separating the trading and aggregating of information from everything else), but it applies to markets and AMMs in full generality, being used in DeFi and Crypto space.)

 

 

1. ^

   If you don't know calculus, this is the important part.

2. ^

   Ok, I'm completely overstating my case here and these paragraphs are largely joking, there are other solutions to this problem if you pick different probability functions matching these desiderata or come at prediction market design from a different cases, many of which have their own pros and cons, and Hanson explicitly wrote about Constant Function Market Makers. It's just that this one is very intuitive and has useful properties for a purely probabilistic YES/NO market which is why I wrote about it

Discuss

Published on February 11, 2026 4:49 AM GMT

punctilio (n.): precise observance of formalities.

Pretty good at making your text pretty. The most feature-complete and reliable English micro-typography package—transforms plain ASCII punctuation into typographically correct Unicode, even across HTML element boundaries.

Smart quotes · Em/en dashes · Ellipses · Math symbols · Legal symbols · Arrows · Primes · Fractions · Superscripts · Ligatures · Non-breaking spaces · HTML-aware · Bri’ish localisation support

import { transform } from 'punctilio' transform('"It\'s a beautiful thing, the destruction of words..." -- 1984') // → “It’s a beautiful thing, the destruction of words…”—1984npm install punctilioWhy punctilio?

As far as I can tell, punctilio is the most reliable and feature-complete. I built punctilio for my website. I wrote[1]and sharpened the core regexes sporadically over several months, exhaustively testing edge cases. Eventually, I decided to spin off the functionality into its own package.

I tested punctilio 1.7.13 against smartypants 0.2.2, tipograph 0.7.4, smartquotes 2.3.2, typograf 7.6.0, and retext-smartypants 6.2.0.[2] These other packages have spotty feature coverage and inconsistent impact on text. For example, smartypants mishandles quotes after em dashes (though quite hard to see in GitHub’s font) and lacks multiplication sign support.

Inputsmartypantspunctilio5x55x5 (✗)5×5 (✓)

My benchmark.mjs measures how well libraries handle a wide range of scenarios. The benchmark normalizes stylistic differences (e.g. non-breaking vs regular space, British vs American dash spacing) for fair comparison.

PackagePassed (of 159)punctilio154 (97%)tipograph92 (58%)typograf74 (47%)smartquotes72 (45%)smartypants68 (43%)retext-smartypants65 (41%)FeatureExamplepunctiliosmartypantstipographsmartquotestypografSmart quotes"hello" → “hello”✓✓✓✓✓Leading apostrophe'Twas → ’Twas✓✗✗◐✗Em dash-- → —✓✓✗✗✓En dash (ranges)1-5 → 1–5✓✗✓✗✗Minus sign-5 → −5✓✗✓✗✗Ellipsis... → …✓✓✓✗✓Multiplication5x5 → 5×5✓✗✗✗◐Math symbols!= → ≠✓✗◐✗◐Legal symbols(c) 2004 → © 2004✓✗◐✗✓Arrows-> → →✓✗◐✗◐Prime marks5'10" → 5′10″✓✗✓✓✗Degrees20 C → 20 °C✓✗✗✗✓Fractions1/2 → ½✓✗✗✗✓Superscripts2nd → 2ⁿᵈ✓✗✗✗✗English localizationAmerican / British✓✗✗✗✗Ligatures?? → ⁇✓✗✓✗✗Non-English quotes„Hallo”✗✗✓✗◐Non-breaking spacesChapter 1✓✗✗✗✓Known limitations of punctilioPatternBehaviorNotes'99 but 5' clearance5' not converted to 5′Leading apostrophe is indistinguishable from an opening quote without semantic understanding«Bonjour»Not spaced to « Bonjour »French localization not supportedTest suite

Setting aside the benchmark, punctilio’s test suite includes 1,100+ tests at 100% branch coverage, including edge cases derived from competitor libraries (smartquotes, retext-smartypants, typograf), and the Standard Ebooks typography manual. Further, all transformations are stable when applied multiple times.

Works with HTML DOMs via separation boundaries

Perhaps the most innovative feature of the library is that it properly handles DOMs! (This means it'll also work on Markdown: convert to HTML, transform with punctilio, convert back to Markdown.)

Other typography libraries either transform plain strings or operate on AST nodes individually (retext-smartypants can’t map changes back to HTML). But real HTML has text spanning multiple elements. If you concatenate text from <em>Wait</em>..., transform it, then try to split it back, you’ve lost track of where </em> belonged.

punctilio introduces separation boundaries. First, insert a “separator” character (default: U+E000) at each element boundary before transforming (like at the start and end of an <em>). Every regex allows this character mid-pattern without breaking matches. For example, “.[SEP]..” still becomes “…[SEP]”. 

import { transform, DEFAULT_SEPARATOR } from 'punctilio' const sep = DEFAULT_SEPARATOR transform(`"Wait${sep}"`) // → `“Wait”${sep}` // The separator doesn’t block the information that this should be an end-quote!

punctilio validates the output by ensuring the separator count remains the same. 

For rehype / unified pipelines, use the built-in plugin which handles the separator logic automatically:

import rehypePunctilio from 'punctilio/rehype' unified() .use(rehypeParse) .use(rehypePunctilio) .use(rehypeStringify) .process('<p><em>"Wait</em>..." -- she said</p>') // → <p><em>"Wait</em>…"—she said</p> // The opening quote inside <em> and the closing quote outside it // are both resolved correctly across the element boundary.

Options

punctilio doesn't enable all transformations by default. Fractions and degrees tend to match too aggressively (perfectly applying the degree transformation requires semantic meaning). Superscript letters and punctuation ligatures have spotty font support. Furthermore, ligatures = true can change the meaning of text by collapsing question and exclamation marks. Non-breaking spaces are also opt-in since they alter whitespace throughout the text.

transform(text, { punctuationStyle: 'american' | 'british' | 'none', // default: 'american' dashStyle: 'american' | 'british' | 'none', // default: 'american' symbols: true, // math, legal, arrows, primes collapseSpaces: true, // normalize whitespace fractions: false, // 1/2 → ½ degrees: false, // 20 C → 20 °C superscript: false, // 1st → 1ˢᵗ ligatures: false, // ??? → ⁇, ?! → ⁈, !? → ⁉, !!! → ! nbsp: false, // non-breaking spaces (after honorifics, between numbers and units, etc.) checkIdempotency: true, // verify transform(transform(x)) === transform(x) })

• Fully general prime mark conversion (e.g. 5'10" → 5′10″) requires semantic understanding to distinguish from closing quotes (e.g. "Term 1" should produce closing quotes). punctilio counts quotes to heuristically guess whether the matched number at the end of a quote (if not, it requires a prime mark). Other libraries like tipograph 0.7.4 use simpler patterns that make more mistakes.
• The american style follows the Chicago Manual of Style:
  • Periods and commas go inside quotation marks (“Hello,” she said.)
  • Unspaced em-dashes between words (word—word)
• The british style follows Oxford style:
  • Periods and commas go outside quotation marks (“Hello”, she said.)
  • Spaced en-dashes between words (word – word)
• punctilio is idempotent by design: transform(transform(text)) always equals transform(text). If performance is critical, set checkIdempotency: false to skip the verification pass.

 

Feedback welcome

If you run into bugs, please create an issue! You can read about my other open-source projects on my website.

1. While Claude is the number one contributor to this repository, that’s because Claude helped me port my existing code and added some features. The core regular expressions (e.g. dashes, quotes, multiplication signs) are human-written and were quite delicate. Those numerous commits don’t show in this repo’s history. 

2. The Python libraries I found were closely related to the JavaScript packages. I tested them and found similar scores, so I don’t include separate Python results. ↩︎

Discuss

Published on February 11, 2026 12:15 AM GMT

Save the date: LessOnline will be back again in 2026! As usual, it will take place at Lighthaven in Berkeley, CA.

Further details will be posted to LessWrong, or subscribe here for further updates, such as ticket sales going live. Details about who's attending, housing, relation to Manifest will be forthcoming.

What is LessOnline?

A Festival of Writers Who are Wrong on the Internet

LessOnline is a festival celebrating truth-seeking, optimization, and blogging. It's an opportunity to meet people you've only ever known by their LessWrong username or Substack handle... or hang out with them irl again if you met them at the first or second LessOnline!

 

Lighthaven Campus

Discuss