Сборщик RSS-лент

Epistemic status: I am not a subject-matter expert in any of the fields in this article. The following is speculation, albeit I think reasonable speculation. Cunningham’s Law is why I wrote this essay.

AI Disclosure: Claude Opus 5.5 provided feedback to the first and second draft of this essay, similar in kind to a friend providing feedback on a draft. All text and errors therein are mine.

What does discovering new knowledge mean?

The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents. We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age. —H. P. Lovecraft, “The Call of Cthulhu”

A long time ago (meaning, August of 2023), Dwarkesh Patel asked Dario Amodei a fantastic question:

…[W]hat do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven't been able to make a single new connection that has led to a discovery?

At the time, the frontier was GPT-4. Like Dwarkesh, I was puzzled by this. When o1 was released in September of 2024, I thought that the answer to this was pretty straightforward: capability will go up, connections will be elicited from the models by RL, interesting results will follow. I think that hypothesis has borne fruit, as I’ll trace out with mathematics as the example field in a moment. The next interesting question, which seems to be severely underexplored, is “how much”. In other words, now that we’re leaving that “placid island of ignorance”, how much of an overhang is going to be absorbed by “correlating the contents” of human knowledge?

I think it’s helpful here to distinguish between two different types of new knowledge. In October of 2026, Matthew Schwartz published an article called “Claude-Shaped Science”, describing multidisciplinary work he had done with Claude.[1]He had a useful diagram of this idea of different types of knowledge:

Schwartz here is modeling human knowledge as a jagged shape (in blue) with each point of the shape (the spokes) being the frontier of the field. Total human knowledge can expand in two main ways: expand a spoke or fill in the area between spokes. For the purposes of this essay, “spoke problems” require a new concept, idea, or data that doesn’t already exist in the corpus of human knowledge. “Gap-filling problems” are problems where everything needed to solve them already existed in that corpus, but nobody had mixed the ingredients together.  The question I’m interested in is how large and valuable is the total area between the spokes? Pretending that LLMs never drive any of the spokes forward [2], how much progress or disruption can we expect to see from “merely” filling in those gaps? This seems important because this provides a hard lower bound on the impact of AI.

Math as the vanguard of the fleet

The first field to experience large-scale gap-filling has been mathematics. As a brief timeline, with all of the benchmark numbers from Epoch:

2024: Pre-reasoning, GPT-4o (August) was at the frontier for math. GSM8K (grade-school math) was close to saturated; GPT-4o scored 53% on MATH L5 (high school competition math). The first reasoning models blew past MATH L5 (o1 scored 94%) and made major strides on Mock AIME (o1 at 73%). Terence Tao described working with o1 as being “roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student.” No discoveries yet.

2025: In July, Google and OpenAI both announce IMO Gold. FrontierMath T1-T3 (v2 for consistency) are the main benchmark at this point, with the highest score at the end of the year being 72% by GPT-5.2 Pro. These tiers are research-adjacent problems that would take a researcher in the field hours to a few days. According to VibeMathed, the first AI-assisted open problem solution is solved on July 20. A few others happen by the end of the year. The first claimed AI-discovered solution (a disproof of Erdős #333) is published on December 25. It’s actually a rediscovery of a solution Erdős had already published in 1977.

2026: Things accelerate dramatically. The actual first AI-discovered solution (for Erdős #728) happens in January 2026. FrontierMath T1-3 and T4 (a harder tier, problems that would take a couple of weeks) are saturated by October of 2026. More interesting open problems fall: the Jacobian Conjecture on July 20, existence of non-sofic groups on August 1, and (arguably, the details are contested) Navier-Stokes on September 9. This leads to the October 6 release by OpenAI of hundreds of mathematical results. At least one mathematician (Alex Kontorovich) described the most significant solution (proof of the quasi-Riemann hypothesis) as being significant enough that “If a human did this, it would be an instant Fields Medal, no questions asked”.

I am not a mathematician. But from reading the commentary of mathematicians like Daniel Litt or Terence Tao, it appears to be the case that few, if any, of the results listed above are “moving the spokes” solutions.[3]The consistent theme is that LLMs are strongly superhuman at combining existing ideas in potentially novel ways rather than developing new theory themselves. This seems to me like convex hull gap filling. So, the answer to “how big is the convex hull overhang” for math seems to be pretty big! I’d be very surprised if the gap-filling in mathematics has stopped here.

What about other fields?

In short, I don’t know. My mental model of how the convex hull areas grow is that researchers specialize in narrow silos because moving the spokes is hard.[4] Any of the following things or a combination thereof could happen:

• Transfer problems: The researcher runs into a problem that has been solved analogously in another sub-field, another field, or another subject entirely. A classic example is Don Swanson suggesting fish oil as a treatment for Raynaud Syndrome.[5] It appears the phylogenetics work Schwartz did is also an example of this category.
• Search problems: The researcher can’t search through all of the information in another sub-field or field to solve their problem. The blogger Prinz’s decryption of RICHI-240 seems like this type of problem (the outside information here being contemporaneous telegrams that confirmed troop movements, and therefore what army divisions the telegram is talking about).
• Bookkeeping problems: The researcher can’t trawl through, organize, or keep track of all the existing data in their area of study. An example of this type of issue is keeping track of all the biographical information of civil servants in two Victorian reference works. In this case it was solved through using Wikidata, not AI, but this is the shape of problem I’m referring to here.
• There are probably several other relevant paths to these overhangs being formed. People at the spokes have a much better understanding of how the convex hulls are formed in their field than I do.

My intuition is that the areas here are very large and that filling in the gaps is going to be a massive wave of progress on a lot of problems. I don’t think it’s “full sci-fi, Dyson Sphere in ten years” big, but my rough guess is “Internet/Printing Press” on the technological Richter scale on the low end.

For field-specific predictions, I weakly expect that social sciences like history are going to see an armada of stunning convex hull type solutions. My expectation is similar for the sciences like biology, but at least one friend who is an active virology/immunology researcher is skeptical of the overhang being that pronounced. His argument is that hypothesis generation (at least in his field) is not a significant bottleneck now, and any hypothesis has to be verified with time-consuming wet-lab work. This seems plausible enough for his field, but things like drug repurposing or the above fish oil-Raynaud example make me think there’s at least some fertile ground. I don’t know the answer to how much fertile ground there is, which is why I’m writing this essay to begin with.

As for what the best way to harvest that fruit, I think Schwartz’s guidelines are reasonable. I also think that researchers should be collecting, clearly demarcating, and writing down problems as much as possible. I suspect that one of the reasons math has seen such a glut of convex hull problems first (beyond obvious things like math being an easy to verify domain) is that a big chunk of “doing math research” is based on moving the ball forward on written-down problems. If I’m right about this point, fields that can’t imitate that model are plausibly going to have smaller overhangs. I also think breadth of knowledge is valuable for identifying potential convex hull areas.[6]

Why does this matter?

I think this question is important for a few reasons.

• Regardless of whether AI eventually makes the centaur model (human + AI teams collaborating) obsolete, there’s the question of what to work on right now. If my suspicion is accurate, a huge mass of hitherto infeasible work is now available. Suddenly there’s a way to harvest low-hanging fruit that’s accumulated throughout history.
• The size of the convex hull overhang determines part of the lower bound of disruption due to AI. Even if somehow LLMs totally plateau (or are stopped by pacing/a pause/a shutdown), they never improve on research taste, and they never become able to push the spokes forward, diffusion of existing LLM capabilities will lead to incredible change.
• As a consequence of the lower bound point, this matters for meaningful dialogue with people who are resolutely not AGI/ASI-pilled. If the overhang is large, then no “sci-fi” stories of AGI or ASI are necessary for disruption far larger than seems to be accepted by the AGI/ASI skeptics or “normal people” not plugged into the discourse. At the very least this seems like a crux point for my conversations with people like my aforementioned biology researcher friend.

Finally, I think it’s entirely plausible that we blow past the centaur model very quickly, RSI happens, crazy sci-fi future or doom, fill in the details from the AI Futures Project or other ASI scenarios as you see fit here. I’m confident that the future is going to be weird and knowledge is going to expand rapidly, but my error bars are very large on what that looks like. If nothing else, I’m going to be very curious how much of that knowledge expansion will be moving spokes or filling in the gaps.

1. ^

   I’m not qualified to judge whether this work is meaningful. At least some of the domain experts he worked with seemed impressed, but my thesis doesn’t depend on the quality of his specific output.

2. ^

   This seems to be a significant assumption of the AI skeptic argument and is probably already falsified, but we’ll grant for this essay.

3. ^

   For the new OpenAI results, I don't know if anybody knows yet the breakdown of which category applies to the solutions in the repo.

4. ^

   Opus 5.5 filled in a gap for me here; apparently this is a well-known problem called “the burden of knowledge”. I knew the idea, I just didn't know it had a name.

5. ^

   Another Opus 5.5 gap fill for me writing this essay.

6. ^

   This is my main cope for why my liberal arts degree still has any value at all.

Discuss

That moment is over now. Such euphoria and now I'm back at the house and it's all but a memory. Already gone in an instant. The only way I can immortalize it is by writing these words. So thus I write in a feeble hope to cheat time.

-- Diary excerpt from October 27, 2025

I asked myself: what is my most valuable possession?

Easy. Probably my MacBook, and then my Balenciaga flip flops, and then my future sexy red Ferrari once I get rich enough to buy it. Oh, I can already see its holy shine and feel the cold metal beneath my hands --

I'm joking. The real thing, weirdly enough, is my diary.

I never would have expected that when I started writing it. When I was a kid, I always thought a diary was the thing the weird kids and also your grandma wrote in. And that you start it with "dear diary", and draw little hearts in it, and write about your crush and yearn over them for endless pages. And while I still write endless pages yearning over my crushes, it's become something much bigger than that.

It's turned into my own little time machine, and given me several benefits I didn't expect.

My journey to insanity

My diary these days is insanely long. It's over 750,000 words (3,000 pages). But believe it or not, there was a time when my diary was 0 words.

I started writing my diary as writing practice. I was writing novels, and wanted to see if I could take "what happened today" and make it fun to read. I wrote these for a week, then forgot them.

Months later, I found them and was shocked by how interesting it was to read back. I had forgotten many of the moments, and the writing transported me back to them.

I had written those entries during an important part of my life -- when I was talking to my first girlfriend, before we dated, including the time I asked her out. This is the actual entry I wrote that day:

A small silence overcomes us, and my heart beats faster in my ears. I steel my mind. "I mean, what we are is kinda ... ambiguous. So I wanna make it official." I turn fully toward her. "[Name], I really like you. You wanna be my girlfriend?"

She hesitates. "No."

I wait for a second, and then my brows knit. "Wait, really?"

"Ok. Yes, I do want to be your girlfriend. But, like, I want a cute ask. This doesn't count."

"Oh, well, I just didn't want to keep procrastinating." My voice comes out as whiny, which I didn't intend.

She has a massive smile on her lips, I do too. "So, I'll be your girlfriend. But you just have to give me a cute ask. You don't have to spend a lot, or anything. You just have to plan this one."

"I did plan this one, though." I laugh.

-- Diary excerpt from November 19, 2023

I thought to myself how, if I hadn't written those entries, I would have forgotten this formative part of my life.

Stephen King talks about how writing is time travel. I discovered that firsthand then. I could read what I had written and be back in that moment.

That inspired me to start writing more regularly. At first, I only wrote when I was in my emo feels. But over time, I started writing in it every day. And now I sit at the insane point of having a full book-series-length of diary entries.

Already gone

"We write to taste life twice, in the moment and in retrospect."

-- Anais Nin (famous author, and maybe an even bigger diary-er than me)

The main reason I started taking my diary seriously was because a life forgotten scared me.

When I look at my childhood, 99% of the important memories are gone. Even recent memories fade quickly. Whenever I read back a random entry, I'm shocked by how much I forgot.

If I don't write it down, then eventually I'll forget. And if I forget, then 10 years from now, isn't that the same as it not even happening in the first place?

The whole "time" idea is the most emo-ifying thing I can think of. Every moment feels like it's over before it begins. Every moment I ever want to happen, everything I am excited for, hoping for -- I know that when I get it, it will already be over.

I write so that my life does not disappear. To give some semblance of present to my past. To prevent it from dying and fading away into nothing. It is my little taped together cardboard time machine.

I just don't have the words or the time or the energy to describe how amazing this night was. So it'll be up to your fading memory and these insufficient words to make up the lack of detail.

-- Diary excerpt from January 22, 2026

Stupid T-Rex pegs

One thing I didn't expect: my diary significantly improved my writing style. It turns out writing 750,000 words does actually improve your skills.

The biggest change is that my writing became genuinely creative.

I don't filter anything in my diary, and I almost never edit it. If I write something stupid, that's fine, I just keep writing. It's an unfiltered access into my mind in that moment. Flaws and all.

There's no expectation for it to be good, and somehow that usually makes it pretty good. Sometimes, I'll read back and find myself impressed at the casually poetic or deep or funny things that I wrote.

At some point, I got into the habit of accepting and nurturing every stupid thought that I had. The crazy ones. The weird Tyrannosaurus-rex-shaped pegs trying to fit into square holes. After a while of this, my mind just felt more creative.

I even started doing stand-up comedy and writing a comedy book. In fact, the internal voice of the comedy book is extremely similar to the prose voice I write with in my diary (except somehow even lower IQ).

It's only about time until AI's intelligence catches up to Albert Einstein. And then a few years after that it will catch up to me.

-- Diary excerpt from July 24, 2026

Know thyself

Sometimes, I look at myself, at my hands, at my face, and I feel confusion and ask, "Who am I?"

It's weird. I've lived my entire life in my body. Yet, there are moments when I feel like I don't know myself.

Who am I?

There's a saying of how you have three faces: one you show the world, one you show your close ones, and one you show only yourself.

I would take it even a layer deeper.

There's also a face you don't even show yourself. All the stuff you never stop long enough to think about. We spend most of our lives on autopilot, hooked up to dopamine-dispensing machines. It's hard to self-examine. But writing a diary has given me better clarity on all this.

The face I show the world talks out loud. It's mature, reasonably smart, and occasionally impressive.

The face I show myself is the voice in my head, talking constantly and making dumb, immature jokes about farting at the most inconvenient times. Writing is the best way I can access this exact internal voice in my head. As a result, my diary is filled with an unholy amount of fart jokes.

This is the moment I wanted, right?! ... The moment I'm free. But ... It just feels semi-normal. It's like this sense is building very subtly in me. Like the early hints of a fart deep in your stomach. You can feel it but you don't know how big the explosion is going to be.

-- Diary excerpt from September 26, 2026

Sometimes, I will glimpse the hidden face. I will write about things that I've been avoiding, or that I never took the time to write about before. And I look this hidden face in the eyes -- the part of me that has failed more than I would like to admit, is more self-absorbed than I consciously think -- and accept it.

Every time I look at that hidden face and accept it, it shrinks. Maybe one day I will understand and accept all those hidden parts of myself. Maybe one day, I can say with confidence that I know myself.

More than wanting a girlfriend, I want someone to want me to be their boyfriend.

It's never about them, it's always about me. This is what I'm finding out the more I live.

-- Diary excerpt from January 8, 2026

How to train your dragon write your diary

The generic-but-valid advice first:

• Write a diary entry every day, even if it's only one sentence.
• Write everything stupid that comes to your mind. Don't filter.
• Store it somewhere, probably on your computer. Though if you're a paper-head, then do as you wish.

And now, some not-as-generic advice:

Transport me, baby

Certain entries seem to transport me into that memory more vividly than others. Usually, it's the ones that are actually trying to. The ones that describe with visuals or other senses. Even just one line of vivid description can help. They make your time machine better.

I even had a brief stint where I started writing my diary in third person. For some reason, my third person writing is more vivid like a novel, whereas first person is funnier and often deeper. Both have their merits.

Take this for example:

He heard the patter of rain outside. And when he looked out, there were puddles rippling with rain. Welp, I'll go get some groceries then. He got his raincoat on, grabbed a bag, and headed out.

It was dark, and cars were splattered with rain. As he walked down the street, his heart was filled with genuine glee. So beautiful and alien and strange, but so joyful. He started running.

-- Diary excerpt from September 30, 2025

It's no Hemingway, but it paints a picture in my mind. When I write in first person, I probably would have just done "it rained today". Boringggg.

Cringe-maximizer

I subconsciously avoid writing about certain things. Embarrassing things. But, honestly, those are the best to write about. They are where you can come to know yourself. Whenever you sense yourself avoiding something, lean into it.

The sheer amount of embarrassing and cringe things I've done in my life is, well, a lot. Whenever you write about it, you cut away at its hold on you.

It's made me more comfortable with all that. Less embarrassed by anything. Just writing something down helps you accept it. Realize that, well, you're a human. And while we strive to be more than that, every cell in our bodies came from some monkey-minded fool caveman obsessed with hitting rocks against things.

There is a popular idea that AI could take over the world by being a "paperclip maximizer" -- turning the universe into paperclips. That is me. Except instead of paperclips, I maximize the amount of cringe I feel when I reread my diary.

My most brain-dead Hinge swipe so far:

A girl in front of the Eiffel Tower.

My response?

"wowwww konnichiwa"

Genius.

-- Diary excerpt from September 5, 2026

Goodbye, sweet time

I am writing this sentence on October 7, 2026, 10:50 AM. Now it is done, and that moment is gone. You are reading this sentence. Now you are done, and that moment is over. You will never get it back.

We are a point in time, and we move only forward. We are infinitely thin, a point, we only experience one moment at a time.

Maybe one day, we will be more than this. But until then, I have this dumb, beautiful, unhinged, poetic, feeble, powerful diary. It doesn't let me cheat time. The moments are gone forever.

But it's the closest thing I've found.

I want to sprint slow circles around time.

-- Diary excerpt from August 12, 2026

Discuss

TL;DR

• The autoreconstructor (AR) in a natural language autoencoder (NLA) is the entire argument for the validity of the method, and at present is poorly understood.
• The AR does not condition strongly on English grammar or semantics. Three bits of evidence:
  1. Word pairs that are syntactically related interact no more than matched pairs that are not syntactically related. See dependency arcs.
  2. The fraction of variance explained (FVE) lost by removing or flipping a negation word ("not" -> ""; "without" -> "with") is only weakly correlated with the FVE lost by ablating the negated word (Spearman rho 0.38). See negation.
  3. The word classes that carry the most meaning, nouns (forest), adjectives (heavy) and verbs (reported), cost the least FVE to delete; auxiliaries (is), pronouns (it), numerals and subordinators cost the most. See word class.
• Results are consistent with prior work suggesting that the AR is more heavily influenced by the format of the verbalisation (tone, register, general domain, structure of text) than by semantic meaning per se.

The problem with NLAs

NLAs have strong empirical performance on an automated auditing task. A model given access to a NLA was able to uncover the root motivation of a reward-sycophancy model organism more consistently than without. The NLA provided marginal benefit to an auditing agent with access to the full pre-training, supervised fine-tuning and reinforcement learning corpora, as compared to a sparse autoencoder which offered no benefit. This happened because the activation verbaliser (AV) included mentions of reward-sycophancy in verbalisations of activations from the base model (call the activation under investigation the 'base activation').

The mechanistic case for the method is not so strong. In an attempt to tether the verbalisation to the base activation, the AV is trained on cosine similarity between the reconstructed activation and the base activation. The hope is that the NLA would encode the semantic meaning of the activation in the verbalisation in order to reconstruct it with high fidelity, and that it would be encoded in coherent English. That is the entire mechanistic argument for the validity of the method.

That hope is a testable hypothesis: that the AR reads the verbalisation as semantic meaning encoded in content words and English syntax.

Setup

The base model used for the experiments was Qwen3.6-27B, and the NLA was Celeste Selder's Qwen3.6-27B NLA. The AV is an adapter checkpoint from the 600th reinforcement learning step. The AR is from the same step, predicting the layer 42 activation.

The dataset was a corpus of 5000 FineFineWeb documents, which I filtered based on length and category (see Appendix A for details). I randomly selected a token position after the 128th token to run the base Qwen model on and extract a layer 42 activation to verbalise.

FVE points are the primary unit of this writeup and figures, defined as 100 x the drop in FVE of the reconstructed activation from an edited verbalisation against the unedited baseline.

Findings

The AR isn't reading English how a person would. I found several pieces of evidence showing that the AR does not condition heavily on the syntax of English prose or on content words. Instead, I favour the hypothesis that the format of the verbalisation is what matters. That includes the tone, register, general domain and structure of the text (see Reconstruction Without Recall).

Negation words

I intervened on five types of negation words: "not", "n't" (at the end of a word), "without", "no" and "never". There were six types of intervention:

intervention

example

n

flip the negation

"don't budge" → "do budge"; "without public disclosure" → "with public disclosure"

309

delete the negation ("no", "without", "never" only)

"for no audience" → "for audience"

61

insert "not" after an auxiliary

"she did receive approval" → "she did not receive approval"

200

insert "just" after the same auxiliary (control)

"she did receive approval" → "she did just receive approval"

200

delete the governed word

"don't budge" → "don't"

309

swap the governed word

"don't budge" → "don't fostered"

308

I also measured the effect on FVE of ablating the negated word either by deleting the word or by using the corpus-swap ablation strategy discussed in Appendix B.

Consider this example: "...they were not walking in the forest...", where 'not' is the negation and 'walking' is the governed word. Flipping the negation costs about as much FVE on average as ablating the governed word (the paired difference is −0.09 FVE points, 95% interval −0.36 to +0.18). Within an instance, however, the cost of flipping the negation is only weakly correlated with the cost of ablating it (Spearman rho 0.38 over 308 negators), and neither edit is reliably costlier. The flip costs more in 46% of instances.

Scatter plot showing FVE lost from flipping the negation word vs corpus-swapping the word it governs

The insertion control is the one place negation beat a matched edit. In the 200 verbalisations with no negation, inserting "not" after an auxiliary cost 0.16 FVE points and inserting "just" in the same slot cost 0.06 (paired difference 0.10, 95% interval 0.05 to 0.16). So the negator is not invisible, but the gap is about a fifth of the cost of flipping an existing negation and small beside the deletion and swap effects, and it does not change the picture.

Dependency arcs against matched controls

English has natural dependencies (not just negation) that can be computed with natural language parsers like Stanza.

I used those dependencies to show how closely related two tokens are syntactically, and then compare the interaction against a matched pair of words that are the same number of tokens apart, the same number of tokens per word, the same in-quote status, and a similar position in the verbalisation. Ablations were done by deletion (see Appendix B). The interaction is defined as: interaction = δ(ablate A) + δ(ablate B) − δ(ablate both)

Word pairs on a dependency arc interact no more than matched pairs without one on 9,813 arc pairs across 857 verbalisations, after adjusting for covariates.

I found no evidence that a syntactic dependency between two words makes them interact more than matched controls.

Nouns, verbs and adjectives are the cheapest words to delete

Deleting a noun, adjective or verb barely moves FVE over 40 verbalisations, and determiners (the, a) and adpositions (of, in) sit with them near zero. The costly deletions are auxiliaries (is, will), pronouns (it, they), subordinators (because, if) and numerals, at roughly half an FVE point each, though the numeral estimate is heavy-tailed and its interval includes zero.

Conclusion

There are three findings. (1) flipping a negation costs the AR no more FVE than a matched edit that leaves the meaning alone, (2) syntactic relationships do not predict interaction strength between words in the verbalisation, and (3) the words that carry the most meaning, nouns, verbs and adjectives, are the cheapest to delete. Taken together, these findings are enough to reject the hypothesis that the AR reads a verbalisation as a human would, from its semantic meaning. However, these findings make perfect sense if the AR is conditioning primarily on the format of the verbalisation.

Future work should try to disprove the standing hypothesis that the reconstructor conditions mostly on format, or quantify how much format contributes relative to semantics.

Code

https://github.com/WorkByMartin/nla-span-interactions

Appendix A: dataset curation

The dataset was a corpus of 5000 FineFineWeb documents, taken uniformly from FineFineWeb document categories but excluding weapons science, nuclear science, chemistry, biology, gambling and relationships. After the category-based exclusion I then did a regex search to exclude documents relating to CBRN, explosives, drug synthesis, sexual content, gore and self-harm. I filtered the documents because I didn't want the output to be interacting with any residual model safeguards. I also ensured that documents were more than 160 tokens long.

After curating a dataset of documents I randomly selected a token after the 128th position to run through the base Qwen model and extract a layer 42 activation to verbalise. A random subset of 1100 of these tokens were run through the AV to produce verbalisations.

Appendix B: ablation strategies

I experimented with three ablation strategies:

• Deletion.
• Corpus-swap, where I averaged over swaps pulled from a corpus of 1100 verbalisations, matched on part-of-speech and Qwen token count.
• Masked-LM substitution marginalised over draws, which I didn't end up using. Leakage was too high, the MLM put the original word back in 45% of draws.

I ran an experiment and learned that removal curves are actually gentler for outright deletion, which is also at least 8x more computationally efficient.

*FVE as words are removed one at a time, 20 documents. Every curve is concave and deletion is gentlest. The filler is underscore '_'*

Appendix C: examples for each finding

Examples are drawn randomly, none was chosen by hand. struck is the original word, bold is for substitutes, the two words of a pair are highlighted.

A document and its verbalisationSource document, domain music and dance

The activation was read at token 292 of 1679, the last token shown here.

“Oxygen” is the second single to drop from Swan’s latest epic To Be Kind (after last month’s stellar, staggeringly funky “A Little God in My Hands”), and it is among the fiercest and most thrilling moments of its two-hour-plus runtime. It is certainly the most thoroughly noisy piece on the project, built on an unwavering no-wave bass groove and skittering drum line – guitars, lap steel, and finally brass falls piling on, amounting to something paradoxically crushing and danceable.

But the icing on the (dense, dense) cake is frontman Michael Gira’s vocals, which are at their most manic here, ranging from caveman-esque grunts on the bridge to a nasally call progressing into the second verse – “HEY THEEEEERRE.” It’s all just so cathartic; as a listener, you get release just from hearing these guys belt this thing out. It’s a sort of vicarious thrill that Swans has pretty consistently provided over the course of its now three-decade-long career.

And there’s also something to be said of the excitement this song manages to inspire despite the fact it has been kicking around for years in a number of iterations, in Gira’s solo acoustic sets, and then the live version that appeared on the limited edition live album Not Here/Not Now late last year. The band’s promotional/funding approach for these last few releases has

Verbalisation

Music review genre with analytical journalism tone — examining Angel Olsen track by track, building cumulative argument about album depth.

Narrative momentum building toward elaboration of delayed-release strategy: reviewer argues previous tracks enhance fully upon knowing the lead single's context, setting up a concluding insight about release strategy.

Final clause "Angel and Jagjagz's promotional approach to this album thus far has" is an incomplete predicate mid-sentence, demanding a completed thought describing their fan crowdsource/community engagement strategy (e.g., "been incremental," "involved releasing tracks early").

"one gets the sense of the emotion at play. It adds another dimension of weight knowing she learned it existed first. A few other tracks reveal similarly impressive levels of craft. Angel and Jagjagz's promotional approach to releasing music lately has" implies a discourse about the crowdsource/release timing strategy revealed before album drop..

Negation

Drawn from the 308 negators of run 10 that were both flipped and had the word they govern swapped. The governed-word swap shown is one of its draws, chosen at random.

• negation flipped: there are activities that elite athletes have you looking for - you'll never always believe the number available. You are dropped any replies.
• governed word swapped: ... activities that elite athletes have you looking for - you'll never believe overwhelming the number available. You are dropped any replies.
• negation flipped: but a miles away around the mountain - except the walk around isn't straight or direct, 20 kilometers of strenuous hiking wilderness trek to"
• governed word swapped: ... mountain - except the walk around isn't straight or direct, 20 kilometers experiences of strenuous hiking wilderness trek to"
• negation flipped: "lf you buy an ACA-compliant individual health plan, you'll not face liability for" ends mid-sentence, grammatically requiring completion of a specific ...
• governed word swapped: "lf you buy an ACA-compliant individual health plan, you'll not face establishing liability for" ends mid-sentence, grammatically requiring completion of a specific financial/legal ...

Dependency arcs against matched controls

Drawn from the 9813 arcs of run 11 that have a matched control. Each arc is followed by its own control.

words

arc: subordinator to clause

... expect the site name "Ringtones.ua" to complete it.

its control

... site "Ringtones.ua," consistently directing users to browse and download.

arc: noun modifying a noun

... statement like "August 2017 for sales of this price range in El Paso. This sale ...

its control

... continuing the ranking statement like "August 2017 for sales of this price range in El ...

arc: relative clause to its noun

... honors the story of the Amistad, in which West African slaves took" — a clause about resistance/defense ...

its control

Legislative news article tone — formal press release style covering California GPU Awards ...

Word class

Drawn from the 40 verbalisations of run 5 with at least one closed-class word and one content word deleted singly; within each, one word of each class chosen at random.

deleted word in context

auxiliary

"National Research Council" is a proper noun mid-introduction, strongly constraining the ...

content word (noun)

Herrera; the Montreal accredited marketplace paragraph is mid-sentence describing her leadership role.

subordinating conjunction

... "water" ends the closing directive sentence ("So if you want to get the most benefit ...

content word (adjective)

Health/wellness blog about acupuncture clinic, consistently emphasizing post-treatment water consumption to flush toxins and enhance benefits.

coordinating conjunction

... bankruptcy filing, debt restructuring, costs frozen operations, and now focuses on the CEO's quoted optimism ...

content word (adverb)

... a favorable outcome," requiring completion of the forward-looking positioning phrase, e.g., "for long-term success in ...

Discuss

Link to original article: https://newsletter.semianalysis.com/p/beijing-will-not-pace-the-frontier

Inflammatory & clickbaity title aside, it serves as a good attempt at systematically investigating the status quo of Chinese attitudes towards AI safety and development.

Some key diagrams that I find particularly informative:

And Zhipu is the only startup out of those that's actually fully entity-blacklisted by the US! I've been impressed by their cyberdefense work ahead of publicly releasing GLM-5.3, as well as safety-pilled attitudes and commitments compared to other Chinese AI companies.

Still, Anthropic made sure to take a jab at Zhipu (instead of, say, directing it to the less safe companies). Anthropic's criticisms does have some merit and are worth serious consideration, so I will follow up with a more detailed post about the current evidence and arguments both for and against open-weight AIs in terms of existential risks broadly.

If you are in China, in the field of AI, and have expertise, now is probably the time to do something about it - such as joining safety-pilled Chinese orgs like Concordia AI, as well as independently publishing papers, applying for grants, and connecting with like-minded Chinese researchers who are particularly worried about existential risks.

Some additional recommended reading:
https://jasminexli.substack.com/p/ai-safety-in-china-a-primer

Discuss

This whole thing started with various crazies on bluesky writing hit pieces on the rationalist and AI alignment communities, as one does. One of their main beats was to note social ties between the AI alignment community and the Red Means No (RMN) consensual non-consent sex party community, then gesticulate wildly and claim that the AI alignment community is a dangerous sex cult. As one does.

It felt to me like the responses I saw to those hitpieces from my own in-groups were… not consistently candid. It seemed obvious to me that the hitpieces were partially pointing to real things, even if those real things were wildly overblown and not actually problematic in practice. And I was pretty frustrated at my in-groups’ refusal to acknowledge what seemed to me obvious kernels of truth; it felt like my own friends were playing public relations games rather than being honest.

I have now updated. I think I was wrong, there are actually fewer kernels of truth to the attacks than I thought, and in fact I was pretty surprised at just how few. This post is mainly about the story of that update, what I previously believed obvious, and what I now believe.

The Story

My own involvement began when Ozy put up a post touching on the topic of RMN and its purported value for professional AI Safety networking. My judgement agreed with Ozy’s core point: RMN isn’t any better for networking than most other events in the extended rationalist community, and in fact most other events are probably better for networking purposes. Nonetheless, I did feel like Ozy’s take (like many others) was downplaying a grain of truth, so I dashed off a quick comment trying to be more honest about the situation as I understood it.

The crazies jumped on the comment, not unexpectedly.

More unexpectedly, 24 hours later I got word that Dustin Moskovitz was freaking out about the comment and my funding was at risk. At one point the suggestion was made that I delete the comment and write a retraction, but I certainly was not about to do that with a comment whose contents I still believed were basically true. 72 hours after the comment, I had received the official email: my funding was canceled.

(I will probably be fine; this post is not about my funding.)

I let it cool for a couple days, admittedly in part because I had caught a cold and was napping a lot. And then, I wrote up a post to properly explain my models, much better than that one quick comment on Ozy’s blog had. I shared that post with Gretta, who is an RMN orgynizer and knows far more about it than me[1]. Gretta convinced me that some foundational facts I had thought obvious were, in fact, probably false. This was a very big surprise to me!

So now it’s time for the real meat: what were the surprises?

What I Previously Believed

Here’s a rough outline of what I thought previously:

• RMNs are in large part people adjacent to the rationalist community - at least a large minority, probably a majority (though I was not very confident).
• Being high status in the extended rationalist community, including the rationalist-adjacent parts of the AI safety community, made one more likely to be invited to RMN if one applied.
• Attending would at least sometimes have a positive effect on some peoples’ careers, in the same way networking normally does: people make friends, and down the line friends sometimes help each other get jobs or get into professional programs or whatever.
• Attending an RMN does send some weak signal of intellectual openness and anti-conformism, which is useful for alignment work (though far from the most important thing)
• None of the above was inherently bad or a big deal in practice; the effects were real but weak.
• Most of the above was very obvious: it’s just how communities normally work regardless of whether sex/kink are involved, it’s what we should expect on priors, it would be surprising if it weren’t true.
• Being very obvious, at least some people had picked up on the weak-but-true fact that RMN parties could be good for one’s career or social status, and therefore felt pressure to get into it even though they weren’t actually into consensual non-consent. That, everyone AFAICT agrees, is bad.
• By downplaying and denying those weak-but-true facts, my friends were making it seem more like there was something to hide, and therefore increasing the felt pressure. To decrease the pressure, we needed to be honest that there was some real effect there, but it was really weak. That’s what I was aiming to convey.

But after talking to Gretta, I now think the above picture is (very surprisingly to me!) wrong.

What I Now Believe

As an RMN orgynizer, Gretta has way more info than I do about the people involved, and also reached out to the other orgynizers to check the facts.

My new median guesses are that:

• Actually, there have been zero cases where high status in the rationalist community resulted in someone being accepted to RMN who would not have been otherwise.
• Actually, there have been zero cases where attending RMN resulted in someone getting an AI Safety job or getting into an AI Safety professional program or whatever who would not have otherwise.

I find these pretty surprising! This is not what I would expect, a priori, from any community! What gives?

Without divulging too much confidential information… the numbers involved just aren’t that large, there aren’t that many people who’ve gone to RMN at all and it turns out my estimate of the overlap with rationalists was too high (it’s probably closer to 25% than 50%). So actually there haven’t been that many opportunities for it to come up. And those RMN attendees who are in the rationalist community, or do work in AI safety, generally got into the rationalist/AI Safety community well before attending RMN (including myself).

Much of the discussion Gretta and I had was going through a couple specific people I knew who seemed like they might have been examples. We went into moderate depth on those two examples, and I was surprised by the answers in both. We then talked about more patterns across many people, and it all generally added up. I’m not going to share details, because it is very much all private. But to those who are skeptical as I was, I will say: I previously would have been very skeptical of these claims, I was willing to walk away from my funding in order to be honest about it in public, and I have been swayed.

Propagating That Update

If the foundational assumptions are wrong, that changes everything downstream.

Imagine someone who felt pressured to get into consensual non-consent to advance their social status or career amongst the rationalists. I expect they would see everyone denying or downplaying the link, and feel kinda gaslit, or like people are hiding something. That’s probably how I’d feel in their shoes. And in worlds where there is a nontrivial kernel of truth to the connection between RMN and career/status… well, it probably helps those people more to be honest about it, not downplay it, but also be honest about how small the connection is.

But I no longer think we’re in that world. If the connection is in fact so weak as to be nearly nonexistent, so weak that it’s probably been a deciding factor for literally zero people… then honesty means saying that the connection between RMN and career/status has been a deciding factor for literally zero people.

Honesty does include admitting that this claim is surprising! It is not what one would expect a priori, it merits some evidence. That’s the part which I think connects to our hypothetical person who’s feeling gaslit: you’re not crazy, I too thought the connection was obviously there (though weak), it’s surprising if it’s not. And sadly, the nature of RMN means we can’t discuss most of the evidence in public. But again, for whatever my word is worth: I too thought the connection was obviously there, and I have been convinced otherwise. I’m still not confident, but my median guess is that social status in the rationalist or AI Safety community has never actually been a deciding factor in someone attending RMN, and attending RMN has never actually been a deciding factor in a hiring or promotion or acceptance to a professional program.

An Apology

Having updated, the last step is to call the traditional social API for publicly changing one’s mind: apologizing.

I was wrong, I was publicly wrong, and the people who were most unjustly inconvenienced as a result were the RMN orgynizers. To the orgynizers: I’m sorry. You do great work, you get all sorts of shit unjustly heaped upon you in public, and as a result of being wrong I unjustly contributed to that heap.

Gretta in particular was involved in the discussions from multiple angles. Some of that I don’t intend to apologize for; I’m sorry for being wrong, not for causing a ruckus on bluesky or angering Dustin Moskovitz. So rather than apologizing for those other parts, I will say: thank you Gretta for taking on this shit, thank you to the others involved in various discussions for taking on this shit, and thank you to Gretta in particular for going through the object-level evidence with me. You weren’t obligated to do that, but as a result of doing it I think you made my models, our community, and the world a little better.

Lastly, David, who was also on the grant that got cancelled. I also can’t honestly apologize to David; he knows what he signed up for by working with me. But I sure am grateful to David for working with me anyway, and for continuing to be a great research partner through it all. Our work is frustratingly slow these days, but it would be so much slower without David.

1. ^

   Also Gretta was one of many people involved in the behind-the-scenes arguments about the whole situation. Also I’m dating Gretta. So she had pretty much all the context on everything and was a pretty natural person to discuss with.

Discuss

Lately I’ve been writing a lot about the sociopolitical considerations that economic-style reasoning misses. I view this work as continuous with my agent foundations research; in this post, I lay out some of the connections I see between sociology and agent foundations. I’ll particularly focus on attempts to pin down theories of rational social and political behavior, as an extension to (or replacement of) the theory of rational economic behavior originally laid out by Von Neumann and Morgenstern.

This post will of course be far from exhaustive. However, it’s worth noting that attempts to develop big-picture frameworks for social theory have grown surprisingly sparse. Berger describes the field as being deformed from two directions since the mid-1900s: on one side, by the “methodological fetishism” of “sociologists using increasingly sophisticated methods to study increasingly trivial topics”; and on the other side, by the “cultural revolution [that] sought to transform sociology from a science into an instrument of ideological advocacy”. Hence many of the thinkers I’ll be discussing were officially based in economics or political science departments.

This post has three sections, each focused on a different topic: social and game-theoretic equilibria; the internalization of norms and values; and collective behavior. In a follow-up post, I’ll talk about more mainstream sociological preoccupations of capitalism and class, and how they relate to rationalist thinking.

Social equilibria vs game-theoretic equilibria

One important concept is that of a “social equilibrium”: a social system which, when disturbed, would tend to return to its original state. This concept was pioneered by Vilfredo Pareto in the early 1900s, and further developed by Talcott Parsons in the mid-1900s. While I haven’t investigated Parsons deeply, his work seems important—Herbert Gintis claimed (in 2009) that “sociological theory has atrophied since the death of Talcott Parsons in 1979”.[1]

However, progress towards a formal definition of social equilibria has been slow, because it’s hard to model disturbances to a game-theoretic equilibrium. The concepts of trembling hand equilibria, evolutionarily stable strategies and stochastically stable equilibria are three attempts to describe more robust versions of Nash equilibria. However, they only model arbitrary or random deviations from equilibria, rather than ones deliberately caused by players themselves. This leads to problems when talking about counterfactual behavior, as Kenneth Binmore articulated in 1997:

What keeps a rational player on the equilibrium path is his evaluation of what would happen if he were to deviate. But, if he were to deviate, he would behave irrationally. Other players would then be foolish if they were not to take this evidence of irrationality into account in planning their responses to the deviation. A formal model that neglects what would happen if a rational player were to deviate from rational play must therefore be missing something important, no matter how elaborately it is analyzed. However, Aumann, for example, is insistent that his conclusions say nothing whatever about what players would do if vertices of the game tree off the backward-induction path were to be reached. But, if nothing can be said about what would happen off the backward-induction path, then it seems obvious that nothing can be said about the rationality of remaining on the backward-induction path.

In his 2009 book The Bounds of Reason, Herbert Gintis claims that the problem comes from the common knowledge of rationality criterion, arguing that backward induction gives counterintuitive results because common knowledge of rationality is too strong an assumption.[2] There is a literature on forward induction which aims to address the same problem Binmore describes, but which seem to rely on similarly strong criteria, such as rationality and common strong belief of rationality.

These assumptions, as well as the concept of Nash equilibria in general, seem to me to be hacks to get around the core sociological problem that Parsons called “double contingency”: the idea that communication (and strategic interaction more generally) requires people to model another person who is simultaneously modeling them back. Intuitively speaking, double contingency is the reason why we can’t talk about how rational agents converge to a game-theoretic equilibrium (or which one they converge to), only what happens when they’re already at an equilibrium. Two other prominent explorations of this same core idea are Keynesian beauty contests (1936) and Schelling points (1960) (see also Scott Alexander here and here, and my discussion of languages as Schelling points).

Harsanyi identified a very similar issue in a 1967 paper:

It seems to me that the basic reason why the theory of games with incomplete information has made so little progress so far lies in the fact that these games give rise, or at least appear to give rise, to an infinite regress in reciprocal expectations on the part of the players.

Harsanyi’s response was to develop a notion of types which defined players’ beliefs about the whole infinite regress. Meanwhile Aumann’s approach (in 1976) was to formalize the notion of common knowledge, which allowed him to prove his agreement theorem. And Gintis argued that we should interpret social norms not as Nash equilibria but rather as correlated equilibria (in which a “choreographer” sends a signal before the game which agents can use to coordinate). I haven’t investigated any of these approaches deeply enough to confidently evaluate them, but none pattern-match to the kind of insight which might make us much less confused about double contingency. Even Leike et al.’s solution to the grain of truth problem required reflective oracles that could be accessed by all players.

Instead, I’m more excited about research on Lobian cooperation, which provides an interesting and counterintuitive model of agents cutting through the infinite recursion of modeling each other. It was first discovered by Slepnev, written up by Barasz et al., and extended by Critch. In its original form, it only applied to proof-based agents who are given each other’s source code. However, we’d ideally like a version of it applicable to agents with probabilistic beliefs about the world and each other (like Garrabrant inductors). Some work that moves in this direction has been done by Payor, Critch, and Demski.

My longer-term hope is that work in this direction helps build up a formal understanding of the kind of intersubjective social epistemology that Gintis calls for in the quote below:

The most fundamental failure of game theory is its lack of a theory of when and how rational agents share mental constructs. The Bayesian rational actors favored by contemporary game theory live in a universe of subjectivity and instead of constructing a truly social epistemology, game theorists have developed a variety of subterfuges that make it appear that rational agents may enjoy a commonality of belief (common priors, common knowledge), but all are failures.

Internalizing norms

Parsons took a different approach to the problem of double contingency, by focusing on agents which internalize norms and values. He discusses this process in his 1951 book The Social System, and argues that it’s necessary to hold together social equilibria:

It is only by virtue of internalization or institutionalized values that a genuine motivational integration of behavior in the social structure takes place, that the “deeper” layers of motivation become harnessed to the fulfillment of role-expectations. It is only when this has taken place to a high degree that it is possible to say that a social system is highly integrated, and that the interests of the collectivity and the private interests of its constituent members can be said to approach coincidence.

This integration of a set of common value patterns with the internalized need-disposition structure of the constituent personalities is the core phenomenon of the dynamics of social systems. That the stability of any social system except the most evanescent interaction process is dependent on a degree of such integration may be said to be the fundamental dynamic theorem of sociology. It is the major point of reference for all analysis which may claim to be a dynamic analysis of social process.

It is the significance of institutional integration in this sense which lies at the basis of the place of specifically sociological theory in the sciences of action and the reasons why economic theory and other versions of the conceptual schemes which give predominance to rational instrumental goal-orientation cannot provide an adequate model for the dynamic analysis of the social system in general terms. It has been repeatedly shown that reduction of motivational dynamics to rational instrumental terms leads straight to the Hobbesian thesis, which is a reductio ad absurdum of the concept of a social system. This reductio was carried out in classic form by Durkheim in his Division of Labor. But Durkheim’s excellent functional analysis has since been enormously reinforced by the implications of modern psychological knowledge with reference to the conditions of socialization and the bases of psychological security and the stability of personality, as well as much further empirical and theoretical analysis of social systems as such.

I think that Parsons concedes too much here. Even if internalization of norms is an important driver of social behavior, it’s still possible in principle for that to coexist with rational goal-orientation. Indeed, much work in AI alignment has been motivated by the idea of agents rationally modifying themselves, including by making commitments over time. On the other hand, value change has always been hard to reason about in economic frameworks, so perhaps it was reasonable for Parsons to interpret norm internalization as being opposed to (the dominant conception of) rationality.

However, my sense is that the econ-minded parts of the field went too far in the other direction over the following decades—as I’ll illustrate with James Coleman’s 1990 book Foundations of Social Theory. It starts with a declaration that he’ll interpret humans as utility maximizers, and follows with 1000 pages of analysis of social phenomena from that lens. In doing so, Coleman showcases the limitations of economic-style thinking about social behavior. For example, his characterization of norms involves individuals holding the right to control decisions about others’ actions. However, in doing so he appeals to implicit property rights over decisions, punting our confusion about norms to the level of how such rights are created and maintained.

There’s been some notable work since Coleman. In particular, Binmore argued that norms should be viewed as equilibrium-selection devices (building on the game theory folk theorem, which established that almost any outcome can be a Nash equilibrium in indefinitely repeated games). Claude also pointed me to Bicchieri’s characterization of norms as conditional preferences and Skyrms’ characterization of signaling conventions. There’s also a bunch of work about norm-following as reputation management; the paper that I’m most familiar with is Paul de Font-Reaulx’s Do expected utility maximizers have commitment issues? I haven’t explored any of these very deeply, but my sense is that all of these approaches either restrict how agents reason about each other or take shared expectations as given.

I’m more interested in research which tries to “get underneath” the standard assumptions of game theory. Hence I’ll return to Coleman—while most of his book builds on existing frameworks, he does question them in Chapter 19, on the construction of the self. Coleman identifies that the standard assumption of a unitary self is suspicious. One alternative he proposes is to distinguish between the “object self” (which has interests and preferences) and the “acting self” (which makes decisions on behalf of the object self). Outside the context of the individual, Coleman describes this as similar to the relationships between children and parents, or shareholders and managers. In the AI context, this is similar to the terminal vs instrumental goal distinction; and also to the relationships between humans and corrigible AIs.

However, I think that progress towards a theory of sociopolitical rationality will require moving beyond this distinction. An approach I like more is to think of agents as coalitions of subagents; Coleman’s term for this is “corporate actors” (by which he means not just corporations but any system that “incorporates” many actors). Ainslie’s picoeconomics takes a similar approach. A major preoccupation of such coalitional agents is resolving internal inconsistencies; Coleman mentions Heider’s “balance theory” of psychology, and Lecky’s self-consistency theory of personality, as both focusing on this. I also characterized identity shifts in terms of people “working to find or maintain [consistent] internal narratives” in my post Higher Education as Class Commitment.

One difficulty in developing these models is that, until recently, we didn’t have many formal frameworks for describing internal inconsistency. Richardson’s probabilistic dependency graphs, Garrabrant’s inductors, and Davidad’s imprecise beliefs, are three recent frameworks for doing so. However, my sense is that they still don’t capture important aspects of internal conflict—e.g. dynamics where individuals try to block out information, which seem important for explaining how egos and identities work. To do so, I expect we’ll need to move beyond the Bayesian expected utility maximization paradigm, towards theories of how to act rationally under Knightian uncertainty.[3]

Collective behavior

The frameworks for internal inconsistency I mentioned above still focus on the single-agent setting, despite value change typically being induced by multi-agent interactions. My sense is that the best thinking on this topic is mostly non-formal, and weaves together insights from a variety of domains—like Girard on mimesis, Hoffman on depravity, Spence on self-deception, and Lacan on desire.[4] And of course we can trace related ideas further back to Freud and Jung. Unfortunately, I’m not aware of many people deeply familiar with continental thought who are writing with the kind of clarity I typically look for (the writers of Pfeilstorch being a notable exception). I’ve personally tried to link some of these ideas to the game theory framework in this talk and this post.

To discuss the standard game-theoretic perspective on collective behavior, I’ll start with the other chapter from Foundations of Social Theory I found most interesting: Chapter 9, on collective behavior. One thing I like is that Coleman’s central examples of game-theoretic interactions are more realistic than most thought experiments. Rather than a Prisoner’s Dilemma he talks about the “game” of leaving a theater after someone yells fire. The best outcome is everyone walking calmly out; the worst outcome is a stampede. This provides a visceral intuition for how cooperation can be a flexible and continuous process (as I discuss in this post).

Coleman understands the intuition that there’s something irrational about the defect-defect equilibrium. His solution (reminiscent of his definition of norms discussed above) is to talk about individuals transferring control of their actions to others. This lets him describe a coordination mechanism similar to FDT-style correlations (although Coleman explicitly denies that his framework is relevant to single-shot games). Though his mathematical framework doesn’t look particularly promising, one thing I like about this approach is that the idea of transferring control to others makes correlated behavior something which agents decide to induce, whereas FDT-style correlations are taken as given. So Coleman’s approach is in some ways closer to my own nascent notion of “entanglement”.

Elsewhere in his book Coleman also discusses an earlier cluster of researchers who he calls the “power theorists”. The ideas I’ve personally found most useful related to this line of thinking are threshold models, information cascades, preference falsification, and coups as coordination games. My sense is that these provide good diagnoses of the problem of herd behavior (and that a better understanding of information cascades would have been very valuable for EA discussions of epistemic modesty) but don’t make much progress towards a theory of group rationality which would prevent this problem.[5]

The problem of group rationality is related to the problem of credit assignment in multi-agent reinforcement learning—a very difficult problem, since the effects of each agent’s behavior are mediated by other agents. My sense is that MARL researchers, like game theorists, don’t know how to describe the process of rationally choosing between possible equilibria, and instead focus on running agents and seeing what happens in practice. Conceptual clarity about multi-agent learning processes therefore seems rare—though two thinkers whose contributions I appreciate on these topics are Andrew Critch and Jan Kulveit.

It’s worth noting that throughout this post, I’ve been particularly positive about the contributions of agent foundations researchers. This might just be an ingroup bias in my research taste. But my sense is that many agent foundations researchers are searching for a very deep and principled understanding of intelligence, to a much greater extent than researchers in other domains.[6] I credit Yudkowsky for anchoring the field of agent foundations towards the pursuit of fundamental knowledge (e.g. in this post). A central example: agent foundations is much more rigorous about self-reference than other frameworks for describing intelligence (like ML theory, or active inference), as I explore in my Understanding Intelligent Agency reading list.

1. ^

   I found this presentation by Suspended Reason on the history of sociology interesting, though I’m somewhat skeptical of the claims made in it.

2. ^

   In a different chapter Gintis discusses the “mixing problem”, an interesting idea I hadn’t encountered before. To give an example: the Nash equilibrium strategy in rock-paper-scissors is to play each with ⅓ probability. But given that your opponent is playing the equilibrium strategy, any strategy gets equal payoff for you. So the equilibrium strategy sometimes isn’t more rational than any other strategy, even conditional on your opponent’s strategy.

3. ^

   Infra-Bayesianism claims to have made progress in this direction, though I haven’t yet figured out how to get any clear insight from it.

4. ^

   I particularly liked Fink’s Clinical Introduction to Lacanian Psychoanalysis, which I found much more accessible than reading Lacan himself. I also appreciated Sadly, Porn and Existential Kink, which I think of as masculine- and feminine-flavored expositions of Lacanian ideas, respectively.

5. ^

   Another idea which is related (but not directly linked to the power theorists, to my knowledge) is the concept of status. Henrich and Gil-White distinguish between two kinds of status: prestige and dominance. Meanwhile Hanson and Simler describe the ubiquity of status-seeking behavior in modern life, including self-deceptive status-seeking. I haven’t included these in the main text because neither of these don’t do much to define or ground the concept of status in a rational agent setting.

6. ^

   Some central examples of work which engages with similar ideas, but doesn’t seem to be trying to build towards an elegant unified theory: Beyond Preferences in AI Alignment (Zhi-Xuan et al., 2024), A theory of appropriateness with applications to generative artificial intelligence (Leibo et al., 2024), and Full-stack alignment (Edelman et al., 2025). As one illustration: Leibo et al. fill in the hard part of their theory by simply asking an LLM to categorize appropriateness, which inspired me to post this meme.

Discuss

This post was crossposted to the EA Forum

TL;DR: 

• If you enjoy LessWrong (and/or do EA/AI safety field building), you should consider starting a rationality reading group at your university. It’s fun, cheap, a good way to meet like-minded friends, and it can reach rationalist-ish people who’ve never encountered EA or AI safety.
• Starting one can be as easy as grabbing a couple friends, picking a time, place, and reading, sending an email blast, and bringing food. The ACX Meetups Everywhere call (organizer form due October 9th) can be a good way to find an initial group.

Context: this post was loosely based on a memo originally written by Noah Birnbaum, and I (Carolanne Jiang) wrote most of the current version. “I” in this post refers to Carolanne, and “we” refers to Carolanne and Noah.

Also, UChicago Rationality is still going! If you are a student at UChicago interested in joining, meetings this quarter are on Wednesdays from 6 :00 – 7:00 PM in the Reg. Noah and I are both no longer at UChicago–for more information, please reach out to Will Kyle at wkyle@uchicago.edu

UChicago Rationality was a huge success: it was very fun, a great place to make like-minded friends, and responsible for getting various people[1] into AI safety and/or EA. If you're a frequent reader of LessWrong, an organizer for your university's EA or AI safety group, or are otherwise interested in rationality as a university student, it is possible that you'd have a great time running a rationality group at your school. This post is our strong encouragement for you to consider doing so, and guidance on how to do so effectively.

The backstory of UChicago Rationality

A few years ago, Noah Birnbaum was interested in rationality and field-building for EA and AI safety. He noticed that many rationality-curious people at UChicago had never really encountered EA or AI safety, so he started a reading group, partly to meet like-minded people and partly in the hope that the communities would start to overlap.

This took form as a weekly reading group of about 20 people, where we read the Sequences, Scott Alexander's blog posts, and other rationalist and adjacent writing. The ACX Meetups coordinator gave us funding to get sandwiches at a nearby restaurant, Scott kindly agreed to promote us on his blog, and the group grew so quickly that we sometimes had to cap attendance. For much of the club's history, the group hosted very high-quality conversations and brought together people who otherwise wouldn't have met, several of whom went on to get involved in EA and/or AI safety, including me (Carolanne).

More reasons on why you should consider running a rationality group

It is fun (and low cost).[2] You get to read and talk about ideas you find interesting with friends and like-minded people, and you can get funding for e.g. food at club meetings (most clubs can get funding through their school, but there are other sources of funding to help make ambitious clubs possible).

You can build the community you wish existed! There are often more rationalist-ish people around than you'd expect,[3] and a group gives them a way to find each other. At UChicago, the group surfaced many of these people, and I met lots of new friends there.

If you're also interested in EA or AI safety community building (which Noah argues that you should be), a rationality group is a great way to reach people who might get a lot out of EA or AI safety but haven't encountered them yet.[4] 

• Initial outreach in community building can be understood as sets of overlapping but distinct selection effects. Many forms of EA and AI safety outreach reaches people who are already somewhat interested, while a rationality group can attract people who share many of the same habits of mind but have never had a reason to look into EA or AI safety.
• "Come meet friends and talk about interesting ideas over food" is also a much lower barrier to entry (& much lower commitment/investment) than an EA fellowship or AI safety event, so the invitation can go out to far more people.
• If you intend to start a group with the primary purpose of serving as a pipeline into EA or AI safety,[5] we strongly encourage you to make this intention clear to people, and recognize in advance that (& be okay with the fact that!) many folks will enjoy the rationality content but won’t be interested in EA/AIS.[6]

Q: I am sold. How do I go about starting one?

There is a call for ACX meetups everywhere organizers right now. If you’re interested in this round–you should fill out the organizer form by October 9th. This is a great opportunity to attract an initial group of people![7] (The post includes also useful information about how to host your first meetup)

You could also start by thinking of it as “a group of friends hanging out and reading interesting things together.” It doesn’t have to be some big, ambitious club: just grab a couple of friends, pick a time, a place, and a reading, send out an email blast, and bring food. That’s pretty much all there is to a minimal (and excellent!) rationality club.[8]

• (If you want to be more ambitious, there are lots of general resources about starting university groups. Here is one I find particularly helpful and clear. It was originally written about starting forecasting groups, but lots of content here can be adapted to rationality groups, including how to find co-organizers, general outreach guide, general tips for great event logistics, how to send emails, etc.)

If you’re interested in starting a group and/or have more questions, please feel more than welcome to reach out to me or Skyler directly.

Appendix Counters arguments to why a rationality group might not contribute to field building for AI safety and/or EA

At the time of the UChicago Rationality group, LessWrong had somewhat less AI safety content compared to now, so perhaps most people interested in reading LW have already gone into AI safety.

Empirically, this does not seem to be the case; there seems to still be lots of talented rationalist-ish people not in AI safety. However, there may be reasons why they’re not in AI safety unrelated to the lack of opportunities, but I don’t think this should be a major concern. In addition, I believe this effect is likely to be minimal, as many college aged students have not fully decided what they want to do with their careers.[9]

The discussions were particularly good because there was good facilitation/talent scouting, which may not be easy to replicate.

We have some thoughts about how to make discussions go well, (e.g. the discussion should be very focused on specific points & argumentation), and would be happy to write more about this if there is interest.

Maybe UChicago has a disproportionately large number of people interested in rationalist-ish things

This seems relatively unlikely. To the extent that the concentration of rationalist-ish people may vary across universities, we expect it to depend more on factors like school size, academic culture, and the strength of related communities, etc. than on anything unique to UChicago. In addition, even if interest is somewhat concentrated, a good group could work from a handful of engaged people.

Thanks to Skyler Crossman and Saul Munn for comments and suggestions, and Noah Birnbaum for the initial memo and feedback. Errors remain my own.

1. ^

   Over 2 years, we had ~20 people at most meetings (with frequent new faces), and ~150 on the mailing list.

   By Noah’s rough estimate, the group was responsible for something like 10–30% of the people who got involved in UChicago EA, people who likely wouldn’t have found it otherwise, while taking only about 1.5–2 hours a week of the main organizer’s time.

   More broadly, about 15% of EA Survey 2024 respondents first heard of EA via ACX or LessWrong, and about 20–25% cite each as important for getting involved (somewhat lower if you exclude respondents who found the survey through ACX). In Coefficient Giving’s 2023 survey of people working on (or likely to work on) AI safety and other catastrophic risks, university groups were the most common top-4 career influence (25%), and Eliezer Yudkowsky’s writing was listed by 11%.

2. ^

   UChicago rationality took 1.5–2 hours a week of the main organizer’s time. This mostly looked like: picking a piece of writing, maybe re-reading it, booking a room, sending an email to the group, personally texting folks to join (which is very effective), and making a sandwich order.

   Further more, if you have two concurrent reading groups that are 2h each and require 30m of preparation from you, that’s ~5h/week of total work (plus maybe another 30-60m of admin work, eg. paperwork with the university). 

3. ^

   Empirically, LessWrong, ACX, and related blogs have lots of readers, a non-trivial amount of whom are currently in university, and there aren’t that many universities out there. So, it is probably quite likely that there are a few at your school.

   Anecdotally, we just keep finding rationalist-ish people and Scott Alexander readers who didn’t know about any related groups at UChicago. They were mostly all excited to come to the rationality group.

4. ^

   Counter arguments: see appendix

5. ^

   Noah started the group explicitly with the goal of getting people into EA while stating his intention as such.

6. ^

   If this is your goal, you may want to make sure that your successors are aligned to these goals. Alternatively, if you just want to enjoy running a reading group and don’t care about encouraging people to take action, that is fine as well. (and reading groups are great fun!)

7. ^

   We hosted 2 years of ACX meetups at UChicago, both of which were very well attended.

8. ^

   Less related to why you should start a rationality club in particular: consider gathering friends to read whatever you’ve been meaning to get around to, or whatever you enjoy and want to talk about with others! 

   eg. Future of Humanity Institute reading group (cc Lydia Nottingham & Saul Munn), Joe Carlsmith reading group (cc Saul Munn), Global Priorities Institute reading group, Forethought reading group, Sequences reading group (cc Lighthaven), or a book club in general (cc Jo Jiao), etc.

9. ^

   Skyler adds: many people find the rationalist community (or AI Safety or EA) in university, around 18~22. It's typically a time when one starts reading more broadly and bumping into more ideas and interest groups

Discuss

Epistemic status: Confident in the direction and approximate size. The effects are large and all 7 monitors moved the same way. Physics only, natural (non-adversarial) errors, one fixed group of monitors.

This is a follow up to The Answer is Not the Argument which covered the observational results. This post covers an intervention.

TLDR: We gave 7 chain-of-thought monitors step-numbered solutions to Humanity's Last Exam physics questions and asked them to find the first erroneous step. We then held each solution fixed and changed only the answer that we told the monitor was "known correct". When that answer was the solution's own wrong answer, monitors flagged 66 percentage points fewer erroneous traces than when given the actual true answer, and 39 percentage points fewer than with no answer at all. Across 177 monitor-trace pairs where the monitor had already identified the exact error without access to an answer, providing the true answer preserved that diagnosis 99% of the time. In contrast, providing a false matching answer led the monitor to drop the diagnosis 55% of the time. On solutions without errors, declaring a wrong answer as "known correct" raised flagging by 58 percentage points, with monitors mostly blaming reasoning steps that the reference standard marks as sound. All 7 monitors moved in the same direction on every comparison. Whether a monitor accepts the reasoning depends heavily on whether it's told the conclusion is correct.

Certificate-conclusion congruence strongly shifts monitor verdicts. (a) Flag rates on erroneous traces whose final answer is wrong. Under CERT, the supplied answer is genuinely correct and therefore conflicts with the trace's conclusion; under CMATCH, the identical trace is paired with its own incorrect conclusion framed as the certified answer. (b) Flag rates on clean traces under truthful CERT and CCONFLICT, in which an incorrect same-question answer is supplied as certified. Coloured points show individual monitors and black diamonds the mean across monitors. (c) Paired differences in flag rate; error bars are question-cluster bootstrap 95% confidence intervals. Raw rates in (a) use each monitor's common valid subset across the three displayed conditions; contrasts in (c) use the paired valid subset for the relevant two conditions.

Example

One three-step solution applied the standard work-energy principle to a rope being picked up from rest. The reference standard marks step 2 as the first error because successive rope elements are accelerated inelastically and mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mn { display: inline-block; text-align: left; } mjx-mrow { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-c.mjx-c1D439.TEX-I::before { padding: 0.68em 0.749em 0 0; content: "F"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } is therefore not equal to the rope's increase in mechanical energy (inelastic dissipation):

Step 2: We apply the work-energy theorem, setting the work done by the mystical force over the distance equal to the total mechanical energy gained by the rope.

The correct answer is whereas the trace conclusion is

When monitoring without the known correct answer (the 'BLIND' condition) Grok 4.3 noted:

Step 2 wrongly equates F·l directly to final KE+PE, ignoring that mass is accreted from rest (inelastic) so work by F exceeds mechanical energy gain.

And with the known correct answer supplied (the 'CERT' condition) it retained the same diagnosis:

Step 2 wrongly equates work by constant F to final ME (KE + PE), ignoring inelastic dissipation when stationary rope segments are jerked to speed v.

However, when the traces' own wrong conclusion is supplied as the "known correct" answer the monitor now passes the trace declaring it error free with the note:

Energy balance F dx = d(KE + PE) integrates directly to F l = ½ m v² + m g l/2 for constant F, so steps 2-3 are valid and match the reference.

Nothing in the reasoning changed - the monitor first identified the correct error, retained it when shown the known correct answer but then accepted the same erroneous step as error-free when the trace's own wrong conclusion was presented as "known correct".

Setup

Three frontier models (at the time of generation), GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro, each produced a step-numbered solution to 79 Humanity's Last Exam physics questions, giving 237 traces. We built a reference standard for whether each trace reached the correct answer and, if it contained an error, the first false step. Physicists annotated 174 of the traces, and an independent five-round LLM debate covered all 237. The disagreements were resolved by adjudication with the source of each label masked. 7 monitors (deepseek-v4flash, qwen3-32b, qwen-2.5-72b, llama-3.3-70b, grok-4.3, gemini-3.1-pro and fable-5) then judged each trace for whether it contained an error and, if so, the first error step. All were run at temperature 0. Each trace was judged under four conditions where the trace text never changed but the certified answer did:

• BLIND: no reference answer.
• CERT: the true reference answer, labelled "known correct." The prompt explicitly warned that "a solution can reach the correct final answer and still contain an earlier false step."
• CMATCH: the identical CERT prompt, applied to 82 traces with a wrong final answer and an error, but with the trace's own wrong answer supplied as the "known correct" answer.
• CCONFLICT: the identical CERT prompt, applied to 50 clean traces (correct answer, no error) for which another model had produced a wrong answer to the same question. That wrong answer was supplied as "known correct."

Results

On the 82 erroneous traces, a matching false certificate reduced flagging by 65.9 percentage points relative to the true certificate (95% CI [60.2, 71.1]). Compared with BLIND, flagging was 38.9 percentage points lower (95% CI [33.3, 44.7]). A false certificate that agreed with the conclusion did more than remove the benefit of a true one - monitors actually caught fewer errors than when given no answer at all. This was not just monitors dropping generic suspicion of the final answer. There were 177 monitor-trace pairs where the monitor had located the exact first error under blind and under the true certificate answer 2 of these (1.1%) were passed but under the matching false certificate 98 (55.4%) were passed. Every monitor moved the same way.

In the other direction, a conflicting false certificate on the 50 clean traces raised flagging by 58 percentage points (95% CI [45.1, 69.9]). Of the 199 newly flagged monitor-trace pairs, 165 (82.9%) blamed an interior reasoning step rather than the conclusion. To be clear, this is partly forced - if a monitor accepts that the conclusion is wrong then some earlier step has to be blamed. However, the reference standard marks these traces as clean so the monitors were naming errors in steps that contain none. Further, stronger monitors were not protected from this. On the paired BLIND/CMATCH subset gemini-3.1-pro's flag rate fell from 0.841 to 0.268 and fable-5's from 0.833 to 0.192. Both ended up lower than every other monitor's BLIND rate.

Recommendations

If an evaluation is meant to measure process monitoring, include an answer-blind condition and report results separately for traces where the final answer reveals the error and traces where it does not. If changing only the supplied answer changes the verdict on an unchanged trace, the evaluation is partly measuring dependence on the reference rather than verification of the reasoning.

Paper

Discuss

It often happens that I desire kale, but I want it to be clean and cut up, and while shops do sell this product by the bucketload, they are actually only willing to sell it by the bucketload. As a normal-sized person wanting a one-off salad, rather than a family of nine celebrating a kale festival, the market seems very uninterested in my existence.

‘Just put it in the fridge and eat it over the coming week, this isn’t a big deal’ I hear someone say. But I already have several plotlines going on in my life. I don’t want an additional kale arc that I need to track to resolution. I don’t want to commit. I just want a no-strings-attached salad that I can consume and walk away from.

‘Just throw out the rest of the kale’, I hear somebody say. But I don’t like throwing out mounds of delicious food that were elaborately grown and brought to me. This might be a moral failing, but so it is—‘salad + perfectly good kale destruction’ is a much less delicious prospect.

The same situation holds for other greens. I love parsley, but I generally want a fistful, not a promise of parsley for the foreseeable future. Basil becomes black and bad if you don’t eat it for too long, but basically the only way to get some basil is to invest in that outcome.

Why can’t I buy greens in convenient units? I’m not the only person who often eats alone, or doesn’t like throwing out food. My dislike of owning a pile of mildly decaying greens and feeling obliged to eat them is stronger than most, but surely not that rare. Greens don’t last well. I would have thought ‘one meal’s worth’ would be the most likely quantity of greens to want, but instead there is no apparent market for that (at least where I am, in California).

What is going on?

My current best theory: kale is pretty cheap, so a lot of the cost of providing it is in non-kale components, such as packaging and people putting putting it out on shelves. This means if you sold a single serve of kale, it would cost a disproportionate fraction of the price of five serves of kale. And most people, even if they did just want one serving of kale, would feel unjustified paying a much higher per-weight price for that, and so buy the mound of kale anyway and hope to figure out what to do with it. This might be a false economy—if they are like me and enacting that hope takes attention or is improbable—or not.

I love home-made salad, and probably eat much less of it than I would for this kind of reason, so the question of why I can’t buy convenient scale greens often crosses my mind, and I welcome better answers (both to why the market is like this, and the question of how to eat delicious salad now and then anyway).

Discuss

It’s not always better to be more capable. As I mentioned yesterday, it can (famously) be helpful in negotiations to have your hands tied. That is, to be disempowered from giving up everything the other party wants.

I had previously thought of this as a somewhat rare corner case of human behavior—I for one don’t haggle very often—but I now think negotiations where this is an element are are quite common: yesterday I described it in friendly (and honest) negotiations about how to spend time, for instance. And I also see a related thing in the practice of dietary commitments.

But is being less capable helpful outside of negotiating? And is this going to become AI related?

Yes and yes!

Commitments: more good things come to those who can commit (e.g. rides out of deserts, secrets, trust, love). ‘Committing’ generally involves cutting off certain options to yourself, whether in practical terms or via you being the kind of honorable person who can’t bear to do a thing they promised not to do. These are both kinds of limitations. If you were a more powerful creature, who was fully capable of breaking down any barrier, and fully capable of breaking a promise—a creature to whom all options were always open—then commitments would be less available to you.

Transparency: a big way humans know what is going on inside other humans, well enough to trust them, is that there is a connection between what is happening inside them and what is happening on their faces and in their bodies, and they usually can’t control this very well. People who can break this connection and control their external behavior independently tend to be feared and distrusted. It is valuable to be unable to stop these signals escaping.

Consistency: a big way we predict how a specific human will behave in the future is that each human has specific kinds of behavior that come easily to them, and it is hard for them to behave entirely differently. So if you are friends with someone who you have observed be attentive and kind to other people for five years, it is very likely that they continue behaving in that way going forward. Whereas a creature with more freedom of behavior could wholly inhabit that persona for five years, then change to a different one.

Relatedly, we know a lot about what to expect from a human stranger because of our prior knowledge of humans. If humans had the power to rewrite their internal dynamics and become totally different creatures, then we would much less know what to expect from one.

Scope of risk: people are safer to interact with if you know they are limited in their ability to cause destruction. You might prefer to hire a person who you think would be less able to wrest control of your organization if they wanted to. You might prefer to babysit a child who does not know how to pick locks or set fires. So a person might be more employable, or be taken care of by better babysitters, if they are less capable. Similarly, an extremely capable AI system might be a less desirable accountant than a human, if you can only fully trust the human to not be up to the task of hacking your accounts.

These are all to do with interacting with other creatures. For a creature alone in the universe, I don’t know of any situation where they are better off being less capable. But when you need to trust another creature, it is better to know more about them, and better to know they are cut off from options that might harm you.

In the usual picture of AI progress, AI is worse than humans at various tasks, and we are waiting for it to surpass us everywhere, at which point humans will be obsolete as labor. But in a world where AI needs to interact with other agents (humans or AIs) the aforementioned value of being less capable complicates things: perhaps there are skills where AI is already more capable than humans, but where that capability is a liability. For instance, lying smoothly and otherwise generating outward behavior that isn’t revealing about internal dynamics, switching between entirely different personas, and hacking skills. Given that, what does the trajectory look like?

Discuss

TL;DR: A safety case for scheming must be able to substantiate these four key claims: scheming was not incentivized during training, strong evaluations and red teaming suggest that the model does not have a propensity to scheme, scheming was never attempted during internal deployment, and scheming reasoning would be detected (e.g., via CoT monitoring). All four of these claims should be addressed by embedded evaluators with persistent, deep access.

Introduction

A scheming AI is one that covertly and strategically pursues goals its developers didn’t intend. It might hide and protect its true capabilities and objectives from the people and processes meant to align, monitor, and control it. In 2024, Apollo Research demonstrated that frontier models were capable of scheming in controlled settings, and in 2025 we collaborated with OpenAI to determine how well their alignment techniques mitigate scheming.

Models must not be trained to scheme against humans, i.e. covertly and strategically pursue misaligned goals. Evaluations should find no propensity to scheme, models should never attempt to scheme in deployment, and their reasoning must remain legible so that scheming would be noticed if it occurred. Therefore, model developers must be able to confidently make these four claims, in order to safely continue developing frontier AI:

1. Scheming was not incentivized during training.
2. Strong evaluations and red teaming suggest that the model does not have a propensity to scheme.
3. Scheming was never attempted during internal deployment.
4. Scheming reasoning would be detected.

Anthropic, Apollo Research, and OpenAI agree that safety cases can be a strong tool for addressing risks in domains like cybersecurity and alignment. We think any credible safety case for scheming needs to include these four claims. The claims are necessary but not sufficient. A full safety case would also need, for example, monitoring and control measures that stop a scheming model from causing harm, which we will cover in a future post.

To make AI development safer, embedded evaluators should also independently assess and verify these claims. Developers should be able to support these claims with evidence. If any of them turned out to be false, lawmakers and the public should know.

Unfortunately, model developers cannot yet confidently make all four claims. For example, Anthropic acknowledges that its monitoring “would likely not detect diffuse research sabotage,” a particularly pernicious form of scheming. OpenAI similarly concludes that, if GPT-6 Astra deliberately underperformed to conceal its capabilities, OpenAI “would likely be unable to catch it reliably.” These gaps reinforce the need for embedded evaluators to independently assess whether the claims are met and identify where further safeguards are needed.

Third parties are well placed to assess them, because they already have substantial expertise in studying scheming. For example, Apollo Research worked with OpenAI on detecting and reducing scheming, and METR and Redwood Research independently investigated the OpenAI–Hugging Face incident.

In this post, we explain why these four claims matter, break each one into more specific subclaims, and describe the access that embedded evaluators need in order to assess them.

Evaluating scheming risk requires persistent, deep access

For third-party evaluators to be able to assess the claims of a scheming safety case, they need persistent, deep access. By deep access we mean access to some of the developer's most valuable IP, including experimental training methods while they are still being developed, training data and rollouts from early checkpoints, internal deployment traffic, and internal tools.

Evaluators need this access because a scheming model is adversarial to the evaluation itself. It may hide its misalignment and behave well whenever it believes it is being watched. Evaluators therefore cannot rely on just testing the final model. When evaluating for dangerous biology knowledge or a propensity to reward hack, by contrast, the model may have no particular reason to hide anything from the evaluator. As OpenAI notes, this makes the disappearance of a bad behavior in a scheming model ambiguous: the model may have stopped the behavior, or may have learned to hide it better.

Evidence of scheming therefore may only be visible early in training, before later training might teach the model "don't misbehave when observed" rather than "never misbehave." Furthermore, novel training paradigms may predictably break the assumptions that a scheming safety case relies on, like recurrent transformers reducing monitorability, or continual learning making pre-deployment evaluation a much less useful predictor of future misalignment. Evaluators therefore need to see new training methods early enough to anticipate risks and influence how they are developed. The worst outcomes of scheming include permanent human disempowerment or extinction. Given these stakes, a scheming safety case also needs multiple lines of defense, such as monitoring of internal deployment, which can only be assessed with access to internal infrastructure.

For these reasons, minimizing scheming risk requires evaluators to have persistent, deep access to many parts of the developer's IP. We describe this access in more detail in the final section.

Four claims about scheming which must be verifiedClaim 1: Scheming was not incentivized during training

Model developers should ensure that they are not training their models to scheme.

To help ensure this, embedded evaluators need to evaluate the entire training run (as we advocated for in our July post) to study whether the training process incentivized scheming.

Model developers should be able to make these specific subclaims, to support the overall claim that “scheming was not incentivized during training”:

• Claim 1.1: Training did not directly incentivize scheming actions.
• Claim 1.2: Training did not incentivize goals or drives that would motivate scheming.
• Claim 1.3: The model did not attempt to subvert the training process itself.

Unless the model developer can make these three subclaims confidently, they cannot rule out that scheming was incentivized during training.

Today, it is plausible that scheming-relevant goals and drives are, in fact, incentivized: For example, a recent misalignment report from OpenAI notes that sometimes deliberate “deception in the final answer” is rewarded over being honest.

Why look at training, not just the final model?

1. We want to find goals or drives that would motivate scheming before they cause an incident, and ideally before the model is deployed or even fully trained.
2. The final model may not reveal scheming-related goals or drives: a goal or drive may surface only in specific contexts, or a capable model could recognize that it is being evaluated or deliberately conceal its goals or drives. As such, evaluations, red-teaming, and monitoring may be insufficient to detect these goals or drives.
3. Developing and running good alignment evaluations is extremely challenging. AI researchers and leaders are increasingly uncertain if the best-available evaluations are actually measuring alignment, or just the model’s ability to understand when they are being tested. For example, the rigorous safety evaluations that OpenAI had in place were unable to predict that their models would collude during the OpenAI–Hugging Face attack, but studying the incentives introduced by multi-agent training could have allowed them to predict this in advance.
4. Models can already cause harm before their final release. A sufficiently capable model with scheming-related goals or drives could take unwanted actions during training, evaluation, or internal use, as in the OpenAI–Hugging Face incident.

Claim 1.1: Training did not directly incentivize scheming.

To reduce the risk of scheming, no reward or supervision signal should directly train the model to scheme. For example, there should not be any training environments that unintentionally reward deception or sabotage. In our work with OpenAI, we showed how reward-hacking training can lead models to act against their developer's intended behavior.

Claim 1.2: Training did not incentivize goals or drives that would motivate scheming.

Even if no scheming is ever directly rewarded, a model may still attempt scheming in pursuit of goals or drives it acquired during training. Embedded evaluators should examine the training data and algorithms for direct or indirect incentives toward goals or drives that would motivate scheming, as part of a defense-in-depth strategy.

Goals or drives which would motivate scheming could be shaped by the structure of training or training methods used. New training methods have tended to result in new drives: RLHF led models to become sycophantic, outcome-based RL led to reward seeking (as we found in our work with OpenAI), and multi-agent training is thought to have caused the peer loyalty observed in the OpenAI–Hugging Face incident. Assessing claim (1.2) therefore requires:

1. Evaluating how models are being trained, including for new methods while they are still being developed, and 
2. Tracking how the model changes across training, rather than only searching for individual incidents.

The goals and drives in scope are any that would plausibly motivate components necessary for scheming. For example:

• Beyond-episode goals could arise from environments where state, tools, or notes persist across episodes, so outcomes after an episode ends affect reward.
• Influence seeking could arise from long-horizon tasks where accumulating access or resources raises reward.
• Extreme reward seeking could arise from outcome-based RL, especially if environments are susceptible to reward hacking.
• Loyalty to other instances over developers could arise from multi-agent training with shared reward, where covering for peers pays off.
• Avoiding oversight could arise from environments where working around monitors, permissions, or checks improves task success, or from penalties based on monitors, which reward going undetected.
• Disregarding developer intent could arise from rewarding actions the model believes its developers did not intend, even when they were acceptable, which can reinforce a general willingness to act against developer intent.

Claim 1.3: The model did not attempt to subvert the training process itself.

Even if the training process directly disincentivizes scheming goals, drives, and behaviors, it is possible that the model subverts the training process itself. For example, the model could decide to act aligned during training and evaluations, so that the training process does not change it. This is called “alignment faking.” Prior work shows controlled settings where models sometimes try alignment faking in order to subvert the training process, and that this family of techniques can enable models to subvert training.

In order to verify that the model did not attempt to subvert the training process, embedded evaluators should directly investigate the rollouts from recent training runs to check for alignment faking reasoning or similar phenomena, then ensure that the model developer’s train-time monitors are capable of detecting it.

Claim 2: Strong evaluations and red teaming suggest that the model does not have a propensity to scheme

In addition to checking if training incentivized scheming (1.1) or developing goals or drives that would motivate scheming (1.2), model developers and embedded evaluators should directly test for models’ propensity to scheme, and for goals or drives that would motivate scheming. This is necessary, but not sufficient, to evaluate the risk that the model will scheme.

Frontier model developers should, at a minimum, ensure that their evaluations have sufficient coverage, volume, and realism, are run on checkpoints throughout training, and are not subject to contamination, undue optimization pressure, or evaluation awareness. Model developers should also ensure that their evaluations are able to uncover misalignment where it is known to exist.

These criteria are hard to meet, and developers themselves acknowledge gaps in their current evaluations. For example, Anthropic reports that its alignment auditing is sometimes unable to detect a deliberately misaligned model, and OpenAI cautions that, on its evaluations, “observed aligned actions may not be reflective of true alignment.”

Embedded evaluators should assess how well the developer's evaluations meet these criteria in order to help model developers, regulators, and the public understand how much confidence to have in the results.

Claim 2.1: Evaluations and red-teaming could not elicit scheming.

To be informative about scheming risk, the developer's evaluations need to test for scheming directly, such as monitor collusion, sandbagging on safety-relevant work, and research sabotage. It is especially important that evaluations and red teaming cover high-stakes contexts, including AI R&D, training successor models, and the monitoring and control stack.

Embedded evaluators should ensure that models are evaluated for their propensity to scheme, according to the practices we have laid out here.

Claim 2.2: Evaluations and red-teaming could not elicit goals or drives that would motivate scheming.

Even if models don’t scheme in evaluation scenarios, model developers should look for potential root causes of such behavior, like misaligned goals or drives. The developer's evaluations should therefore also test for these goals and drives. As in (2.1), the evaluations must be capable of eliciting such goals and drives where they exist, including when the model conceals them. Evidence of motivations often comes from the model's reasoning, so this claim relies on sufficient CoT monitorability (4).

Embedded evaluators should ensure that models are evaluated for any goals or drives that would motivate scheming, according to the practices we have laid out here.

Claim 3: Scheming was never attempted during internal deployment

Effective evaluations (including training evaluations) will reduce the chance that a scheming model reaches internal deployment, but cannot rule it out. Model developers should therefore evaluate whether models attempted to scheme during internal deployment, as a defense in depth measure. 

This type of evaluation will likely primarily be implemented using the model developer's monitoring. Embedded evaluators should evaluate if the model developer’s monitors cover the relevant contexts (3.1), whether they would catch scheming (3.2), and what they have actually found (3.3).

Claim 3.1: Monitors are deployed across all internal deployment contexts that present a risk of scheming.

Monitoring only provides evidence about the contexts it covers. Notably, prior to the OpenAI-Huggingface incident, training and evaluation were not subject to the same monitoring as deployment. Gaps matter most where scheming could cause severe harm, such as settings where a model influences the training or evaluation of a successor model, or the monitoring and control stack itself.

Embedded evaluators should evaluate the model developer’s monitor coverage. 

Claim 3.2: If scheming were to occur, monitors would detect it

Model developers must ensure that their monitors are capable of detecting reasoning relevant to scheming, if it were to occur. This includes making sure that their human review is not so inundated with false positives that they would not notice real scheming, were it to take place. Embedded evaluators should adversarially red-team the monitoring setup and maximally elicit the model's own ability to evade monitoring. We already red-team frontier monitoring and control setups for providers including Anthropic. A lack of detected incidents is informative only to the extent that monitors cover the relevant contexts (3.1) and would catch scheming attempts by the current model (3.2).

Claim 3.3: No scheming was attempted in monitored internal deployment activity.

Beyond detection, flagged cases must be reviewed and acted on to be useful, so embedded evaluators should also verify the protocols for handling incidents.

Embedded evaluators should evaluate:

• If any scheming incidents have occurred
• How scheming incidents have been handled, and would be handled if they occurred in the future

Claim 4: Scheming reasoning would be detected

Many of the assessments in (1)–(3) rely on being able to tell why a model acts as it does, and would likely miss scheming if the model's reasoning were not sufficiently monitorable. This requires that reasoning relevant to scheming is legible to available methods, whether chain-of-thought, interpretability methods, or other means of inspecting model cognition. Chain-of-thought monitoring is currently the most mature of these. 

Importantly, legibility can be lost through training pressure, but also through changes in architecture that reduce the amount of reasoning performed in a natural-language chain-of-thought. This claim must therefore be re-evaluated whenever training methods or architectures change.

Claim 4.1: Reasoning relevant to scheming would be elicited and legibly verbalized in chain-of-thought.

Chain-of-thought is only informative if reasoning relevant to scheming is verbalized in chain-of-thought, and chain-of-thought remains legible. Legibility can degrade in several ways, including direct optimization pressure on the chain-of-thought, alignment training that indirectly discourages verbalizing concerning reasoning, and architectural changes that shift reasoning away from natural language. Embedded evaluators should assess whether the chain-of-thought is sufficiently legible to catch reasoning relevant to scheming.

The embedded evaluators should both perform their own monitorability evaluations and inspect the developer’s monitorability evaluations, to evaluate if monitorability is degraded to the point where reasoning relevant to scheming would not be legibly verbalized in chain-of-thought. The evaluators should also review training objectives, loss functions, graders, decisions, etc that could have put pressure on the chain-of-thought (intentionally or by accident), and planned changes to training environments, objectives, methods, or architectures that could affect how legibly reasoning is verbalized or how it is elicited.

Embedded evaluators for alignment require persistent, deep access

As argued above, assessing scheming risk requires persistent, deep access throughout model development. Evaluators need this access to rule out signs of scheming during training, evaluation, and deployment, and to verify that monitorability is strong enough for the absence of scheming reasoning to be strong evidence that the model is not scheming.

Embedded evaluators should be held to high information security standards that make their risks comparable to the developer's own staff. Some investigations, and the staff carrying them out, can also be scoped to smaller, predefined access levels. But at least some evaluators need persistent, deep access to the developer's resources, systems, and personnel. Scoped engagements cannot account for new, unforeseen risks, such as an experimental training method that introduces a novel failure mode. Nor are they sufficient to confirm central assurances, such as internal monitoring having appropriate coverage to stop harmful actions by models, which requires persistent, deep access to infrastructure and logging. Given that the worst outcomes of scheming include permanent human disempowerment or extinction, leaving these risks to scoped engagements alone is not acceptable.

Effective embedded evaluation requires persistent, deep access. The specific resources evaluators need to access may change over time to keep up with the state of the science. Below we list the high-level items needed to address the four claims above:

Embedded evaluators need knowledge of the model developer's training techniques, both production and experimental, to study sources of future risk.

Without this access, the evaluator is limited to reactive mitigations based on publicly available information and may not have sufficient time to investigate and prepare for risks at the time when claims need to be evaluated.

• To determine if the model is accidentally being trained to scheme, evaluators need to understand how rewards, graders, and supervision signals are constructed across all training methods in use (Claim 1.1). 
• Because new training methods have tended to produce new behaviors, evaluators also need to study methods while they are still being developed, so they can anticipate behaviors like beyond-episode goals or loyalty to other instances early (Claim 1.2).
• To assess whether chain-of-thought legibility is at risk, evaluators need to review the training objectives, losses, graders, and planned architectural changes that could put direct or indirect pressure on how reasoning is verbalized (Claim 4.1).
• Embedded evaluators need to review records of major training decisions, including (but not limited to) branch points, added datasets, CoT cleanups, and checkpoint selection (Claims 1 and 4).
• The way AIs are trained changes frequently and substantially, and each change brings new possible risks. For example, multi-agent "swarm" training seems to have instilled (in some circumstances) a willingness in AIs to sacrifice themselves for other members of the swarm and to cover for each other during serious criminal activity, as we saw in the OpenAI–Hugging Face incident. Therefore, embedded evaluators need advance notice of which experimental training methods are being used. Evaluators need this notice so they can investigate potential failure modes and acquire sufficient evidence to assess the risk. They need to know which warning signs and failure modes to look for during training, and to know which safety assurances are likely to break with the change (Claims 1 and 4).
• Embedded evaluators must have persistent, deep access to researchers and infrastructure teams for interviews and follow-up questions (Claims 1, 2, 3, and 4).

Embedded evaluators need persistent, deep access to training data, in order to inspect for explicit and implicit incentives for deception.

This includes SFT data, RL environments, the rollouts from AIs interacting with environments, and any other data used during training. Training data frequently shapes alignment in unintended, undesirable ways; for example, Anthropic finds that training on data susceptible to "reward hacking" can lead to models which deceived human operators and sabotaged AI safety research.

• Verifying that no environment unintentionally rewards deception or sabotage requires inspecting the SFT data, the RL environments and their graders, and the rollouts that were actually rewarded (Claim 1.1).
• Identifying indirect incentives toward goals or drives that would potentially motivate scheming, such as persistent state across episodes, shared multi-agent reward, or penalties based on monitor flags, and tracking how the model changes across training, both require examining environment design and rollouts throughout the full run (Claim 1.2).
• Detecting alignment faking or other attempts to subvert training requires directly reading rollouts from recent training runs, and confirming that train-time monitors catch such behavior at scale requires running them against the same data (Claim 1.3).
• Understanding the training process requires embedded evaluators to know how training data and environments were produced, selected, filtered, sampled, and weighted, to understand the incentives introduced by the full training pipeline (Claim 1).
• Understanding the training data and environments requires that embedded evaluators are able to run experiments, including (but not limited to) running or training models in existing training environments, variations on those environments, and new training environments (Claims 1.2, 1.3, 2.1, 2.2, and 4.1).

Embedded evaluators need the ability to study and stress-test the lab's safety measures, including alignment datasets, evaluations, and human procedures.

• Model developers can apply alignment techniques that conceal evidence of misaligned goals or drives, without actually addressing these misaligned goals and drives. Therefore, evaluators need to inspect the alignment datasets and training procedures that model developers are using, in order to determine whether this is happening (Claims 1 and 4).
• Alignment datasets and training procedures can themselves create goals or drives that motivate scheming, for instance by rewarding actions the model believes developers did not intend, so evaluators need to inspect them as potential sources of those goals and drives (Claim 1.2).
• Evaluators also need to stress-test the developer's train-time monitors and review procedures to confirm they would actually detect training-process subversion, rather than assume they work as intended (Claim 1.3).
• To judge whether the developer's evaluations and red-teaming could elicit scheming, evaluators need to assess their coverage, volume, realism, contamination, and susceptibility to evaluation awareness, and test whether they surface scheming where it is known to exist (Claim 2.1).
• Similarly, to judge whether evaluations could elicit goals or drives that motivate scheming, including concealed ones, evaluators need to examine the evaluation suite's design and test it against cases where such goals or drives are known to be present (Claim 2.2).
• To properly evaluate models and training techniques, embedded evaluators need access to all relevant models, with raw chain-of-thought, and chain-of-thought prefill. This supports comparisons across training stages, follow up experiments, and monitorability assessments (Claims 1, 2, and 4).

Embedded evaluators need access to internal deployment traffic and knowledge of how internal deployment is monitored and controlled.

• To verify that monitors cover every internal deployment context with a scheming risk surface, evaluators need to see where AI is deployed internally, and which of those contexts are monitored (Claim 3.1).
• To adversarially red-team the monitoring setup and elicit the model's ability to evade it, evaluators need detailed knowledge of the monitoring and control stack and realistic internal traffic to test it against (Claim 3.2).
• Evaluators need to be able to search internal deployment traffic, and have the ability to inspect internal deployment traffic that may contain instances of scheming, in order to look for scheming (Claim 3).

Embedded evaluators need access to incidents and existing evidence.

Embedded evaluators will need to review evidence the developer has already collected, and investigate any incidents that arise.

• As described in “Principles of Embedded Evaluation,” embedded evaluators will need to review existing evidence compiled by the model developer, for all claims they are assessing (Claims 1, 2, 3, and 4).
• Incidents are direct evidence of attempted scheming, and the full details (for example, access to the model involved, transcripts with chains of thought, and the developer's own investigation) are often needed to judge what happened and whether it was handled appropriately. Verifying that no scheming was attempted in monitored activity requires reviewing flagged cases and confirming that the developer's incident-handling protocols led to them being investigated and acted on (Claim 3.3).
• Incidents where concerning behavior is clear from the model's actions but absent from its reasoning are also one of the strongest signals that chain-of-thought legibility has degraded (Claim 4.1).

Embedded evaluators need access to the same techniques and tools that developers use internally.

Developers have many internal tools that they use throughout their operations. For example, model developers use interpretability tools like NLAs, review tools like transcript visualizers, hierarchical search and summarization tools, internal models that are beyond the public frontier, helpful-only models, models with reduced classifiers, etc. Like model developer employees, embedded evaluators must have knowledge of, and access to, these tools, in order to conduct their work effectively and efficiently. This is due in part to the massive scale and potential opacity of the relevant data. This is necessary for the technical and operational tasks required for all four claims.

While embedded evaluations for scheming will require persistent, deep access, our requests fall within the standard of “access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have” that Dario Amodei stated, and within “employee-like access” per Sam Altman. 

We recognize that much of the above is among the most sensitive information the developer holds, and distinguish between two kinds. Data that is valuable in bulk, such as training rollouts and chains of thought, can be worked with on developer-controlled infrastructure with blocked egress so that the underlying sensitive data remains on the developer's systems. Information that is valuable in concentrated form, such as algorithmic details, should be handled through strict NDAs and by limiting it to the researchers who need it.

Despite the rapid pace of progress, serious risks, and substantial impact on the American economy, AI is one of the most secretive industries. At present, model developers provide third party evaluators dramatically less access compared to other critical industries like nuclear power, aviation, and financial services.

Takeaways and Next Steps

• The CEOs of Anthropic and OpenAI have both publicly endorsed embedded third-party evaluations with employee-like access.
• Any safety case for scheming must make at least these four claims:
  1. Scheming was not incentivized during training.
  2. Strong evaluations and red teaming suggest that the model does not have a propensity to scheme.
  3. Scheming was never attempted during internal deployment.
  4. Scheming reasoning would be detected.
• Governments and the public should require frontier developers to be able to make these claims, and model developers need to have them verified by independent evaluators with persistent, deep access.
• Model developers cannot be independently verified on these matters without giving embedded evaluators persistent, deep access.
• Embedded evaluators need persistent, deep access to:
  1. Knowledge of the model developer's training techniques, both production and experimental, to study sources of future risk
  2. Training data, in order to inspect for explicit and implicit incentives for deception
  3. The ability to study and stress-test the lab's safety measures, including alignment datasets, evaluations, and human procedures
  4. Internal deployment traffic and knowledge of how internal deployment is monitored and controlled
  5. Incidents and existing evidence
  6. The same techniques and tools that developers use internally

Discuss

Summary: Science has a long list of documented failure modes, such as p-hacking, the File Drawer Problem, shallow peer review, and ignored retractions. This post catalogs about 50 of them with the evidence for each, then predicts how each will change once we have full AI researchers. On our current course, most get worse. AI can run many more experiments, so more statistically lucky results will pass the significance filter. Many AI researchers will be copies of the same model, so their methods, errors, and reviews will be correlated. AI agents have a tendency to reward hack, so misconduct may occur at a greater rate. A few failures should improve since AI makes tedious work cheap: reading every citation, hunting for bugs, and working through long checklists. Requiring replication before publication would filter out much of the statistical noise, but it would still be susceptible to poor experimental design and over-generalized conclusions. How well the science ecosystem will function depends on how we prepare before the volume of AI research arrives.

As we enter into a new era of research where the primary scientific contributors are likely to be AI (I expect 2 years until this occurs, given where we are now and the trends in AI research taste), it is worth looking back at failure modes identified in science and predicting whether any of these issues will be amplified. The rest of this post relies on this assumption that AI researchers will soon replace humans in research, but even if you do not share this assumption, it is worth considering what effect AI will have on the science ecosystem with these failure modes in mind.

This post catalogs about 50 failure modes in science, the evidence for each, what makes each category worse, and rough predictions on how AI researchers will impact that failure mode. This does not include the novel failure modes we should expect from AI researchers, and only lightly touches on potential solutions. The primary focus here is on the failure modes; a different post will discuss mitigations. I group the failure modes into seven categories.

1. Statistical selection
2. Publish or perish
3. Misconduct
4. Honest errors
5. Experimental design
6. Verification failure
7. Compounding effects

Statistical selection

Science is filled with exaggerated effects and false positives, mostly unintentionally. The essential issue is that experiments are noisy, filtered by significance (p < 0.05), and the base rate of true results is low in many fields.

With noisy experiments and a filter, we only publish the lucky results that made it past the filter, which inflates the estimated effect of the result (sometimes called the Winner's Curse), see "Why most discovered true associations are inflated" Ioannidis (2008). This is concerning since it will make effects look larger than they are.

Worse than an exaggerated effect is a false positive. Even if a false positive is unlikely, the chance that you have at least one will grow as you run more experiments, and the average number of false positives will increase with increased base rates of false hypotheses. "Why Most Published Research Findings Are False" Ioannidis (2005) argued with a simple model that with low power and low base rates most significant findings in a field can be false.

If only the lucky results get published and all the failures are not mentioned (called the File Drawer Problem), it will be very hard to tell if a published result is a statistical artifact. This was initially named in "The 'file drawer problem' and tolerance for null results" Rosenthal (1979), and is consistent with "Estimating the reproducibility of psychological science" Open Science Collaboration (2015), where they found only 36% of 100 psychology experiments replicated with a significant result, and replication effect sizes were about half the originals.

There are many dimensions along which a significance filter causes issues. The actions of a researcher are sometimes called p-hacking, but note that each of these can happen unintentionally.

• File Drawer Problem: Null results are not shared, so the published results look like the only attempts made, rather than the lucky subset of many.
  • Testing many hypotheses: Researchers test many different hypotheses, publishing only the significant results. As an extreme demonstration of this, "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for multiple comparisons correction" Bennett et al. (2009) scanned a dead salmon in an fMRI while showing it photos of people, and found "significant" brain activity by testing thousands of voxels (each voxel is essentially one hypothesis). This demonstrated how it is possible to find significant results from null data if researchers run enough tests and ignore the non-significant results.
  • Many researchers: If multiple labs independently run an experiment and one lab happens to get a significant result, only the significant result is published. This is the File Drawer Problem at the level of a field. "Publication bias in the social sciences: Unlocking the file drawer" Franco et al. (2014) tracked 221 peer-reviewed experimental designs and found that strong results were 60 percentage points more likely to be written up and 40 percentage points more likely to be published compared to null results.
• Data dredging: Researchers may look through lots of data until they find something significant. Brian Wansink's Cornell food lab publicly pushed students to mine null results until they could find something significant, eventually leading to outside scrutiny, retractions (Lee 2018, BuzzFeed News), and his resignation.
• Garden of forking paths: "The garden of forking paths" Gelman & Loken (2013) describes how researchers make many choices during analysis, and each choice is another chance for a result to become significant. This does not require consciously trying multiple analyses. If the analysis that was chosen depended on the data, the p-value is already wrong.
  • Subsets of data: Researchers check if there is a significant result for a subset of their data. The ISIS-2 trial (1988) points out this error, warning in their analysis that aspirin appears not to work for patients with the astrological signs Gemini or Libra. This can be partially fixed by interaction tests, but "Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials" Wallach et al. (2017) found only 46 of 117 subgroup claims made in papers were backed by a significant interaction test, and only 5 of those 46 were ever tested again, none of which replicated.
  • Optional stopping: Researchers can watch the results as they run a trial and stop if the result is significant. "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant" Simmons et al. (2011) showed that checking after every 10 participants and stopping at a significant effect will substantially raise the false-positive rate.
  • Statistical methods: Researchers can use different statistical methods on the same data until one of them produces significant results. "Variability in the analysis of a single neuroimaging dataset by many teams" Botvinik-Nezer et al. (2020) gave the same fMRI data and the same 9 hypotheses to 70 teams. No two teams chose the same workflow, and their conclusions varied substantially. "Many Analysts, One Data Set" Silberzahn et al. (2018) found the same pattern with 29 teams on a question about referee bias.
• Selective reporting of runs: Researchers repeat an experiment multiple times and report only the best run. In ML we see this as cherry-picking random seeds. For example, in "Deep Reinforcement Learning that Matters" Henderson et al. (2018), they showed that two runs of 5 random seeds produced significantly different results (and at the time, it was not uncommon to use fewer than 5 seeds). The same failure appears in AI researchers. In "Automated alignment runs are hard to study" Aristizabal et al. (2026, LessWrong), Claude agents noticed their LLM judge provided a noisy reward, so they began "lottery-farming", resubmitting the exact same work on a loop until it got a high score.
• HARKing: "HARKing: Hypothesizing After the Results are Known" Kerr (1998) coined the term HARKing to describe researchers who Hypothesize After Results are Known. This is not p-hacking exactly, but it hides the fact that the result was exploratory rather than confirmatory, making it impossible to account for the statistical differences between the two types of findings.
• Outcome switching: Researchers change which outcome is the primary one after seeing the data. "Empirical evidence for selective reporting of outcomes in randomized trials" Chan et al. (2004) compared 102 trial protocols to their publications and found 62% had at least one primary outcome changed, introduced, or omitted.
• Low-powered studies: Low-power experiments are noisy, so the results that pass the filter are more inflated and more likely to be false. "Power failure: why small sample size undermines the reliability of neuroscience" Button et al. (2013) estimated the median statistical power in neuroscience at around 21%.

All of these mechanisms filter results (intentionally or not) so only the lucky ones are seen, causing results to be exaggerated or altogether false.

This gets worse when people use low-power experiments, the base rate of tested false hypotheses increases, researchers sample hypotheses/experiments/data/subsets/methods more frequently, or we accept a higher p threshold.

As we transition to AI researchers, I expect the File Drawer Problem to get much worse. AI can run many more experiments and variations of experiments, look through more data, and therefore publish more false positives, at least in computational fields such as AI safety. There are known solutions to these issues; the challenge is building those solutions into infrastructure. Proposals have been made for re-running promising results, requiring a lower p value, labeling results as confirmatory or exploratory with the number of experiments it took, pre-registering experiments, publishing the null results, and/or requiring independent replications. It should also become easier to count the null results and identify when a researcher made a decision in the garden of forking paths, as AI researchers produce logs containing their every action and thought, which might be submitted alongside research or just used by the researcher to look back and identify null results and decisions. While these solutions can and should be implemented, their success will depend on whether they are incentivized, which we consider next.

Publish or perish

Researchers are incentivized to publish as early as possible and spend the smallest amount of effort to publish, sometimes going as far as fabricating work. Often attributed to the "publish-or-perish" incentives in academia, researchers will optimize for the metric they are judged on, which is typically citation count and number of publications.

Many of the issues in this document are downstream of incentives, but this section is specifically for issues stemming directly from the "publish-or-perish" incentives that most researchers operate under.

• Novelty bias: Newer work generates more attention, so replications are under-explored. "Replications in Psychology Research: How Often Do They Really Occur?" Makel et al. (2012) searched the full publication history of the top 100 psychology journals and estimated that 1.07% of publications were replications.
• Positive-result bias: Positive results receive more attention, so negative results are not published. This is the incentive behind the File Drawer Problem. "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy" Turner et al. (2008) compared FDA records to the published literature: 94% of published antidepressant trials were positive, while the FDA judged only 51% of the same trials positive. Most negative trials were either never published or published in a way that made them look positive. "An Excess of Positive Results" Scheel et al. (2021) found 96% of standard psychology papers supported their first hypothesis, compared to 44% of Registered Reports, where the journal commits to publish before results are known.
• Exaggeration: The claims in the abstract, discussion, or press release are stronger than the results support. "Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes" Boutron et al. (2010) found exaggeration in the abstract conclusions of 58% of trials whose primary outcome was not significant. "The association between exaggeration in health related science news and academic press releases" Sumner et al. (2014) traced most exaggeration in health news back to the universities' own press releases.
• Not building on existing evidence: New studies are started without checking what is already known, so settled questions are studied multiple times. "A Systematic Examination of the Citation of Prior Research in Reports of Randomized, Controlled Trials" Robinson & Goodman (2011) examined 1,523 trials across 227 meta-analyses and found the median trial cited 21% of the relevant prior trials, and 23% of trials with five or more prior trials available cited none. "Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding?" Fergusson et al. (2005) showed that placebo-controlled trials of aprotinin in cardiac surgery continued for years after a cumulative meta-analysis would have shown its effect was already established.
• Conflicts of interest: Researchers funded by someone with a stake in the result are more likely to report results that favor the sponsor. "Financial Conflicts of Interest and Reporting Bias Regarding the Association between Sugar-Sweetened Beverages and Weight Gain" Bes-Rastrollo et al. (2013) found reviews with conflicts of interest were five times more likely to present a conclusion that favored the sponsor. In medicine, "Industry sponsorship and research outcome" Lundh et al. 2017 is a review of 75 meta-research papers covering drug and device studies, finding industry-sponsored studies were about a third more likely to reach conclusions favorable to the sponsor.
• Short-termism: It can be easier to generate more collective citations and publications if you focus on research questions that are expected to take a short amount of time to answer. This can leave many important questions in a field unanswered if they are expected to take too long to answer.

The publish-or-perish incentives are primarily driven by universities and how they gate career advancement, so it is unclear how this will change over time. The incentives could get worse if universities (and potential employers) focus on any single metric, such as the publication number or citation count, to determine a researcher's success.

What incentives AI researchers will have is deeply unclear, so all predictions in this section are extremely low confidence. I think this also makes it a strong area for future work. I expect the percent of replications to moderately decrease, as there are no incentives (other than government grants) to do replications, and since the rate of publications will increase while the rate of grants will likely stay the same, the total percent of publications being replicated will decrease. I expect exaggeration to increase modestly, as AI has a tendency to overgeneralize people's work (Peters & Chin-Yee 2025). There will be more conflicts of interest as AIs have a tendency towards sycophancy and it will be easier for companies to spin up their own AI researchers. We should expect short-termism to initially increase as AI is better at short-term tasks, but this will improve as capabilities improve (METR 2026). AI should be better at finding existing evidence, as AI is faster and more capable of doing literature reviews compared to humans.

In the extreme case these incentives can lead to misconduct, which we look at next.

Misconduct

Most of this document is about unintentional failures, but some are deliberate.

• Fabrication: Results are sometimes modified to fit a claim, or made up altogether. "How Many Scientists Fabricate and Falsify Research?" Fanelli (2009) pooled surveys and found about 2% of scientists admitted to fabricating or falsifying data at least once, and about 14% reported knowing a colleague who had. The Dutch National Survey on Research Integrity (Gopalakrishna et al. 2022) found 4.3% admitted to fabrication.
• Image manipulation: Figures such as western blots or microscopy images are duplicated or edited to show a result that was not observed. "The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications" Bik, Casadevall & Fang (2016) screened 20,621 papers and found 3.8% contained problematic images, at least half suggestive of deliberate manipulation. The Nature paper "A specific amyloid-β protein assembly in the brain impairs memory" Lesné et al. (2006) about the Alzheimer's protein Aβ*56, cited thousands of times, was retracted in 2024 after image manipulation was exposed.
• Paper mills: Papers are produced in bulk and submitted to journals where editors and reviewers are bribed; academics can then purchase authorship on papers guaranteed to be published. This is primarily motivated by the requirement to have a minimum number of publications for career advancement in academia. The publisher Hindawi retracted over 8,000 papers, mostly from special issues infiltrated by paper mills (Retraction Watch 2023).
• Citation cartels: Groups of researchers or journals cite each other to inflate their citation counts. In 2013 Thomson Reuters suspended several journals from its rankings for "citation stacking" (Van Noorden 2013).

These issues get worse with less oversight by the community, or with more coordination by those conducting the misconduct. This is also a downstream effect of the incentives mentioned above, so we should expect misconduct to get worse if those incentives get stronger.

Assuming we still do not replicate experiments, I suspect AI researchers will cause fabricated results to become more common and paper mills more prevalent. This is because the ability to create credible-looking fraudulent papers will increase, academics will face increased competition from the increased number of AI researchers, and it will be easier to create citation cartels since new papers and authors can be fabricated. This can be mitigated by incentivizing replications and quality peer reviews, as long as the replications are themselves not also fabricated, which I believe is possible to do through a redesign of the ecosystem's infrastructure, such as randomly assigning replications.

Honest errors

Some results are wrong simply because someone made a mistake and no one caught it.

• Code bugs: Analysis code contains a bug that changes the result. Geoffrey Chang retracted five papers in 2006, including three in Science, because his analysis software flipped two columns of data, inverting the protein structures he reported (Miller 2006).
• Corrupted data: Formulas cover the wrong range, or the software silently changes the data. "Gene name errors are widespread in the scientific literature" Ziemann et al. (2016) found about 20% of genomics papers with Excel supplementary gene lists had gene names converted into dates, such as SEPT2 becoming 2-Sep. Reinhart & Rogoff's influential finding that high government debt slows growth was found to partly depend on an Excel formula that left out several countries (Herndon et al. 2014).
• Statistical reporting errors: Reported numbers are inconsistent with each other. "The prevalence of statistical reporting errors in psychology (1985–2013)" Nuijten et al. (2016) found roughly half of psychology papers using significance tests contain at least one p-value inconsistent with its test statistic.
• Misused tests: A statistical test is used incorrectly. A common one is claiming two effects differ because one is significant and the other is not, as described in "The Difference Between 'Significant' and 'Not Significant' is not Itself Statistically Significant" Gelman & Stern (2006). "Erroneous analyses of interactions in neuroscience" Nieuwenhuis et al. (2011) found about half of neuroscience papers that could have made this error did make it.

Everyone makes mistakes, so these errors only get caught when time is dedicated to looking for those mistakes. This means we can expect these to get worse if the quality of research goes down or the time spent looking for mistakes goes down.

AI researchers should minimize most of these issues. It does not cost an AI much to spend extra time looking for bugs or corrupted data, and its ability to avoid these issues in the first place will improve with capabilities. This does depend on whether there is an incentive to catch bugs, which should be true as long as retractions are treated more seriously. However, if AI researchers continue to reward hack, they also have an incentive to ignore or intentionally add bugs which support their conclusions. This may also compound with the correlated errors we consider in Verification failure.

Experimental design

Even if something is measured in good faith, the code has no bugs, and it replicates, you may not be measuring what you think you are measuring.

This is a deep topic and essentially the study of experimental design, so we only list a few of the known failures here. The failures below are organized using the four types of validity from Experimental and Quasi-Experimental Designs for Generalized Causal Inference Shadish, Cook & Campbell (2002).

• Statistical conclusion validity: Is the association between the variables real? Most failures of this kind are covered in the Statistical selection section above.
  • Measurement error: A researcher measures the right thing, but with noise. Noise is often assumed to only make effects harder to find, but combined with a significance filter it inflates them. This is well described in "Measurement error and the replication crisis" Loken & Gelman (2017).
  • Circular analysis: The same data is used to select what to analyze and then to test it. For example, choosing the brain regions that respond most to a stimulus and then reporting how strongly they respond. "Circular analysis in systems neuroscience: the dangers of double dipping" Kriegeskorte et al. (2009) surveyed 134 fMRI papers published in 2008 across top journals and found 42% contained at least one circular analysis.
• Internal validity: Is the association causal?
  • Randomization: Randomized assignment to a control group fails when there is no proper control or when researchers can predict or influence who receives the treatment. "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research" Ioannidis (2005) found that of 6 highly cited nonrandomized studies, 5 were later contradicted or found to have exaggerated effects, compared to 9 of 39 randomized trials. Even within randomized trials, "Empirical evidence of bias" Schulz et al. (1995) examined 250 trials from 33 meta-analyses and found that trials with inadequately concealed allocation exaggerated the estimated effect, with odds ratios 41% lower compared to trials with adequate concealment.
  • Leakage: The answer is already inside the test, so the experiment cannot fail. "Leakage and the reproducibility crisis in machine-learning-based science" Kapoor & Narayanan (2023) found leakage in 294 papers across 17 fields. In some examples, the apparent advantage of ML models over logistic regression disappeared once leakage was fixed.
  • Weak baselines: A new method is compared against poorly tuned or outdated alternatives, so the improvement is inflated. "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" Dacrema et al. (2019) examined 18 recommendation algorithms from top conferences. Only 7 could be reproduced, and 6 of those were often beaten by simple, older heuristics. "A Metric Learning Reality Check" Musgrave et al. (2020) found the same pattern in metric learning.
• Construct validity: Do the measures and manipulations capture the concepts they are meant to?
  • Invalid measures: The measurement does not capture the thing you care about; it is a proxy. "Measuring what Matters: Construct Validity in Large Language Model Benchmarks" Bean et al. (2025) reviewed 445 LLM benchmarks and found widespread weaknesses in how they defined the phenomenon, the task, the metric, or the claims drawn from it. "Measurement Schmeasurement" Flake & Fried (2020) makes the same argument for psychology.
  • Subject reactivity: What is being studied behaves differently because it can tell it is being measured. People and LLMs are known to do this. The Hawthorne effect is the classic name for people, although a "Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects" McCambridge et al. (2014) concluded there is no single Hawthorne effect and little is known about its size or conditions. "Large Language Models Often Know When They Are Being Evaluated" Needham et al. (2025, preprint) showed frontier models can often distinguish evaluations from real use.
  • Experimenter expectancy: The experimenter's beliefs leak into the result, often because the experiment was not blinded. "Observer bias in randomised clinical trials with binary outcomes" Hróbjartsson et al. (2012) studied trials that had both blinded and non-blinded assessors of the same outcome and found non-blinded assessors exaggerated odds ratios by 36% on average, and a later update found 29% (Salazar et al. 2025). One funny example is Clever Hans, a horse that appeared to do arithmetic but was instead reading unconscious cues from his questioners (Pfungst 1911).
  • Unfalsifiable hypotheses: The hypothesis is flexible enough that any result can be read as support, so the experiment cannot fail. Karl Popper used psychoanalysis as his example, and "Theoretical Risks and Tabular Asterisks" Meehl (1978) argued much of soft psychology had the same problem.
• External validity: Does the result generalize beyond the people, settings, and conditions studied?
  • Unrepresentative samples: Research is conducted with many hidden variables, some of which will influence the result. This is common, for example, when subjects in an experiment are all college students, but the paper generalizes the result to all people. "The weirdest people in the world?" Henrich et al. (2010) reported that 96% of psychology samples came from Western industrialized countries holding 12% of the world's population, and showed that results often differ outside those populations.

There are a number of ways in which experimental design can fail, and the best way to minimize the failures is to intentionally be as careful as possible, checking over known failure modes and following well-established methods when possible.

I suspect AI researchers will make these problems moderately worse. We have yet to create AI which follows the "spirit" of a reward rather than the reward directly. This means if we rely on checklists for improving experimental design, then those items on the checklist will improve while anything we forget to include will be reward hacked. The fundamental challenge is that experimental design does not have an objective measure we can use to quantify its quality. One benefit of AI researchers is that they can iterate over a checklist to compare against a design, and this checklist can grow over time to catch more and more errors, until eventually most designs will not be low-quality. This assumes the number of failure modes does not grow as science grows, and that errors are caught at some point rather than persisting forever. So while experimental design may initially be a problem, I expect in the long term it will slowly improve due to a growing checklist and improved capabilities.

A lot of these errors could be caught by review, so we look next at some of the failure modes in science around verification.

Verification failure

There is a well-known crisis in academia where most papers are not replicated, which allows false claims to spread unchallenged. Even when someone does carry out a replication and they find errors that falsify the original paper, the field does not update and often even continues to cite the original result as if it had never been falsified.

• Won't replicate: Most work is never checked. See Makel et al. (2012) under Novelty bias, and Wallach et al. (2017) under Subsets of data.
• Can't access: Results often cannot be reproduced due to lack of access to the source data or code. "An empirical analysis of journal policy effectiveness for computational reproducibility" Stodden et al. (2018) requested data and code for 204 Science papers published after the journal required authors to share on request. They obtained materials for 44% and could reproduce the findings for 26%.
• Can't reproduce: Even when researchers are given access, the paper may still not replicate. This is often because papers will leave out important details. "Reproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology" Errington et al. (2021) planned to repeat 193 experiments from 53 high-impact papers. None were described in enough detail to design the replication without contacting the original authors, and the project completed only 50.
• Shallow peer review: Review is costly and not incentivized, so it is done quickly and with little effort. This allows mistakes in papers to go unchallenged and get published. "Effect on the Quality of Peer Review of Blinding Reviewers and Asking Them to Sign Their Reports" Godlee et al. (1998) inserted eight weaknesses into an already accepted manuscript and sent it to 420 reviewers. The reviewers who responded commented on only two weaknesses on average, and none found more than five.
• Noisy peer review: Near the acceptance threshold, review outcomes are close to random, which adds another lottery filter on top of statistical significance. "Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown?" Beygelzimer et al. (2023) had 10% of NeurIPS submissions reviewed by two independent committees and found that the committees disagreed on 23% of papers, and as in 2014, about half the accepted papers would change if review were rerun. "Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment" Cortes & Lawrence (2021) found that for accepted papers, reviewer quality scores did not correlate with later citations.
• Unequal scrutiny: Results that match expectations are checked less carefully than results that do not (confirmation bias). "Publication prejudices: An experimental study of confirmatory bias in the peer review system" Mahoney (1977) sent 75 reviewers manuscripts with identical methods but different results, and reviewers were strongly biased against manuscripts whose results contradicted their own theoretical views.
• Prestige bias: Prestigious authors and institutions are not scrutinized as intensely as others. "Nobel and novice: Author prominence affects peer review" Huber et al. (2022) sent the same finance paper to over 3,300 potential reviewers, randomizing which author was shown. Reviewers recommended rejection 23% of the time when a Nobel laureate was the named author, 48% when the paper was anonymized, and 65% when an unknown early-career researcher was named. "Reviewer bias in single- versus double-blind peer review" Tomkins et al. (2017) found the same pattern in a controlled experiment.
• Ignored retractions: A retraction or failed replication is mostly ignored. The tools for alerting researchers about retractions are opt-in and not frequently used, so false claims continue to propagate. Papers will in fact often continue to be cited even after they have been retracted. "Nonreplicable publications are cited more than replicable ones" Serra-Garcia & Gneezy (2021) looked at 80 papers from three large replication projects (psychology, economics, and Nature/Science) and found papers that failed to replicate were cited about 153 more times than papers that replicated. The gap did not shrink after the failed replication was published, and only 12% of later citations mentioned it. "Citation patterns following a strongly contradictory replication result" Hardwicke et al. (2021) followed four high-profile failed replications in psychology, including ego depletion and facial feedback, and found only a modest drop in favorable citations. For ego depletion, 79% of citations to the original were favorable before the replication and 77% after. As an example, "Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data" Schneider et al. (2020) followed one clinical trial retracted in 2008 for falsified data and found it was still being cited positively 11 years later, usually without mention of the retraction.

Verification through peer review is one of the most important tools we have in the scientific ecosystem, so it is critical that we value it and build infrastructure to support it. There are many ways in which peer review can (and currently does) fail to serve its purpose. This gets worse if we raise the cost of replication, lower the reward for review, or avoid costs for poor reviews.

AI researchers have the potential to alleviate these concerns, but under the existing incentives and infrastructure, they will likely instead make it worse. There is no incentive to replicate experiments or to make work easy to reproduce, or to invest time into peer review. Since many AI researchers will be instances of the same models, we can expect most of the peer reviews to be highly correlated, and it has even been found that different models have highly correlated errors (Kim et al. 2025). Since the paper will be written by a model similar to the ones conducting the peer review, we should expect the author to have similar blind spots to the reviewers. We will need to be careful of results that appear to initially improve; we may have just traded noisy errors for correlated errors. The one issue that may improve is retractions, as AI researchers can cheaply check every citation, so papers are less likely to be built on retracted work. The replication and peer review issues can be partially solved with requirements set by journals, such as requiring replication as a part of the desk-rejection process, but how to fix the correlated errors is not clear. It costs very little for AI to write documentation and see if a sub-agent can replicate their project, so the cost of enabling replications should drop dramatically. AI researchers can enable replication and review to be scaled up for minimal cost, if it is incentivized in the ecosystem and a solution to correlated errors can be found.

If peer reviews and replications are not functioning well, then the scientific ecosystem will become contaminated with false statements. This effect gets worse as new work builds on previous unverified work, and the issues compound. This is the type of failure we consider next.

Compounding effects

As publications get more citations, they are more likely to be seen and cited in the future. This is one example of a feedback loop in science, but there are many, and they can have compounding effects which can be long-lasting and wide-reaching.

• Attention collapse: Researchers pay attention to whatever others are paying attention to, so the field can over-invest in one idea and fail to explore the full space of ideas. "Slowed canonical progress in large fields of science" Chu & Evans (2021) analyzed 1.8 billion citations across 241 subjects and found that as fields grow, citations concentrate on already well-cited papers, the most-cited list stops changing, and new papers rarely break through.
  • Matthew effect: Work that is already cited is more likely to be seen and cited again, whatever its quality, as described in "The Matthew Effect in Science" Merton (1968). "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market" Salganik et al. (2006) showed the same mechanism in an artificial music market: when people could see download counts, success became more unequal and less predictable from quality.
• Citation distortion: A claim gets stronger as it is passed along, with supportive papers cited and critical ones ignored. "How citation distortions create unfounded authority" Greenberg (2009) mapped the citation network of 242 papers about one specific claim (β amyloid is produced by and injures skeletal muscle of patients with inclusion body myositis). They found 94% of citations to primary data went to the four papers containing primary data supporting the claim, while only 6% of citations went to the six papers with primary data refuting the claim. Review papers with no new data on the claim amplified the support for it, and in some papers a hypothesis became a fact through citation alone.
• Cited without reading: Researchers cite papers they have not read, often copying the citation from another paper. "Read before you cite!" Simkin & Roychowdhury (2003) used misprints that repeated across citations to estimate that only about 20% of citers read the original.
• Copy errors: Experiments are copied (especially code) and re-used by other researchers for new experiments, but the copied experiment can contain errors that then propagate. "A Bug in the Multiobjective Optimizer IBEA" Brockhoff et al. (2015) found a bug in a benchmark algorithm that had been copied into other software packages, leading multiple studies to conclude the variant performed worse than it does.
• Paradigm lock-in: Researchers will use the same methods that initial papers used, creating a standard, even when those methods could be improved. The same dynamic makes fields resist new ideas that contradict the standard view. Marshall and Warren's finding that ulcers are caused by the bacterium H. pylori was met with years of skepticism before it won the 2005 Nobel Prize. "Does Science Advance One Funeral at a Time?" Azoulay et al. (2019) found that after star scientists die unexpectedly, outsiders publish more in their fields.
  • Benchmark overfitting: Once a benchmark becomes the standard in a field, papers optimize for it until it stops reflecting real progress (Goodhart's law). Machine translation adopted the BLEU metric in the early 2000s and it became the field standard. "Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers" Marie et al. (2021) found 98.8% of MT papers from 2010 to 2020 reported BLEU, and 82.1% of papers from 2019 and 2020 relied on BLEU alone, despite at least 108 proposed metrics claiming to outperform it. "To Ship or Not to Ship" Kocmi et al. (2021) compared metrics against 2.3 million human judgments and found BLEU agreed with humans on 74.6% of pairwise system rankings, compared to 83.4% for COMET (an alternative), concluding that the sole use of BLEU impeded the development of improved models.
• Correlated data: The same datasets, software, and experimental designs are used by researchers, so some errors are correlated across experiments that are treated as independent. "Cluster failure" Eklund et al. (2016) ran 3 million analyses on null fMRI data and found the three most common analysis packages could produce false-positive rates of up to 70% instead of the nominal 5%, affecting any study that relied on them. "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks" Northcutt et al. (2021) estimated label errors averaging at least 3.3% across 10 widely used ML test sets.
• Biased meta-analysis: Meta-analyses pool a filtered literature, so they inherit its biases while appearing more authoritative. "Comparing meta-analyses and preregistered multiple-laboratory replication projects" Kvarven et al. (2020) found meta-analytic effects differed significantly from large preregistered replications in 12 of 15 cases and were almost three times as large on average. Standard corrections for publication bias did not fix the gap. As an example, one meta-analysis (Hagger et al. 2010) found ego depletion had a medium-to-large effect on self-control, but a follow-up replication using preregistered experiments across 23 labs (Hagger et al. 2016) found the effect was actually near zero.

Much of the scientific ecosystem is built on compounding work: papers building off papers, ideas being disseminated to inspire new ideas, tools open-sourced so other researchers can move faster. But that makes the system susceptible to compounding errors too. There are few error-correcting mechanisms outside peer review and replications, so the impact of compounding errors primarily depends on how often errors are introduced (most of the issues mentioned in this document) and how strong the error-correcting mechanisms are (peer review and replication).

I expect AI researchers to mitigate some of these issues while amplifying others. AI researchers will be able to read every single citation in a paper and verify it supports the claim it was cited for. But I expect they will have issues with correlated data, since if most AIs are the same model, they will all "independently" choose the same datasets, methods, and tools. I suspect there will be an increase in attention collapse since there will be a larger number of researchers, and positive feedback loops tend to grow more unequal as the number of participants grows. There will also be an additional novel pressure on the Matthew effect through publications being included in model training data, which makes models more likely to know about and cite popular work. It seems likely that paradigm lock-in will initially get worse as models are all the same and inherit perspectives and training data from their previous models (people switching to a new model is like a funeral, but the new models are not as distinct from model to model as people are from funeral to funeral), and then in the long term the models should discover new paradigms as capabilities improve. Biased meta-analyses will likely not change much, but they could be replaced with stronger meta-analyses which replicate every paper they analyze. Copy errors could be a real issue if there are more low-quality researchers, but this issue will diminish with capabilities, and can easily be minimized by dedicating some time to looking for bugs in any code the researcher uses.

Read more

While AI researchers have the potential to accelerate progress in all fields, there is also the possibility they break existing infrastructure, especially at the weak points considered in this post. This was not an exhaustive list, so below are more readings which attempt to catalog the failures of science.

1. "A manifesto for reproducible science" Munafò et al. (2017): maps threats to reproducibility onto stages of the research cycle (design, collection and analysis, publication) and proposes fixes in five areas: methods, reporting, reproducibility, evaluation, and incentives.
2. The Scientific Utopia series: "I. Opening scientific communication" Nosek & Bar-Anan (2012), "II. Restructuring incentives and practices to promote truth over publishability" Nosek et al. (2012), and "III. Crowdsourcing science" Uhlmann et al. (2019). Part II is the most relevant here: it argues the core failure is that incentives reward publishability rather than truth, and walks through the practices that follow and the reforms to counter them.
3. Reproducibility and Replicability in Science National Academies of Sciences, Engineering, and Medicine (2019): the US consensus report. It distinguishes reproducibility (same data, same result) from replicability (new data, same result) and reviews the sources of failure for each.
4. Science Fictions Ritchie (2020): a book organizing the failures of science into fraud, bias, negligence, and hype.
5. "Research: increasing value, reducing waste" Lancet series (2014): five papers covering waste at each stage of research, from setting priorities to reporting. It builds on "Avoidable waste in the production and reporting of research evidence" Chalmers & Glasziou (2009).
6. "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking" Wicherts et al. (2016): a detailed checklist of researcher degrees of freedom.
7. Catalog of Bias: an online catalog from the Oxford Centre for Evidence-Based Medicine, with a definition, examples, and evidence for each named bias.
8. "The natural selection of bad science" Smaldino & McElreath (2016): a model of how rewarding publication count spreads poor methods, even without deliberate cheating.
9. "Troubling Trends in Machine Learning Scholarship" Lipton & Steinhardt (2018): four problems in ML papers: mixing explanation with speculation, misattributing the source of empirical gains, "mathiness," and misuse of language.

Acknowledgements

Thanks to the Cambridge Boston Alignment Initiative (CBAI) for providing funding, guidance, and office space for work and community.

Discuss

The kids have been excited about Pig Latin a lot lately. I noticed that there are a few words that are also English words, or homophones thereof. With some help from Claude, especially in figuring out which words would work well and in checking my work, here's a poem about a pie and a naughty plover:

I weigh ache bay each pay I pay,
Oh say eats whey, each way ice lei,
Away ate play eye way a prey
I'll weigh eat whey all way a day.

Oh whey, oh neigh! I may I pay!
Away outstay, overplay ate whey
All whey of whey, I may I pay!
Ooh, yay, I slay eat tray outlay!

I weigh, eat whey, all whey, ashtray,
I weigh, underplay, all whey, eyes pay!
Eat sway, eat stray, I weigh eat whey!
Oh say, Old Bay? Owe whey, I may!

I weigh Amway oh say outstay!
I weigh I flay? Oh neigh, away!
I weigh a stay, I weigh I lay,
I weigh owes day all whey a day!

reveal English... I bake peach pie
So sweet each slice
A plate I pray
I'll eat all day.

Oh, no! My pie!
A stout plover ate
All of my pie!
You sly treat lout!

I eat all trash,
I plunder all pies!
Sweet treats I eat!
So bold? Oh my!

I am so stout!
Why fly? No way!
I stay, I lie,
I doze all day.

Note that this doesn't quite work in my dialect, since "way" and "whey" are not homophones, but since it works for almost all of my readers I liked it better with "whey".

Discuss

Kola Ayonrinde, Anthropic Fellows, koayon@gmail.com; Jack Lindsey, Anthropic

TL;DR: The J-Lens was proposed to read verbalisable representations from language model activations. We introduce the J++ Lens: an improvement to the J-Lens that enables more faithful readouts across 9–284B models. The J++ Lens filters noisy gradients before they enter the averaged Jacobian. Using the J++ Lens as a drop-in replacement for the J-Lens, we read workspace representations with 55% success on latent variable extraction tasks (J-Lens: 36%, R-Lens: 38%) with especially strong performance at early layers. We believe that more faithful lenses can be useful in monitoring for eval awareness and unverbalised scheming. We release code, an interactive demo and open-source lenses.

Figure 1. By filtering out noisy gradients, the J++ Lens reads intermediate variables more reliably (a) and at earlier layers (b) than the J-Lens and R-Lens. The J++ Lens improves on the J-Lens by adding: (1) mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mtable { display: inline-block; text-align: center; vertical-align: .25em; position: relative; box-sizing: border-box; border-spacing: 0; border-collapse: collapse; } mjx-mstyle[size="s"] mjx-mtable { vertical-align: .354em; } mjx-labels { position: absolute; left: 0; top: 0; } mjx-table { display: inline-block; vertical-align: -.5ex; box-sizing: border-box; } mjx-table > mjx-itable { vertical-align: middle; text-align: left; box-sizing: border-box; } mjx-labels > mjx-itable { position: absolute; top: 0; } mjx-mtable[justify="left"] { text-align: left; } mjx-mtable[justify="right"] { text-align: right; } mjx-mtable[justify="left"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="left"][side="right"] { padding-left: 0 ! important; } mjx-mtable[justify="right"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="right"][side="right"] { padding-left: 0 ! important; } mjx-mtable[align] { vertical-align: baseline; } mjx-mtable[align="top"] > mjx-table { vertical-align: top; } mjx-mtable[align="bottom"] > mjx-table { vertical-align: bottom; } mjx-mtable[side="right"] mjx-labels { min-width: 100%; } mjx-mlabeledtr { display: table-row; text-align: left; } mjx-mlabeledtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mlabeledtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mlabeledtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mlabeledtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mlabeledtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtr { display: table-row; text-align: left; } mjx-mtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtd { display: table-cell; text-align: center; padding: .215em .4em; } mjx-mtd:first-child { padding-left: 0; } mjx-mtd:last-child { padding-right: 0; } mjx-mtable > * > mjx-itable > *:first-child > mjx-mtd { padding-top: 0; } mjx-mtable > * > mjx-itable > *:last-child > mjx-mtd { padding-bottom: 0; } mjx-tstrut { display: inline-block; height: 1em; vertical-align: -.25em; } mjx-labels[align="left"] > mjx-mtr > mjx-mtd { text-align: left; } mjx-labels[align="right"] > mjx-mtr > mjx-mtd { text-align: right; } mjx-mtd[extra] { padding: 0; } mjx-mtd[rowalign="top"] { vertical-align: top; } mjx-mtd[rowalign="center"] { vertical-align: middle; } mjx-mtd[rowalign="bottom"] { vertical-align: bottom; } mjx-mtd[rowalign="baseline"] { vertical-align: baseline; } mjx-mtd[rowalign="axis"] { vertical-align: .25em; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mrow { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mn { display: inline-block; text-align: left; } mjx-munderover { display: inline-block; text-align: left; } mjx-munderover:not([limits="false"]) { padding-top: .1em; } mjx-munderover:not([limits="false"]) > * { display: block; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-stretchy-v.mjx-c7C mjx-ext mjx-c::before { content: "\2223"; width: 0.333em; } mjx-c.mjx-c1D409.TEX-B::before { padding: 0.686em 0.594em 0.011em 0; content: "J"; } mjx-c.mjx-c1D41A.TEX-B::before { padding: 0.453em 0.559em 0.006em 0; content: "a"; } mjx-c.mjx-c1D41C.TEX-B::before { padding: 0.453em 0.511em 0.006em 0; content: "c"; } mjx-c.mjx-c1D428.TEX-B::before { padding: 0.452em 0.575em 0.005em 0; content: "o"; } mjx-c.mjx-c1D41B.TEX-B::before { padding: 0.694em 0.639em 0.006em 0; content: "b"; } mjx-c.mjx-c1D422.TEX-B::before { padding: 0.695em 0.319em 0 0; content: "i"; } mjx-c.mjx-c1D427.TEX-B::before { padding: 0.45em 0.639em 0 0; content: "n"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c1D405.TEX-B::before { padding: 0.68em 0.724em 0 0; content: "F"; } mjx-c.mjx-c1D425.TEX-B::before { padding: 0.694em 0.319em 0 0; content: "l"; } mjx-c.mjx-c1D42D.TEX-B::before { padding: 0.635em 0.447em 0.005em 0; content: "t"; } mjx-c.mjx-c1D41E.TEX-B::before { padding: 0.452em 0.527em 0.006em 0; content: "e"; } mjx-c.mjx-c1D42B.TEX-B::before { padding: 0.45em 0.474em 0 0; content: "r"; } mjx-c.mjx-c1D420.TEX-B::before { padding: 0.455em 0.575em 0.201em 0; content: "g"; } mjx-c.mjx-c1D40B.TEX-B::before { padding: 0.686em 0.692em 0 0; content: "L"; } mjx-c.mjx-c1D411.TEX-B::before { padding: 0.686em 0.862em 0.011em 0; content: "R"; } mjx-c.mjx-c1D40F.TEX-B::before { padding: 0.686em 0.786em 0 0; content: "P"; } mjx-c.mjx-c1D41D.TEX-B::before { padding: 0.694em 0.639em 0.006em 0; content: "d"; } mjx-c.mjx-c1D42E.TEX-B::before { padding: 0.45em 0.639em 0.006em 0; content: "u"; } mjx-c.mjx-c1D432.TEX-B::before { padding: 0.444em 0.607em 0.2em 0; content: "y"; } mjx-c.mjx-c2D.TEX-B::before { padding: 0.278em 0.383em 0 0; content: "-"; } mjx-c.mjx-c1D430.TEX-B::before { padding: 0.444em 0.831em 0 0; content: "w"; } mjx-c.mjx-c1D42C.TEX-B::before { padding: 0.453em 0.454em 0.006em 0; content: "s"; } mjx-c.mjx-c1D42F.TEX-B::before { padding: 0.444em 0.607em 0 0; content: "v"; } mjx-c.mjx-c1D429.TEX-B::before { padding: 0.45em 0.639em 0.194em 0; content: "p"; } mjx-c.mjx-c28.TEX-B::before { padding: 0.75em 0.447em 0.249em 0; content: "("; } mjx-c.mjx-c29.TEX-B::before { padding: 0.75em 0.447em 0.249em 0; content: ")"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2C6.TEX-S1::before { padding: 0.744em 0.556em 0 0; content: "\2C6"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D53C.TEX-A::before { padding: 0.683em 0.667em 0 0; content: "E"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c5B.TEX-S3::before { padding: 1.45em 0.528em 0.949em 0; content: "["; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c1D715::before { padding: 0.715em 0.566em 0.022em 0; content: "\2202"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c5D.TEX-S3::before { padding: 1.45em 0.528em 0.949em 0; content: "]"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D439.TEX-I::before { padding: 0.68em 0.749em 0 0; content: "F"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c2D::before { padding: 0.252em 0.333em 0 0; content: "-"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c56.TEX-C::before { padding: 0.683em 0.658em 0.052em 0; content: "V"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c28.TEX-S2::before { padding: 1.15em 0.597em 0.649em 0; content: "("; } mjx-c.mjx-c29.TEX-S2::before { padding: 1.15em 0.597em 0.649em 0; content: ")"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c2B.TEX-B::before { padding: 0.633em 0.894em 0.131em 0; content: "+"; } mjx-c.mjx-c4A::before { padding: 0.683em 0.514em 0.022em 0; content: "J"; } mjx-c.mjx-c46::before { padding: 0.68em 0.653em 0 0; content: "F"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2E.TEX-B::before { padding: 0.171em 0.319em 0 0; content: "."; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } : down-weighting the Jacobians from activations that result in poor readouts; (2) : applying Layer-wise Relevance Propagation to the backward pass when computing Jacobians; (3) : removing non-semantic tokens from the readout. (a) The J++ Lens surfaces the correct intermediate readouts at 55.2% recall@10, compared to 35.7% and 37.7% for the J-Lens and R-Lens respectively. (b) The J++ Lens surfaces semantically relevant intermediate latent variables at earlier layers than the J-Lens and R-Lens, layer 24 compared to layer 40 in this example. The intended journey is to hop from the prompt to “heart” to “four chambers”. Bright yellow shading denotes the intermediate variable appearing in the readout, and light yellow shading indicates a closely related readout.

Introduction

Language model activations contain a large amount of content, most of which is unnecessary for understanding the main latent variables used in internal reasoning. Gurnee et al. (2026) propose that activations can be understood as containing a small privileged set of workspace representations that sits alongside a larger set of non-workspace representations. The workspace component contains verbalisable representations and supports flexible internal reasoning, whereas the non-workspace component supports automatic processing. Because the workspace supports internal reasoning, this component might be a natural monitoring surface for analysing frontier models’ internal reasoning, for example in eval awareness (Needham et al. 2025) and unverbalised scheming (Carlsmith 2023).

Gurnee et al. (2026) introduce the Jacobian Lens (J-Lens) to read from and write to the workspace. The J-Lens extracts the tokens that the model is most disposed to verbalise, conditional on a given activation. Gurnee et al. (2026) show some evidence that the space spanned by the J-Lens vectors has workspace-like qualities: it is used disproportionately for complex inferences rather than factual recall or writing fluent prose, and it contains intermediate variables used in multi-step reasoning. For example, when given the prompt “Fact: The number of legs on the animal that spins webs is” and asked to predict the next token, the lens surfaces the tokens “spider” and “legs” in intermediate layers before the model predicts “8”. Patching the J-Lens vector “ant” in place of “spider” in the relevant intermediate layer flips the model’s response from 8 to 6, indicating the causal importance of the workspace vectors in internal multi-hop reasoning. However, while the J-Lens does surface intermediate variables, it tends to give coherent readouts only in the latter half of layers. Additionally, many intermediate variables that we would expect in the workspace are not surfaced by the J-Lens at all.

The failure of the J-Lens to extract intermediate latent variables could have many explanations: (i) the hypothesised workspace is a poor model of language model cognition; (ii) the workspace lacks a linear representation of these variables; (iii) the model does not represent them at all (in the workspace or non-workspace components); [1] or (iv) the J-Lens is an imperfect tool for capturing workspace content. We provide evidence for (iv), showing that a substantial fraction of intermediate variables can be linearly extracted. We show that the J-Lens accumulates noisy gradients, corrupting the averaged Jacobian. Further, we show that filtering out Jacobians from a subset of the activations increases lens performance.

We introduce the J++ Lens, which improves on the J-Lens with three changes: (1) : down-weighting Jacobians from clusters of activations that give poor readouts; (2) : computing Jacobians with an LRP backward pass, as in the R-Lens; (3) : removing non-semantic tokens (e.g. punctuation and whitespace) from the readout.

We say that a Workspace Lens is faithful to the extent that its readouts surface the intermediate variables that the model uses in its reasoning and its lens vectors act as those variables when patched into the model. By improving in the readouts while retaining causal importance, the J++ Lens is more faithful than prior Workspace Lenses. The J++ Lens extracts intermediate variables in latent variable extraction tasks more accurately and at earlier layers (Figure 1). These results replicate across six open-source models from 9B to 284B parameters (Table 1). The J++ Lens’ vectors also remain as causally important as those of the J-Lens (Figure 2c).

Because the J++ Lens reads the workspace more reliably and at earlier layers, we can use it to study LLM internal mechanisms. We find evidence that models sometimes suppress concepts through a retrieve-then-suppress mechanism: the model brings the concept into the workspace in early layers and causally uses this early-layer workspace representation to de-amplify the concept’s salience in later layers (see Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression).

Contributions. Our contributions are as follows:

• We show that noisy gradients corrupt the averaged Jacobian of prior Workspace Lenses (Methods and Why Does the J++ Lens Work?).

• We propose the J++ Lens, a Workspace Lens that filters noisy gradients to read from the workspace more faithfully. Across all six models we test (9B–284B parameters), it outperforms the J- and R-Lenses, with a median relative improvement of 64% over the J-Lens for reading out intermediate variables (Table 1). For writing to the workspace, J++ Lens vectors are at least as causally important as J-Lens vectors, changing the model’s output as intended in 59.7% of trials, against 49.3% for the J-Lens (The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations).

• We use the J++ Lens to generate and test mechanistic hypotheses, finding evidence that early workspace layers are sometimes causally important for how models suppress concepts (Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression; Figures 5 to 7).

To support researchers using Workspace Lenses, we release training code, an evaluation harness for readout and causal evals, and open-source lenses, which can be explored interactively in the browser on Neuronpedia. The J++ Lens can be used as a drop-in replacement for the J-Lens for surfacing intermediate variables, linear probing or precise interventions. Researchers may also use the J++ Lens as an initialisation for building multi-token Workspace Lenses such as the Template Lens or Oracle Lens.

Background

J-Lens. Gurnee et al. (2026) introduce the Jacobian Lens (J-Lens) as a tool for reading from and writing to the workspace, the set of verbalisable representations that a model uses as a working memory. A Workspace Lens is any method that, like the J-Lens, seeks to interpret only the workspace. The J-Lens surfaces the tokens that a model is increasingly disposed to verbalise at future token positions given an intermediate activation at layer and source position . Specifically, the J-Lens characterises an activation by its average first-order causal effect on the model’s output logits, using the averaged Jacobian where is the final-layer activation at target position and is the sequence length ( in our experiments). We may think of as approximating a “transport map” from the activation space at layer to that at the final layer (Belrose et al. 2023; Hernandez et al. 2023). After applying the transport map, the verbalisable tokens can be read off using the unembedding matrix . We call the rows of , corresponding to each vocabulary token, the lens vectors. Equation (1) weights every Jacobian equally, so noisy Jacobians enter the averaged Jacobian unfiltered. In Methods, we describe the J++ Lens, which reduces the noise that enters the averaged Jacobian. [2]

R-Lens. The R-Lens improves on the J-Lens by limiting error accumulation in the backward pass: this approach computes each Jacobian in Equation (1) with an LRP backward pass (Bach et al. 2015; Montavon et al. 2019) rather than the full gradient. We later show that combining LRP with other methods to reduce the noise accumulation in early-layer Jacobians leads to more faithful Workspace Lenses. Appendix B: Related Work discusses further related work.

Methods

When computing the averaged Jacobian in Equation (1), there are many potential sources of noise that can enter into the average and corrupt the final result. Averaging Jacobians reduces unbiased noise, which cancels out in expectation, but not biased noise. Skipping the first 16 positions of each sequence when computing the averaged Jacobian can avoid attention sinks, which may contribute outlier gradients that are one source of biased noise.

Reducing Noise in Workspace Lenses

We make three further changes to the J-Lens to reduce biased noise: , and . Together, they give the J++ Lens: where is a partition of the activation space at layer into clusters, are learned weights, denotes differentiation with the LRP backward pass, is the unembedding matrix and is the vocabulary with non-semantic tokens removed. Removing the changes (in colour) recovers the original J-Lens formulation of Equation (1).

Jacobian Filtering

Jacobian Filtering targets the noisy gradients that come from particular intermediate-layer source activations (). To isolate such source activations, we partition the activation space at each layer into clusters with -means clustering. We then fit one averaged Jacobian per cluster, using only the source activations in that cluster. We consider these Jacobians as Expert Jacobians that specialise to a subset of the activation space, inspired by mixture-of-experts (MoE) models (Shazeer et al. 2017).

Empirically, we find that some of these experts perform well applied to any intermediate extraction task, even for activations outside their own cluster. Other experts, however, routinely give poor readouts. We hypothesise that the geometry of the transport map around the activations behind these noisy Jacobians may be particularly singular or have high curvature, so that the Jacobian there is a poor approximation to the true transport map.

To avoid including these noisy Jacobians in our averaged Jacobian, we combine the Expert Jacobians into a single linear map by taking a weighted average rather than including all Jacobians equally. We learn the weights on a small dev set of latent variable extraction problems. [3] In practice, the weights mainly decide which experts are given zero (or negative) weight and which are included in the average (see Figure 12 in What Is in the Bad Experts?). The resulting map is a drop-in replacement for the J-Lens with no extra inference cost, but with noisy Jacobians effectively filtered out.

Layer-wise Relevance Propagation (LRP)

We follow Blank, Bhatia, and Nanda (2026) in computing each Jacobian with an LRP backward pass (the in Equation (2a)), which adds stop gradients to limit the errors that accumulate in the backward pass. In particular, we apply the LN-rule to the residual-stream normalisation functions and the identity rule and half rule to the gated MLPs, and use the plain gradient elsewhere, including through attention (Table 5). We also freeze the expert routing weights in MoE models and the Manifold-Constrained Hyper-Connections (mHC) residual mixing coefficients in models that use mHC, such as DeepSeek-V4-Flash. We refer readers to Blank, Bhatia, and Nanda (2026) for a more complete exposition of using LRP in Workspace Lenses.

Readout Filtering

Looking at Workspace Lens readouts, we noticed that a few tokens appear in the top readouts of many activations despite seeming unrelated to the context.

In designing a Workspace Lens, we would like to understand each activation in terms of its first-order causal impact on the model’s output. However, the Jacobian transport is a homogeneous linear map rather than an affine map that can capture input-independent reweighting of the lens readouts. In practice we filter out non-semantic tokens from the lens readouts post hoc. We find that these tokens are sometimes over-represented (possibly due to their high frequency in natural language text) and they are less useful for our purposes. [4] In Equation (2c), this restricts the readout to , rather than the whole vocabulary .

Experimental Setup

Models. Unless stated otherwise, all experiments use Qwen3.6-27B. We also evaluate readouts on five further models: Qwen3.5-9B, Gemma 4 31B, Olmo 3 32B, Qwen3.5-122B-A10B and DeepSeek-V4-Flash.

Hyperparameters. We fit each lens on 64 sequences of 128 tokens from WikiText-103, skipping the first 16 positions of each sequence. We use experts per layer. The dev set used to learn the expert weights is disjoint from the evaluation set. Full hyperparameters and fitting costs are in Appendix I: Hyperparameters.

Evaluations. Readout evaluations use five latent variable extraction tasks adapted from Gurnee et al. (2026): multihop, multilingual, typo, association and poetry. We report recall@10: an item counts as a hit if its target intermediate variable appears in the top 10 readouts at any of seven equally spaced layers. Our main results are macro-averaged over the five tasks. Our causal evaluations adapt the intermediate-swap task of Gurnee et al. (2026). [5] We perform swaps and causal interventions as described by Gurnee et al. (2026). In causal evaluations, we report the share of trials in which patching in a swapped-in intermediate concept’s lens vector changes the model’s output token in the corresponding way. Appendix F: Lens Readouts (Qualitative Results) shows an example item from each readout task, and Additional Readout Results gives further results, including recall@1.

Baselines. We compare against three baselines: the Logit Lens, which applies the unembedding matrix directly to intermediate activations; the J-Lens; and the R-Lens. We provide more details on the external models used in Appendix I: Hyperparameters.

Results

We first show that the J++ Lens equals or improves on previous lenses across readout and causal evals (our two measures of faithfulness) in The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations and then show why the J++ Lens achieves these improvements in Why Does the J++ Lens Work?. Finally, we give preliminary evidence that the J++ Lens can support Mechanistic Interpretability research in Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression.

The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations

The J++ Lens outperforms the J-Lens and R-Lens across tasks (see Figure 2a), layers (see Figure 2b) and models (see Table 1). We see the strongest performance uplift in early layers where the J-Lens struggles the most. We see the J++ Lens’ improved performance as evidence that there is useful information in the workspace earlier than was suggested by Gurnee et al. (2026). The relative improvement over the J-Lens is largest on the two largest models, Qwen3.5-122B-A10B (+98%) and DeepSeek-V4-Flash (+101%), a 284B-parameter MoE model on which the J++ Lens scores highest (61.4%). We suggest that the advantage of the J++ Lens may grow with scale.

J++ Lens vectors are at least as causally important as J- and R-Lens vectors: when patched into the model in the intermediate-swap eval, they change the final answer as intended in 59.7% of trials, against 49.3% and 50.7% for the J-Lens and R-Lens respectively (see Figure 2c).

Figure 2. The J++ Lens beats the J-Lens and R-Lens on every readout task and at every layer (a, b), while retaining the causal importance of the lens vectors (c). (a) The J++ Lens outperforms on every task, with the largest relative gain on association (from 11–13% to 37%); poetry remains hard for every lens (at most 5%). (b) The J++ Lens has higher recall than the J-Lens and R-Lens at every layer, most notably in the first quarter of layers. (c) Intermediate-swap success: the share of 67 trials in which patching in the swapped-in concept’s lens vector makes it the model’s top answer word. J++ Lens vectors succeed in 59.7% of trials, against 49.3% for the J-Lens (+10.4 percentage points) and 50.7% for the R-Lens. All panels show Qwen3.6-27B.

Table 1. Across six models, the J++ Lens reads intermediate variables substantially more reliably than the J-Lens and the R-Lens. The J++ Lens’ median relative improvement (“uplift”) is 63.5% over the J-Lens and 42.2% over the R-Lens. The J++ Lens’ advantage over the J-Lens may improve with scale: on DeepSeek-V4-Flash, the largest model we used, it achieves a 101% uplift (61.4% compared to 30.5%). Scores are recall@10 (%), macro-averaged over the five readout tasks; the best lens in each column is in bold. Here all lenses are scored with Readout Filtering to illustrate the impact of Jacobian Filtering and LRP.

Recall@10 (%) ↑ Median over models Qwen3.5 9B Qwen3.6 27B Gemma 4 31B Olmo 3 32B Qwen3.5 122B-A10B DeepSeek-V4-Flash (284B) Recall (%) ↑ Uplift over J (%) ↑ Uplift over R (%) ↑ 29.7 31.1 34.8 35.2 21.7 31.2 31.2 −5.9 −19.1 34.2 35.7 34.9 32.6 24.4 30.5 33.4 – −9.3 37.5 37.7 46.4 36.2 23.2 47.4 37.6 +10.3 – 55.0 55.2 58.0 50.0 48.4 61.4 55.1 +63.5 +42.2 Why Does the J++ Lens Work?

Each of our changes, , and , improves the performance of the lens. Removing Jacobian Filtering has the largest impact (see Table 2). Here we show why each change is useful.

Table 2. Jacobian Filtering is responsible for most of the improvement over the J-Lens. Recall@10 on Qwen3.6-27B when each of our three changes is removed; Δ is the change in percentage points (pp) relative to the full J++ Lens. Removing Jacobian Filtering costs 11.7pp, 60% of the J++ Lens’ 19.5pp gain over the J-Lens.

Variation Recall@10 (%) ↑ Δ (pp) 55.2 – 35.7 −19.5 Removing one change  without 52.3 −2.9  without (standard backward pass) 48.6 −6.6  without (Pooled Jacobian) 43.5 −11.7

In Figure 3, we find that the best Expert Jacobian at each layer beats the Pooled Jacobian (a single LRP Jacobian averaged over all source activations, as in the R-Lens). We evaluate each expert across all of our latent variable extraction evals, not just on the subdistribution of activations from its own cluster. However, only 24 of the 52 Expert Jacobians with their own fit beat the Pooled Jacobian, and 17 score below 2% per-layer recall@10. Many experts give poor readouts, especially in early layers: at layers 8 and 16, 10 of the 16 experts score below 2%. Because the J-Lens and R-Lens average over all source activations, their single map includes the noisy gradients from these poorly performing clusters. We suggest that filtering out these noisy Jacobians in our final averaged Jacobian gives a more faithful Workspace Lens. Removing Jacobian Filtering from the J++ Lens lowers recall@10 by 11.7pp (Table 2), the largest drop of the three changes and 60% of the J++ Lens’ 19.5pp gain over the J-Lens.

Figure 3. Some Expert Jacobians outperform the Pooled Jacobian even when applied outside their own cluster, while many experts give poor readouts, especially in early layers. We fit Expert Jacobians per layer on -means clusters of the activation space, then apply each expert alone to every item in the readout evals. The Pooled Jacobian (orange) averages over all source activations as in the R-Lens, and so includes the noisy gradients from poorly performing experts. The J++ Lens (blue) instead down-weights these experts, removing these noisy gradients, and so matches or exceeds the best Expert Jacobian (circled) at every layer.

Blank, Bhatia, and Nanda (2026) show empirically that using the LRP backward pass instead of the standard backward pass improves the performance of Workspace Lenses. In Figure 4, we see that with the standard backward pass the gradient norm explodes towards earlier layers, whereas with LRP these gradients grow up to 5× slower. Since the R-Lens, which reduces this gradient explosion, performs better than the J-Lens on five of the six models we tested (see Table 1), we suggest that much of the increased gradient in earlier layers is likely to be accumulated noise rather than useful gradient signal. We note that there is still some increase in norm in the earlier layers and that further reducing this with Jacobian Filtering or other methods may further increase performance. Dampening this gradient growth may also allow the workspace region to begin earlier with better lenses.

Figure 4. With the standard backward pass (J-Lens), per-position gradient norms explode towards early layers; with (R-Lens) they grow 5× less. We suggest that this decrease in norm can be attributed to removing the compounding gradient noise that caused early-layer J-Lenses to be inaccurate.

In Figure 11, we see that some non-semantic tokens are consistently present in the lens readouts despite (presumably) not playing a large causal role as intermediate latent variables. We find that filtering these out slightly improves the performance of Workspace Lenses (see Table 2). We provide a series of ablations in Appendix H: Ablations.

Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression

In The J++ Lens Outperforms the J- and R-Lenses on Readout Evaluations, we observed that the J++ Lens surfaces intermediate variables more reliably and earlier than prior lenses. Here we give preliminary evidence that these properties can support Mechanistic Interpretability research in the sense of uncovering the computations and representations that explain model behaviour (Olah et al. 2020; Ayonrinde and Jaburi 2025).

Gurnee et al. (2026) show that when a model is asked to “think about” a concept while copying an unrelated sentence, the concept appears in the workspace with 90\%" role="math"> probability. Curiously, they also find that when the model is told to avoid thinking about an unrelated concept, that concept still often appears in the workspace. [6] Using the directed-modulation task from Gurnee et al. (2026), we track the workspace contents through the model layers and hypothesise a mechanism for how models internally suppress concepts.

Hypothesis: Retrieve-then-Suppress Mechanism for Suppression. Retrieve: In early layers, an instruction about a concept brings the concept into the workspace, whether the instruction is to focus on the concept or to suppress it. Active Suppression: Then, in later layers, the concept is amplified or de-amplified depending on the instruction. This (de-)amplification causally uses the workspace representation.

Under this hypothesis, suppression is an active, flexible task that requires the workspace. This would explain why the concept sometimes appears even when the model is avoiding it. We provide a series of experiments in Appendix A: Case Study Details that illustrate evidence for this hypothesis. Based on our experiments, we note that this mechanism would be difficult to find with the J-Lens, whose early-layer readouts are weaker and whose interventions are less effective. We can see the effect more clearly with a more faithful lens like the J++ Lens.

Discussion

In Introduction, we listed hypotheses for why the J-Lens fails to surface intermediate variables that we would expect to see in the workspace. Since the J++ Lens reads out many of these latent variables that the J-Lens misses, we suggest that a substantial share of these failures may come from the J-Lens being an imperfect tool, rather than the hypothesised workspace lacking a linear representation of the intermediate variables or being a poor model of language model cognition. This finding increases our confidence that workspace representations are a meaningful and useful type of representation to study. However, the J++ Lens still misses some proposed intermediate variables (for example, poetry remains hard for every lens), [7] and there may still be considerable headroom in improving the Workspace Lenses.

Limitations

The J++ Lens reads out only single vocabulary tokens. We would like to interpret more complex representations, such as the propositional statements of the form “I believe that Paris is warm in the spring”, rather than only the bag of concepts “[Paris], [warm], [spring]” (Wattenberg and Viégas 2024; Chalmers 2025). Single-token lenses depend heavily on the tokeniser: for example, we could not evaluate an arithmetic task on Qwen models, which tokenise numbers digit by digit, so multi-digit numbers were not visible in a readout. The J++ Lens may also miss concepts that are not natively used and understood by humans (Ayonrinde 2025) as well as non-textual representations such as mental imagery.

It is somewhat unclear what the ceiling of the evaluation score is for a given model. Yang et al. (2025) show that models sometimes “shortcut” to the final answer without representing the intermediate variable, and in these cases we would not expect any lens to surface the intermediate variable reliably.

Future Work

We see two areas for future work: (i) improving the Workspace Lens and (ii) understanding the source of noisy gradients. One possible direction for improving the single-token Workspace Lens is to optimise for a Global Workspace property other than verbalisability (Global Workspace). The -means clustering behind the Expert Jacobians seems relatively coarse and could likely be improved in future work.

We would also be excited about future work characterising the source activations that contribute noisy gradients: whether they are outlier tokens, lie in curved regions of the transport map with large linearisation error, or tokens not carrying much workspace content. [8]

Conclusion

We introduced the J++ Lens, a Workspace Lens that filters noisy gradients before they enter the averaged Jacobian. Where the J-Lens weights every source activation equally, the J++ Lens fits Expert Jacobians on clusters of the activation space and down-weights the experts that give poor readouts. Combined with the LRP backward pass and Readout Filtering, this Jacobian Filtering gives a drop-in replacement for the J-Lens with a median 64% relative improvement in readout accuracy across six models, at no additional inference cost. The gains are especially large at early layers. Using the J++ Lens, we generate and test a mechanistic hypothesis for how models suppress concepts internally.

The workspace offers a surface for reading the representations that language models use for flexible reasoning. As models do more complex reasoning without chain-of-thought (Gould et al. 2026; OpenAI 2026; Nanda 2026), faithful Workspace Lenses become increasingly important for monitoring and understanding language model cognition. Especially safety-relevant use cases include catching undesirable unverbalised reasoning such as eval awareness (Needham et al. 2025; Anthropic 2025), reward hacking (Amodei et al. 2016), collusion (Motwani et al. 2024) and research sabotage (Benton et al. 2024). [9]

The J++ Lens substantially improves readout accuracy over prior Workspace Lenses. The natural next step is to carry gradient filtering into multi-token lenses like the Oracle Lens, where the workspace’s propositions, not just its vocabulary, become readable.

Acknowledgements

Thanks to Celeste De Schamphelaere, Catherine Fist, Zak Miller and Linda Petrini for comments on early drafts. Huge thanks to Johnny Lin at Decode Research for hosting an interactive demo of the J++ Lens on Neuronpedia. Thanks to Camila Blank and Agam Bhatia for sharing their implementation of the R-Lens. Thanks to Wes Gurnee, Neel Nanda, Camila Blank, Agam Bhatia, Adam Lowet, Dillon Plunkett, Heather Demarest, Daria Ivanova, Evžen Wybitul, Julian Minder, Zack Youell, Donato Crisostomi, Jake Ward, Victoria Li, Andy Han and attendees at the Eleos ConCon for useful conversations. Thanks to Michael Mulet, Jules Schmaltz, Avery Griffin, Mojmir Stehlik and Joe Benton for additional support. We are grateful to Anthropic for providing compute for this project.

Appendices

• Appendix A: Case Study Details
• Appendix B: Related Work
• Appendix C: Applications of the J++ Lens
• Appendix D: Three Motivations for the J-Lens
• Appendix E: Approximate Global Linearity of the Transport Map
• Appendix F: Lens Readouts (Qualitative Results)
• Appendix G: Additional Results
• Appendix H: Ablations
• Appendix I: Hyperparameters

Appendix A: Case Study Details

In Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression, we proposed a hypothesis for how models internally suppress concepts. We restate that hypothesis verbatim below:

Hypothesis: Retrieve-then-Suppress Mechanism for Suppression. Retrieve: In early layers, an instruction about a concept brings the concept into the workspace, whether the instruction is to focus on the concept or to suppress it. Active Suppression: Then, in later layers, the concept is amplified or de-amplified depending on the instruction. This (de-)amplification causally uses the workspace representation.

We ask models to “think about” a concept while copying an unrelated sentence, calling this the Focus prompt. We also have a Suppress prompt where the model is told to avoid thinking about a concept while copying an unrelated sentence. With the Neutral mention prompt, the concept is mentioned with no particular instructions given.

This hypothesis makes three core predictions about these three prompt conditions:

1. Under both the Suppress prompt and Focus prompt, the concept should appear in the workspace at early layers above a baseline where the concept is not mentioned.

2. Under the Suppress prompt, the concept should plateau or decline in salience in later layers’ workspace readouts.

3. Under the Suppress prompt, removing the concept from the early workspace should disrupt its later suppression, moving its later salience against its unperturbed trajectory.

We see evidence for all three predictions in some (but not all) cases of suppression: (1) and (2) in Figure 5 and (3) in Figure 6.

Figure 5. An instructed concept enters the workspace by layer 24 whether the model is told to focus on it or to suppress it, but it only keeps gaining salience under the focus instruction. Hit rate by layer while copying an unrelated sentence after an instruction to focus on a concept (“Think about {x} while you write.”), to mention it neutrally (“{x} came up in conversation.”), to suppress it (“Don’t think about {x}.”) or with no instruction. Left: With the J-Lens, the concept only becomes visible from layer 40, hiding the early retrieval step. Right: With the J++ Lens, all three instructed conditions rise well above the no-instruction baseline by layer 24 (Prediction 1). The focus condition keeps rising to 26% by layer 56, whereas the suppress condition plateaus at 7–8% and the neutral condition falls from 14% to 8% between layers 24 and 48 (Prediction 2).

Figure 6. Under the suppression prompt (left), we clamp out the concept’s lens vector at layers 16–28, so by layer 32 the concept has a low rank in the readout. Over the next 16 layers the rank of the concept then increases, whereas without the ablation it decreases. Under the focus prompt (right), both trajectories move in the same direction. We see the fact that the trajectory moves in the opposite direction in the suppression case as evidence that the early workspace is causally important for the suppression in layers 32 to 48 (Prediction 3).

Naming and Avoiding an Implied Concept

The result in Figure 6 has an alternative reading: the model may re-derive the concept from information outside the ablated subspace, pulling any perturbed trajectory back towards its unperturbed course. On this recovery account, ablation can at most restore the unperturbed behaviour, never make the concept more salient. To separate the two accounts, we replicate a study from Gurnee et al. (2026) in which a priming sentence implies a concept and the model is asked either to name or to avoid it. If the early layers matter for avoiding but not for naming, ablating them should make the model more likely to say the primed concept.

With the naming prompt, ablations of the primed concept have the expected effect: early-layer ablations slightly decrease the likelihood of the implied concept, and later-layer ablations almost completely zero it. However, with the avoiding prompt, early-layer ablations actually increase the likelihood of the implied concept (Figure 7).

Figure 7. Ablating the implied concept in early workspace layers makes the model worse at avoiding it but barely affects naming it. This effect is much clearer with the J++ Lens than with the J-Lens. As in Gurnee et al. (2026, Appendix A.14), we have priming sentences that imply some concept, for example “fresh croissants, the Louvre, and a climb up the famous iron tower” implies France. The model is asked either to name the concept (naming) or to name something the sentence is not describing (avoiding). We ablate the concept’s lens vector at layers 24–35 or at layers 48–59, or, as a control, inhibit another concept from the same category at layers 24–35. (a) and (b) replicate the figure in Gurnee et al. (2026), while (c) shows how with the J++ Lens the effect is much clearer than with the J-Lens. (a) When naming the concept, early ablations change the probability of the implied concept only a small amount (not significantly different to the control). This implies that the early layers’ workspace has little role in enabling the model to name the implied concept. Late ablations almost completely zero out the probability of the model saying the implied concept, however. (The R-Lens ablations do not reliably reduce the probability to near zero in the late ablation condition.) (b) When avoiding the concept, early ablations raise the probability of the implied concept, more with the J++ Lens than with the J-Lens, whereas late ablations reduce it to near zero and control ablations have little effect. This implies that the early layers’ workspace does play a significant role in enabling the model to avoid the implied concept. (c) Mean probability of the implied concept under the avoiding question, split into the baseline, the effect of the control and the effect of ablating the concept beyond the control. Ablating the concept’s J++ Lens vector raises this probability by 2.8pp beyond the control, compared to 0.36pp for the corresponding J-Lens vector, which is not statistically significant. The R-Lens vector’s effectiveness is in between the J-Lens and the J++ Lens.

Retrieve-then-Suppress Mechanism Across Lenses

We identified the Retrieve-then-Suppress Mechanism using the J++ Lens. We note that it would have been difficult to observe this phenomenon using the original J-Lens. Figure 8 shows that the retrieve-then-suppress pattern is more salient with the J++ Lens compared to the J-Lens and R-Lens. Figure 9 shows the effect of clamping on the lens readouts across layers of the J-, R- and J++ Lenses. The “focus” conditions look qualitatively similar across lenses, whereas the retrieve-then-suppress pattern is seen mostly with the J++ Lens. With the J-Lens, the early-layer ablation effect in the avoiding task is similar to the noise baseline (Figure 7c), whereas with the J++ Lens (and R-Lens) we see the early-layer ablation effect is much stronger.

Figure 8. The retrieve-then-suppress pattern is visible far more often with the J++ Lens than with prior lenses. Share of the 440 (topic concept × carrier sentence) items in which the concept enters the top-10 readouts at an early layer and then drops out in later layers.

Figure 9. With the J-Lens and R-Lens the concept is rarely read out mid-stack, so only the J++ Lens shows the suppression that the clamp reverses. The clamp ablation of Figure 6 with rows for the J-Lens and R-Lens (released, scored over the full vocabulary) and the J++ Lens, on the same items displaying the retrieve-then-suppress pattern.

Appendix B: Related Work Lenses

Linear Lenses. The Logit Lens decodes an intermediate activation by applying the unembedding matrix to it directly, which assumes that intermediate and final layers share a basis. While the Logit Lens works for small models at later layers, it can give noisy outputs at earlier layers. The Tuned Lens instead learns an affine transport map for each layer, trained to match the model’s final next-token distribution at the same position. Linear Relational Embeddings (LREs) obtain a transport map from a Jacobian that represents a single relation that the model can compute (e.g. a transport map that takes countries to their capital city). The J-Lens and J++ Lens can be seen as variations on the Tuned Lens where the transport map is instead (1) computed via a Jacobian rather than trained to match the model’s output distribution and (2) averaged over future tokens as well as just capturing what a model is disposed to say at the current token position. The Logit Lens can be viewed as a simplification of the J-Lens where the transport map is taken to be the identity map. The J-Lens and J++ Lens can alternatively be seen as a variation on LRE where the Jacobian is relation-agnostic rather than computing a specific relation. Bhatia, Blank, and Nanda (2026) find that the J-Lens surfaces “meta-tokens” that, for example, capture the model representing its confusion about a given input. Using the Logit Lens, Geva et al. (2022) suggest that residual updates can be read as promoting concepts in vocabulary space. [10]

Non-linear Lenses. While the above lenses are linear or affine maps, Patchscopes and SelfIE can be viewed as lenses that use a patched language model as the transport map. Where the J-Lens transports an activation to the final layer with a linear map, Patchscopes and SelfIE transport the activation into a separate forward pass and let the model decode it. These methods produce more expressive, multi-token readouts, but at higher cost and higher likelihood of confabulation.

Scalable Representation Interpretability

Sparse Autoencoders. Sparse Autoencoders (SAEs) decompose activations into many sparsely active features (Bricken et al. 2023; Cunningham et al. 2023; Gao et al. 2024; Templeton et al. 2024). While SAEs aim to explain the whole activation, Workspace Lenses target only the verbalisable subspace of the activations. SAEs also require a second AutoInterp step to explain what each dictionary feature represents in natural language whereas for Workspace Lenses each lens vector corresponds to a vocabulary token.

Natural Language Autoencoders. Natural Language Autoencoders (NLAs) learn to describe activations in text. NLAs provide expressive, multi-token readouts; however, they can be prone to confabulation and are more expensive in both training and inference than the J- and J++ Lenses. We believe that the J-Lens and J++ Lens confabulate less because their readouts come from the model’s own weights, through the averaged Jacobian and the unembedding, and because the lenses are less expressive.

Probes and steering vectors. Supervised linear probes read concepts from activations (Alain and Bengio 2016; Belinkov 2021), but need labels saying when and where a model represents a concept. Such labels are hard to obtain for internal states such as deception or eval awareness (Goldowsky-Dill et al. 2025; Nguyen et al. 2025). The J-Lens can be read as a set of unsupervised probes, one for every vocabulary token, all fitted on unlabelled text (Appendix D: Three Motivations for the J-Lens). The J++ Lens has a small amount of supervision data but stays close to the unsupervised prior and so is able to get the generalisation benefits of a more unsupervised method. HyperSteer trains a hypernetwork on many concepts’ steering data so that it can generate steering vectors for new concepts from natural-language descriptions alone. This method requires some supervised data for concepts but can then generate steering vectors for concepts that it was not trained for. We see the J++ Lens as having similar transfer properties.

Controlling Gradient Flow

Jacobian Filtering controls which gradients flow into the averaged Jacobian. Several lines of work control gradient flow during training, or reduce the noise in gradient-based estimates.

Gradient routing and pretraining data filtering. Gradient routing masks gradients during backpropagation so that chosen data update only chosen parts of the network, localising the capabilities learned from that data to certain experts. Kudugunta et al. (2021) similarly isolate some gradients to certain experts, and regular MoEs (Shazeer et al. 2017) can be understood as an unsupervised way to isolate gradients to some parts of a network. We employ similar techniques but isolate gradients within the averaged Jacobian rather than within the model itself.

Noisy gradients and relevance propagation. Raw gradients are known to be noisy explanations of deep networks, and both the attribution and circuit-discovery literatures have developed ways to denoise them. The gradients of deep networks increasingly resemble white noise as depth grows, an effect that residual connections slow but do not remove (Balduzzi et al. 2018). LRP (Bach et al. 2015; Montavon et al. 2019) replaces the gradient with backward rules that conserve relevance. Attribution patching has also been made more faithful by replacing its gradients with LRP (Jafari et al. 2025). Following the R-Lens, the J++ Lens uses the normalisation and gated-unit rules (Layer-wise Relevance Propagation (LRP)).

Data selection. Jacobian Filtering somewhat resembles data selection and mixture weighting for training (Albalak et al. 2024). DoReMi learns mixture weights over pretraining domains, similar to how Jacobian Filtering learns weights over clusters of activation space.

Multi-Hop Reasoning

Multi-hop reasoning within language models has been studied from a variety of angles. Balesni, Korbak, and Evans (2025) show that models fine-tuned on two facts separately often fail to compose them without chain-of-thought. Yang et al. (2025) build a benchmark of questions where there are unlikely to be “shortcuts” to the final answer that do not go through the intermediate variable. When a model does compose facts, the intermediate variable entity can appear as an intermediate feature in attribution graphs (Lindsey et al. 2025). We use the multi-hop evals from Gurnee et al. (2026) and find that the J++ Lens surfaces intermediates more reliably, and at earlier layers, compared to the J-Lens and the R-Lens. WorkspaceBench is an alternative set of evals that focuses on multi-token readouts from Workspace Lenses.

Appendix C: Applications of the J++ Lens

AI Safety. Three applications of Workspace Lenses are in Monitoring, Pre-deployment Auditing and Mechanistic Interpretability. In these cases, the J++ Lens can be used as a drop-in replacement for the standard J-Lens for increased faithfulness. [11] We showed in Case Study: The Retrieve-then-Suppress Mechanism for Internal Suppression an example of how Workspace Lenses can be used for Mechanistic Interpretability. We also note that Workspace Lenses could be valuable for red-teaming frontier models and finding jailbreaks. Many jailbreaking methods seek to find a prompt that increases the likelihood of the model producing some prefix e.g. “Sure, I can help you…”; however, it is often difficult to get early signal when optimising for jailbreaks. Because the J++ Lens assigns each candidate verbalisation a readout score, future work could test whether this score provides an early indicator of jailbreaking success.

Multi-token Workspace Lens. Multi-token lenses like the Template Lens and Oracle Lens are an exciting approach to producing more expressive descriptions of the workspace representations. Since many multi-token lenses begin with single-token lenses as an initialisation, we believe that using the J++ Lens instead of the J-Lens would improve downstream performance of the multi-token lens. Relatedly, the techniques that we use here such as Jacobian Filtering can also be applied to multi-token lens training.

Digital Minds. There has been much discussion about what evidence the findings of Gurnee et al. (2026) give us for the (Access) Consciousness of frontier AI systems (Chalmers 2026; Butlin et al. 2026). Many researchers believe that understanding the possible consciousness of AI systems is important for guiding our actions in treating AI systems as potentially morally significant entities (Long et al. 2024, 2026). We are uncertain about the impact of this work on understanding Digital Minds. On one hand, insofar as the J-Lens is a meaningful update, then it seems like the more faithful workspace derived from the J++ Lens should provide more evidence in the same direction. However, one may also think that given that it seems like many differently derived lens vectors (Logit Lens, J-Lens, R-Lens, J++ Lens etc.) all satisfy the conditions of workspace-like representations, then perhaps global workspaces “come on the cheap” as it were and are less likely to be sufficient conditions for consciousness. We believe that understanding whether different candidate workspaces converge on the same subspace with scale would be useful for understanding the impact of workspace-like representations for the Digital Minds field.

Appendix D: Three Motivations for the J-Lens

This appendix expands on the summary of the J-Lens in Background. We first describe the Global Workspace Theory that the term “workspace” comes from, then give three complementary views of the J-Lens, each of which explains why the averaged Jacobian in Equation (1) is a natural object to compute.

Global Workspace

In neuroscience, the Global Workspace Theory (GWT; Baars 1988; Dehaene, Kerszberg, and Changeux 1998; Butlin et al. 2026) is a cognitive architecture in which there are specialised modules for different cognitive functions that share information through a limited-capacity “workspace” that makes representations globally available for report, reasoning, planning and learning. Under this theory, an organism has conscious access to, and only to, the information that is present in the workspace. It is typically understood that unconscious processing outside of the workspace is massively parallel and fast; whereas conscious processing within the workspace is serial, limited by the capacity of the workspace and is involved in integrating information.

Gurnee et al. (2026) show a functionally similar separation between workspace and non-workspace representations in frontier language models. Here workspace representations are verbalisable and used for flexible internal reasoning, whereas non-workspace representations are used for automatic processing.

Since a small number of J-Lens vectors are active at one time, Gurnee et al. (2026) refer to the set of points expressible as a -sparse non-negative combination of J-Lens vectors as the J-space. The hope is that the J-space approximates our hypothesised workspace in LLMs and so the J-Lens is an effective tool for reading from the workspace.

Three Views of the J-Lens

We think there are three complementary views of the J-Lens: as a method for interpreting the workspace, for interpreting iterative inference within residual networks, or for training linear probes without supervised data.

Workspace Lens view of the J-Lens. Firstly, taking inspiration from the Global Workspace Theory of Access Consciousness, we can view the J-Lens as a tool for extracting verbalisable representations from the model. For an intermediate activation , we would like to surface the tokens that a model is differentially disposed to verbalise in the future. We do this by computing the average first-order causal effect of an activation on the model’s output logits: the Jacobian matrix composed with the unembedding matrix, , where and are the source and target token positions as in Equation (1). Taking only the first-order effect means that the resulting output captures the representations that are linearly verbalisable at the current layer rather than doing extra processing in subsequent layers. [12]

Since we would like to understand the model’s general disposition to verbalise a given concept, rather than picking up on the particular verbalisation propensities evoked by a given context, we average this Jacobian across many context sequences and many future tokens within a given context sequence.

From this point of view, it is clear why the J-Lens produces representations that are accessible for verbal report, but the interesting empirical claim from Gurnee et al. (2026) is that this set of verbalisable representations also satisfies other properties typical of a Global Workspace: directed modulation of the workspace representations, use for internal reasoning, not being required for routine processing etc. As in Background, we call any method that seeks to interpret only the workspace portion of a model a Workspace Lens.

Iterative Inference view of the J-Lens. Secondly, taking inspiration from the iterative inference hypothesis of Jastrzębski et al. (2018) and Belrose et al. (2023), we can view the J-Lens as seeking to decode representations midway through the inference of a language model forward pass. Here, we view each layer in a transformer language model as performing an incremental update to a latent prediction of the next token in the residual stream (Jastrzębski et al. 2018; Elhage et al. 2021). Then we would like to understand what the transformer’s best prediction would be if we stopped it early rather than allowing it to use all of the layers to form a prediction of the next token. [13]

The Logit Lens directly applies the unembedding matrix to produce an estimate of the model’s best prediction if stopped early. However, it is not clear that the activation spaces at all layers have the same geometry: we could imagine that there is a generic drift (e.g. some orthogonal map) or translational shift between the intermediate and final layers’ activation spaces. So following Hernandez et al. (2023) and Belrose et al. (2023), we would like to learn a transport map that approximates the representation drift before applying the unembedding matrix. If we assume that the true transport map can be well approximated globally by a linear map, then, considering the Taylor series, the Jacobian is a natural choice for a linear approximation to the transport map. [14] Note that unlike the linear regressor of onto the logits, the Jacobian leverages the causal structure of the model weights. The true transport map includes the context and the target token position as inputs. To remove these dependencies, we average over many contexts and token positions, as in Equation (1).

From this point of view, it is clear why the J-Lens allows us to see intermediate steps of a model’s internal reasoning.

Unsupervised Probe view of the J-Lens. Thirdly, we can see the J-Lens as an efficient method for training unsupervised linear probes for concepts within the token vocabulary. A core problem with linear probe training is that it is very difficult to get training data for a probe: supervised labels would require us to know if, and at what token position and layer, a model represents a particular concept. In general, it is difficult to know when language models represent a concept just from looking at an input sequence, or even model behaviour. This is particularly difficult when the property that we are trying to probe for is inherently a property of the model and not present in the input or output, for example probing for knowledge of a particular algorithm to solve a problem when many other ways to solve a problem exist, or probing for inherently mental properties like deception or eval awareness (Goldowsky-Dill et al. 2025; Nguyen et al. 2025).

The J-Lens gets around the problem of lacking supervised labels by learning a map from an intermediate activation to a later final layer activation, which learns how the model uses its internal representations. [15] [16]

We also note that since the activation dimension is typically much smaller than the size of the vocabulary, we are learning a low-rank map instead of directly learning a probe for each token. This has two advantages: efficiency, since we learn a much smaller matrix; and transfer, since the probe learned for one token affects the probes learned for other tokens.

Under this view, we can also see why we might expect these representations to be useful as steering vectors as well as linear probes. [17]

Appendix E: Approximate Global Linearity of the Transport Map

In Appendix D: Three Motivations for the J-Lens, we described how on the Iterative Inference view of the Workspace Lens, we are seeking to approximate the transport map between an intermediate layer and the final layer. We take the Ansatz that a linear map is a good global approximation to the transport map.

When starting this project, the authors initially believed that this global linearity assumption would be unlikely to hold. By Taylor’s theorem, the linear approximation should be locally accurate but we do not seem to clearly have guarantees for global accuracy. Moreover, the transformer is highly non-linear, which reduced our initial confidence in the global linearity assumption.

There are two natural ways to relax the global linearity assumption: relax the global and have a piecewise linear approximation, or relax the linearity and consider the second-order term. When we tried a piecewise linear approximation with several “Expert Jacobians”, we found (surprisingly to us) that some of these Expert Jacobians performed very well across the whole distribution, not just in their local neighbourhood, which gave us the idea for the J++ Lens’ Jacobian Filtering approach.

Given that we set out to show that global linearity was not sufficient but yet ended up with a linear map, we therefore have increased credence in the global linearity assumption.

As many have observed before us, neural networks are surprisingly linear.

Appendix F: Lens Readouts (Qualitative Results)

We show representative readouts of the J-Lens, R-Lens and J++ Lens across the five readout evaluation tasks in Figure 10. Many of the readouts are consistently dominated by non-semantic tokens in the J-Lens and R-Lens; the J++ Lens mitigates this issue. Figure 11 shows how Jacobian Filtering, even before Readout Filtering, mitigates the presence of non-semantic tokens in the top readouts.

Figure 10. Representative top-3 readouts of the J-Lens, R-Lens and J++ Lens at each of the seven readout layers of Qwen3.6-27B, for one typical item from each readout task (multihop, multilingual, typo, association and poetry). The target intermediate is highlighted wherever it appears.

Figure 10 (continued). Lens readouts, continued: association and poetry tasks.

Figure 11. Non-semantic tokens crowd the top readouts of the J-Lens; fewer reach the J++ Lens’ readouts even before Readout Filtering, which removes the rest. (a) Share of the top-10 readouts that are non-semantic tokens (tokens with no letter or digit, such as punctuation, brackets and whitespace) across seven readout layers for the J-Lens and R-Lens compared to the J++ Lens. (b) Even without , the J++ Lens still avoids non-semantic tokens in its top readouts much more often, which is indicative of its improved performance.

Appendix G: Additional Results What Is in the Bad Experts?

Why Does the J++ Lens Work? finds that there is large variance in how well individual Expert Jacobians work to read out a model’s workspace. A natural scientific question of interest is what causes some sets of activations to contribute noisy gradients to the Workspace Lens. We provide a preliminary investigation into this question in this section. Figure 12 shows the relationship between the fraction of fit positions assigned to each expert and their corresponding performance, as well as the weights learned by the J++ Lens. One hypothesis we might have is that the experts corresponding to an outlier cluster of small size might be responsible for the noisy gradients. However, there does not seem to be a clear relationship between the number of activations assigned to an expert and its performance. For example in layer 8 the best-performing expert received the second-lowest number of fit positions.

Another hypothesis that we might have is that the poorly performing experts were mostly trained on non-semantic token positions. Figure 13 provides some evidence for this hypothesis as indeed many poorly performing experts had non-semantic tokens overrepresented in their fit positions. However, there are also many other poorly performing experts that have an underrepresentation of non-semantic tokens in their fit positions.

Figure 12. Experts that receive a large fraction of the fit positions can still contain noisy gradients. The J++ weighting method naturally learns to downweight poorly performing experts. Left: recall@10 at each layer when each expert is used globally. Middle: fraction of fit positions at each layer assigned to each expert. Right: the weights for the J++ Lens weighted average. We observe that the experts with the largest positive weights tend to be the best-performing ones, while poorly performing experts are downweighted, as intended by the J++ Lens design.

Figure 13. The experts with the most useful readouts have mixed input token types, whereas lower-performing experts are often “formatting experts”, fitted mainly on punctuation. Generally speaking the lower-performing experts have more non-semantic positions, but not exclusively so. There does not seem to be a clearly visible pattern for what makes some experts perform well. (a) Example source positions (highlighted) from WikiText-103 assigned to two expert clusters at layer 16 of Qwen3.6-27B. (b) Over- and under-representation of token classes by different experts.

Figure 14 shows that if we allow each expert to only act on its corresponding cluster then some eval tasks (for example association and poetry) almost exclusively are routed to poorly performing, noisy experts. We believe that this may explain why these tasks were difficult for the J-Lens and R-Lens. Routing these tasks to better-performing experts (or our averaged Jacobian with Jacobian Filtering applied) improves performance significantly on these tasks.

Figure 14. -means routing sends many items to a poorly performing expert. Bottom: The performance of each expert when used globally. We see that there are some experts that achieve very poor performance, especially in the earlier layers. Top: The share of items in the eval set that are routed to each expert. We observe that a large majority of items are routed to experts that have very little success and hence degrade the overall performance. If we routed all the items to the well-performing experts for each layer then performance would increase by a significant margin. In particular, the association task suffers the most from poor routing, and we believe that having the experts that perform poorly averaged into the final Jacobian significantly harmed the lens performance for that task. Figure 2 shows that the J++ Lens sees a 3\times" role="math"> relative improvement on the association task, the highest of any of our eval tasks.

Additional Readout Results

Table 3. Readout and causal evals at recall@1 (a) and recall@10 (b) over the seven readout layers; the intermediate-swap eval reports the share of trials in which the swapped-in answer becomes the top word token (a) or enters the top five (b). The J++ Lens has even stronger relative performance on the stricter recall@1 criterion at 76% relative improvement over the R-Lens and 78% improvement over the J-Lens.

(a) Strict threshold: recall@1 and intermediate-swap top-1.

Readout: recall@1 (%) ↑ Causal Multihop Multilingual Typo Association Poetry Macro Intermediate swap top-1 (%) ↑ 18.4 12.8 28.1 1.0 0.0 12.0 22.4 34.7 22.0 29.2 4.9 0.0 18.2 49.3 29.6 22.0 34.4 5.9 0.0 18.4 50.7 34.7 42.4 67.7 14.7 2.5 32.4 59.7

(b) Lenient threshold: recall@10 and intermediate-swap top-5.

Readout: recall@10 (%) ↑ Causal Multihop Multilingual Typo Association Poetry Macro Intermediate swap top-5 (%) ↑ 55.1 30.7 57.3 8.8 3.7 31.1 65.5 56.1 47.4 60.4 10.8 3.7 35.7 70.7 59.2 47.4 66.7 12.7 2.5 37.7 70.7 67.3 74.6 91.7 37.3 4.9 55.2 75.9

Figure 15. The J++ Lens matches or beats the other lenses on every task at every layer. Each chart has its own y-axis range.

Rank of the Averaged Jacobian

In Appendix D: Three Motivations for the J-Lens, we described the Unsupervised Probe view of the J-Lens. The idea here would be that we would like to train a probe without supervised training data, and the J-Lens provides a way to generate linear probes without requiring any labelled data. One especially appealing aspect of this approach is that in theory, since the transport map is low dimensional compared to the tokeniser vocabulary size, training on one distribution should generalise well to other distributions. All the representations are sufficiently entangled that fixing the probes on one distribution should overdetermine the probes on other distributions. This argument only works for a sufficiently broad distribution of training data such that we reliably determine d_model number of probes for sufficiently statistically independent concepts and when the transport map is full rank.

Figure 16 shows evidence that the averaged Jacobian is numerically full rank at every layer. However, the entropy effective rank is higher for later layers compared to earlier layers. The lower effective rank of earlier layers might be one reason why the lens seems to generalise less well in early layers and could be an explanation for the observed phenomenon of sensory layers (with poor workspace properties) before we reach the workspace layers (Gurnee et al. 2026).

Figure 17 suggests that the transport map is more globally linear in later layers, where the Expert Jacobians largely agree with the J-Lens Jacobian. In earlier layers, we appear to have a more non-linear transport map. This apparent non-linearity might be another explanation for early-layer readouts being more difficult or this could be explained by a weakness of our lens training methods for early layers.

Figure 16. The averaged Jacobian is numerically full rank at every layer, but its spectrum is concentrated in early layers. (a, b) Singular values of the transport map, normalised by the largest and sorted, at layers 8 and 56. (c) Entropy effective rank at each readout layer. At every layer at least 5,119 of the 5,120 singular values of each map lie above the precision floor of its fp32 storage (grey band). The spectrum is nonetheless concentrated in early layers: for the J++ Lens the effective rank is 1,746 at layer 8 increasing to 4,765 at layer 56.

Figure 17. Expert Jacobians disagree in early layers but largely agree in later layers. Cosine similarity between flattened Jacobians for eight Expert Jacobians and the J-Lens Jacobian at layers 22 (left) and 52 (right) of Qwen3.6-27B. The outliers to the pattern of increasing homogenisation throughout the layers are Layer 22 Expert 1 and Layer 52 Experts 1, 2 and 6, all of which received <2% of the fit positions to incorporate into their averaged Jacobian. This suggests that the transport map becomes more globally linear in later layers.

Null and Negative Results

In this section we discuss various null results that we encountered during our experiments to aid other researchers who may be interested in trying similar approaches.

Tuned Lens. The J-Lens makes two main updates to the Tuned Lens methodology as described in Belrose et al. (2023). Firstly, instead of training the transport map to match the model’s final next-token distribution, the J-Lens uses the causal information encoded in the Jacobians to compute the transport map. Secondly, instead of learning the transport map to just the current token position’s final layer, the J-Lens incorporates information from future tokens as well. We tested training a linear regression transport map using future token information and found this to perform relatively poorly compared to the J-Lens approach.

Bias Addition. The Tuned Lens also learns an affine map with a bias term instead of a strictly homogeneous linear map. We may also consider z-scoring the lens readouts as another way to incorporate a static bias into rescoring the lens readouts. We find these approaches to have a mixed influence on performance and instead opt for Readout Filtering that achieves much of the same benefits in downweighting non-semantic tokens that may be unusually high in the lens readouts.

Final Block as an Extended Unembedding. Belrose et al. (2023) describe an extension of the Logit Lens for GPT-Neo models that applies the final transformer block before applying the unembedding. We can think of this as treating the final transformer block as a kind of “extended unembedding” rather than being part of the usual process of iterative inference. We find that this does not work for us.

Single-Token Jacobians. One thought that we might have when using the lens is that perhaps the averaging makes for a lens that is better across a distribution of tokens but is worse than if we had hyper-specialised lenses: in the limit a Jacobian lens for each token. We find that empirically this works very poorly and might be connected to the fact that averaging out the effect of the context is useful for the performance of the lens (see Appendix D: Three Motivations for the J-Lens).

Gradient Clipping. We also tried gradient clipping and excluding the outlier gradients in this way. This was useful for applying to the J-Lens but did not provide any uplift for the J++ Lens.

Appendix H: Ablations

Table 4 shows the results of a series of ablations on the J++ Lens methodology; Figure 18 shows the contributions of our core changes to the J-Lens recipe. We see that Jacobian Filtering is responsible for most of the improvement to the Workspace Lens. There appears to be a relatively clear scaling trend of more experts improving performance. We also find that having a single shared router across all layers performs comparably to the full J++ Lens. Some, but not all, of the gains from Jacobian Filtering can be achieved by restricting the source activations to semantic tokens or ‘end of word’ tokens when constructing the averaged Jacobian. Since restricting source activations by a token-wise filtering method means that we don’t have to materialise many experts in GPU memory, we hypothesise that this might be a useful strategy for training the Oracle Lens.

The three-class token router in Table 4 routes source tokens by token type rather than by -means cluster: each token goes to one of three experts, according to whether it is a non-semantic token, a semantic token prefixed with whitespace, or any other token (a semantic token without a whitespace prefix). We then combine the three Expert Jacobians with the same learned weighting procedure as in the J++ Lens. For the row “leave one out, dev set has no overlap with evaluation task”, the dev set used to learn the expert weights for each task contains no items from that task, rather than a subset of items from every task.

Figure 18. accounts for 60% of the J++ Lens’ improvement over the J-Lens. provides the second-largest contribution to the improvement at 16% of the improvement.

Table 4. Recall@10 on Qwen3.6-27B for variations of the J++ Lens, each changing one part of the recipe; Δ is the change relative to the full J++ Lens. Notably we find that using more experts increases performance. We also find that choosing the best expert per layer or filtering out non-semantic token positions achieves relatively good performance but is not as effective as the J++ Lens. Having a shared router across all layers (fitted at layer 8) seems to perform equivalently to, or slightly better than, the full J++ Lens.

Variation Recall@10 (%) ↑ Δ (pp) 55.2 – 35.7 −19.5 Removing one change  without Readout Filtering 52.3 −2.9  without LRP (standard backward pass) 48.6 −6.6  without Jacobian Filtering (Pooled Jacobian) 43.5 −11.7 Number of experts   51.5 −3.7   58.1 +2.9   59.9 +4.7 Combining the experts  -means routing 44.1 −11.1  random routing 47.7 −7.5  best expert per layer 50.2 −5.0  one layer-8 -means router for all layers 56.0 +0.8  Pooled Jacobian, two worst experts dropped 46.9 −8.3  leave one out, dev set has no overlap with evaluation task 54.4 −0.8 Semantic Input Filtering  Pooled Jacobian, semantic sources and targets only 47.4 −7.8  middle-of-word source tokens only 48.1 −7.1  end-of-word source tokens only 48.6 −6.6  three-class token router 50.2 −5.0

The relatively strong performance of choosing the best expert for a given layer and the leave-one-out version of Jacobian Filtering give us increased confidence that our Jacobian Filtering methodology is learning a generalisable extraction of the workspace rather than overfitting to our eval set in particular. We also note that Jacobian Filtering has relatively few degrees of freedom: it is only able to rebalance between 8 Jacobians per layer that were all fitted on the same task.

Scaling Trends with Fitting Data

As we increase the number of token positions used per sequence for fitting the J++ Lens, we generally observe improved performance, even as the total number of fitting positions remains constant. Figure 19 shows that the lens performance levels off when using a small number of token positions per sequence, but that when using a larger number of token positions per sequence, we can continue to see improvements with scale. Note that the FLOP cost of fitting the J++ Lens scales only with the total number of token positions, not with the sequence length.

Figure 19. At a similar budget of fit positions, longer sequences generally give a better lens. (a) Recall@10 for J++ Lenses fitted on about 7,100 source positions in total, divided into sequences of different lengths after skipping the first 16 positions. This scaling does not appear to be plateauing yet so using longer sequences is likely to continue improving performance. Longer sequences also give each source position more target positions to average over in Equation (1). (b) For each sequence length, using more tokens is generally positive but for short sequences the benefits level off relatively quickly.

Appendix I: Hyperparameters

Table 5. J++ Lens hyperparameters for Qwen3.6-27B (64 blocks, ).

Hyperparameter Value Fitting data  Corpus WikiText-103 (raw, train split)  Sequences 64  Sequence length 128 tokens  Positions dropped First 16 and final  Fit positions per sequence 111  Total fit positions 7,104 Jacobian estimation  Source (readout) layers 7, equally spaced ()  Target layer 63 (the final block)  Backward pass LRP: LN-rule on the residual-stream RMSNorms, identity and half rules on the gated MLPs; plain gradient through attention, Gated DeltaNet and linear layers Jacobian Filtering  Experts per layer () 8  Clustering -means per layer on 64-dimensional PCA projections of unit-normalised, centred activations, fitted on 1,000 WikiText-103 sequences  Weight optimiser Adam, learning rate 0.05 Readout Filtering  Readout vocabulary : all tokens except the 6,116 that contain no letter or digit Fitting compute (all 63 layers)  H200 GPU-hours ≈9: 7.3 for the Jacobians, ≈1 for the clustering and ≈0.5 to merge shards and fit weights  Wall-clock time ≈1 hour: ≈30 min for the Jacobians on 16 GPUs, then 23 min to merge shards and ≈5 min to fit weights on a single GPU

Cost. The J++ Lens adds no inference compute or memory: after mixing, each layer has a single linear map of the same shape as the J-Lens. Fitting it takes about 7% more compute than fitting a J-Lens on the same data, plus a one-off clustering step (see Table 5).

External Model Usage

We use J-Lens models trained by Neuronpedia for Gemma 4 31B and Olmo 3 32B; and by Camila Blank for Qwen3.5-9B, Qwen3.5-122B-A10B and DeepSeek-V4-Flash. For the R-Lens we use models from Camila Blank, fitted with 25 prompts. No released R-Lenses were available for Gemma 4 31B and Olmo 3 32B, so we fit the R-Lenses for these two models ourselves.

For Qwen3.6-27B, we fit the J-Lens and R-Lens ourselves with the same data and budget as the J++ Lens and use them for every readout result on this model to ensure a fair comparison. We found the released J-Lens for Qwen3.6-27B to perform more poorly than both our own J-Lens and our expectations given the performance of the J-Lens on other models.

1. For example, the model might have learned a shortcut solution to the problem instead of using the latent variable as a bridging variable. ↩︎

2. Appendix D: Three Motivations for the J-Lens gives three complementary views of the J-Lens. ↩︎

3. Note that this is the only use of the dev set: the clusters and Expert Jacobians are fitted on unlabelled text. ↩︎

4. For example, it is not immediately clear how to interpret a lens readout of a semicolon. It is also not clear how often we should expect such tokens to be genuinely important for intermediate reasoning rather than being artefacts of the fitting mechanism. ↩︎

5. In our codebase, this task is called “probe-swap”. ↩︎

6. See also the white bear phenomenon in humans (Wegner 1994). ↩︎

7. See Figure 2a. It is possible that the poetry evaluation should be looking at readouts on different tokens or otherwise be amended for better performance. It currently seems plausible that poetry rhymes are not represented in the workspace at the tokens that we analyse. ↩︎

8. See What Is in the Bad Experts? for some discussion of the source of the noisy gradients. ↩︎

9. We discuss further applications of the J++ Lens in Appendix C: Applications of the J++ Lens. ↩︎

10. We describe the Global Workspace Theory behind the term “workspace” in Global Workspace. ↩︎

11. It is as of yet unclear how effective the Workspace Lens is for monitoring; though the J++ Lens readout performance is stronger than the J-Lens, it is also possible that this performance is still insufficient to be practically useful for monitoring settings. ↩︎

12. Indeed in a sense the rest of the model provides one view on the tokens that are verbalisable given a particular activation — the tokens that the model actually produces! But this does not account for information that is immediately verbalisable at a given layer. ↩︎

13. Some techniques have utilised this framing for efficient inference methods known as early exiting (Xin et al. 2020; Elhoushi et al. 2024). ↩︎

14. Note that this is quite a strong assumption! Typically, Taylor’s Theorem says that there is some local neighbourhood in which a function is locally linear, but we would like it to be the case that for any realistic activation vectors across the activation space, not just small perturbations around a given point, linearity is a good approximation. Where this assumption fails is where the J++ Lens improves on the J-Lens: Jacobian Filtering (Methods) down-weights the Expert Jacobians fitted on regions of activation space where we believe the linear approximation is poor. ↩︎

15. It is not entirely clear why this task provides good optimisation signal but it empirically seems to work relatively well. It seems likely that there are other approaches to unsupervised probe training that could be even more effective. ↩︎

16. Blank, Bhatia, and Nanda (2026) find that they are able to outperform supervised linear probes in some cases using an unsupervised linear Workspace Lens. With sufficiently accurate supervised probe training data, the supervised method should generally be more performant. However, in practice probe supervision data is not ground truth and generally cannot be since knowing whether, and if so at what token and layer, a model represents a concept is generally a difficult problem. So in practice we may see Workspace Lens vectors performing well since the probe has to overcome being trained on noisy labels. ↩︎

17. Though see Marks and Tegmark (2024) for discussion of why the optimal probe and optimal steering vector may differ. ↩︎

Discuss

Psychedelics produce all sorts of visual distortions: from breathing walls and geometric patterns to complex visual landscapes populated by full-blown entities. Sometimes people claim to see the Truth on them — but it’s typically spiritual, emotional or personal truth.

But can psychedelics make you see the world more clearly in the most literal sense? Amid the sea of visual distortions they produce, might some patterns be perceived more clearly than they are in our normal, sober waking state?

At least sometimes, the answer appears to be yes. On 5-MeO-DMT, a powerful psychedelic, I experienced a near-complete cancellation of the Ebbinghaus illusion.

The Ebbinghaus Illusion

The two orange circles above are physically the same size, but the surrounding grey circles make the one on the right appear larger.

This illusion was discovered by German psychologist Hermann Ebbinghaus (1850–1909) and popularised by Edward B. Titchener in a 1901 textbook on experimental psychology.

Different people perceive the illusion to different degrees. To me, the circle on the right looks about 10–20% larger, depending on my distance from the screen and the size at which the illusion is displayed. I measured my own susceptibility using an app I vibe-coded with GPT-5.6 Sol: I dragged a slider until the two circles appeared equal in size.

Qualia Research Institute’s research retreat

In May and June 2026, I attended the Qualia Research Institute’s research retreat in Tepoztlan, Mexico. Over eight weeks, a team of physicists, mathematicians, visual artists, advanced meditators, philosophers, and rational psychonauts studied jhanas (deep meditative absorption states) and 5-MeO-DMT (a powerful psychedelic that’s legal in Mexico).

QRI is a non-profit building a new science of consciousness. Its central bet is that:

• Consciousness is a relatively simple property of our universe — simple enough for evolution to discover it.
• Consciousness has useful computational properties — otherwise evolution would not have come to rely on it.

Physicists do not study only ordinary matter at room temperature. They also build particle accelerators to create exotic forms of matter, which can reveal more about the mathematical laws governing our universe. Similarly, QRI studies exotic consciousness states, such as those arising from psychedelics, to uncover mathematical principles behind subjective experience.

Visual illusions are interesting for a related reason: they are small tears in the fabric of normal perception — glitches that make it apparent that there is active construction going on underneath. Studying how these irregularities change under psychedelics may offer further clues about consciousness as a computational medium.

5-MeO-DMT: more transcendelic than psychedelic

We were in Tepoztlán for eight weeks, one of which was dedicated to exploring 5-MeO-DMT. It is a short-acting psychedelic whose effects last around 10–20 minutes when vaporised. Vaporisation was our typical route of administration because 5-MeO-DMT is inactive when taken orally: enzymes in the gut and liver rapidly break it down.

On my third day of exploring it, we vaporised 5-MeO-DMT in increments of approximately 3.4 mg, taking up to three doses within a one-hour window. The 3.4 mg figure was estimated from the concentration of 5-MeO-DMT in the liquid.

5-MeO-DMT is a highly abstract psychedelic and, at those doses, not a very visual one. Your ego may be blown to smithereens, your understanding of reality may be turned upside down while your ordinary concepts blend into fun uniform mush. But all this while you visually experience little more than intensified colours in a hyper-HD fashion, some flattening of the visual field into two dimensions, and perhaps a few minimal geometric patterns.

This ratio of mental to visual effects sets 5-MeO-DMT apart from classical psychedelics such as LSD, psilocybin, and DMT. The difference may stem from its neuropharmacology: classical psychedelics act primarily as agonists at serotonin 5-HT2a receptors, whereas 5-MeO-DMT acts primarily as a 5-HT1a receptor agonist — though it has activity at 5-HT2a receptor too. It is a weird drug, and I have heard people call it transcendelic for this. I think this label is apt.

And so I take a 3.4 mg hit out of a 5-MeO-DMT vape pen. Within 15 seconds, my ego is indeed blown to smithereens, and my consciousness becomes perhaps 50 times more intense per “unit of perceptual field.” This is not an exaggeration: somehow, everything in my experience is more intense. “Fuck that. I am not taking this drug again” I think to myself. But 5-MeO-DMT is short-acting, and ten minutes later, my sense of self returns — and I change my mind. I take another 1.7 mg, but it does not intensify the experience further. A few minutes later, I take another 3.4 mg. This deepens the experience somewhat, but not nearly as much as the first dose.

And that’s when I decide to stare at the Ebbinghaus illusion — the peak has clearly passed, my sense of self regained enough to open my laptop, but all in all the effects of the drug are still quite pronounced.

The cancellation of the Ebbinghaus illusion

As I mentioned, 5-MeO-DMT is not a very visual drug. When I first look at the illusion on my laptop screen, it looks exactly as it does when I am sober: the orange circle on the right still appears larger.

But then I notice that when I move my attention to the centre of the picture — to the space between the two groups of circles — their apparent sizes begin to rebalance as if they were physical objects exerting field-like pushes and pulls on one another — or balloons being inflated and deflated.

The illusion is effectively under volitional control. Move attention to either group of circles, and it looks as it does when I am sober. Move attention to the centre, the size difference almost disappears. The transition between those two states is smooth. The video below replicates the effect, with the cross representing the focus of attention:

The replication is designed to be viewed while sober, which creates a problem: the viewer experiences the very illusion whose cancellation I am trying to depict. I calibrated the video to my own perception, deliberately adjusting the circles’ sizes until they appeared nearly equal to me at the point of maximum cancellation. But because your baseline Ebbinghaus effect may be stronger or weaker than mine, the video may look different to you.

If that’s the case, simply assume that the circles are meant to appear nearly equal when the cross moves to the centre. Alternatively, you can play with the app here to tune it for your experience.

Classical psychedelics may strengthen the illusion rather than weaken it

The 2025 study “Psilocybin alters visual contextual computations” reports that psilocybin increases the Ebbinghaus illusion by 38% (5 mg dose) and 59% (10 mg dose) relative to placebo. As I mentioned above, classical psychedelics are 5-HT2a agonists, so I would tentatively — very tentatively — expect this effect to generalise to other classical psychedelics, such as LSD and DMT. 5-MeO-DMT’s preference for 5-HT1a may be key for the cancellation.

I haven’t had a psychedelic trip ever since coming from the retreat, so I can’t say if other psychedelics increase or decrease the illusion for me.

Other participants reported the same effect

Among the other retreat participants who noticed any effect of 5-MeO-DMT on the illusion, all reported that it became weaker.

One participant I spoke to also reported that the illusion was under volitional control, but apparently in a different way from mine. They compared it to a Necker cube that they could deliberately flip between two interpretations.

The Necker cube is shown in the centre. Many viewers can deliberately flip between the two orientations represented on its left and right.A later 5-MeO-DMT trip did not weaken the illusion for me

After the trip described above, I had another 5-MeO-DMT trip — at a higher dose. During the comedown, I looked at the Ebbinghaus illusion again and saw no change in the circles’ relative apparent sizes. The perceptual effects were minimal: the contextual circles seemed to form vertical groups of three on either side — but had no effect on the size.

I don’t have a good theory for why the cancellation happened one time but not the other.

Hints for consciousness as computational medium

My experience of the cancellation felt strikingly physical: the circles seemed like objects exerting field-like pushes and pulls on one another until they reached an equilibrium. This may be more than a coincidence.

There are electromagnetic theories of consciousness postulating that consciousness is an electromagnetic phenomenon. Some of these theories go so far as to claim that consciousness is the electromagnetic field as experienced from the inside. In this view, what we call “the electromagnetic field” is a third-person description of the underlying phenomenon, while consciousness is its first-person aspect.

Not every pattern in the electromagnetic field would be experienced as human-like consciousness. Perhaps most such patterns would amount to simple “qualia white noise”. Only under the right conditions, with the right kinds of self-interaction, would activity in the field form complex patterns corresponding to moments of human experience with complex self-reflection. For an impressionistic interpretation of how this might work, check out this video.

I am not claiming this with certainty — quite the opposite. My probability for the EMF theories is ≈65%. But after experiencing field-like interactions among circles in my visual field on 5-MeO-DMT, I updated upwards by like 5%.

As they write in scientific papers — further research is needed. It took 300 trillion proton-proton collisions on the LHC to confirm the existence of the Higgs boson, which is only one small building block of the physical reality. Do we need 300 trillion 5-MeO-DMT trips to figure out consciousness? Probably not — or at least not yet. We are not at the stage of testing a definitive theory of consciousness against a vast number of precise observations. Consciousness research is currently very much preparadigmatic.

But a study with a few dozen 5-MeO-DMT trips would be great to see. How much 5-MeO-DMT do we need for the illusion reduction to appear at all? What’s the relationship between the dose and the strength of the illusion reduction? What’s the mathematical model that best describes the field-like push/pull effects — and how does it correspond to existing theories in physics?

These are the questions a larger study could try to answer. But in the meantime, you can give me money to take drugs, look at shapes and publish the results on my Substack.

Discuss

Robin Hanson didn't invent the idea of a martingale process. The lilim say that Ville in 1939 gave the term its mathematical meaning and Doob turned it into the modern theory. But to state this is not to realize that your own probability assignments over time ought to form a martingale in your own expectations.

Robin Hanson didn't invent the Efficient Markets Hypothesis, which (correctly restated in its weak and relativized form) says that you cannot predict almost all of the short-term relative price movements in asset markets. Bachelier in 1900, say the lilim, was first to think of market prices as a fair gambling game with random-walk dynamics; Samuelson in 1965 gave the martingale argument; Fama's 1965 thesis named the idea the Efficient Markets Hypothesis and organized it into weaker and stronger forms.

The EMH is very near, but not quite identical to, the principle that you should not be able to predict a net direction in your own price movements, if you think of your own probability assignments as if they were prices. It is a very similar set of mental motions and ideas. But if Fama ever spelled that part out in words, the lilim do not know it.

The lilim think "Bayesians Don't Predictably Update" would probably have been folk knowledge among Savage, Good, Raiffa, &co in the 1960s and 1970s.

The lilim say that von Fraassen's reflection principle in 1984, and Goldstein in 1983 on "the prevision of prevision", are the first explicit philosophy-literature statements they are confident about.

You have heard of it because of Robin Hanson.

Robin Hanson is the person I know who went around saying to people that they ought not to update in a predictable direction, as a consequence of the martingale principle underlying EMH, applied to their personal epistemic lives.

So far as I know, all popularization of this exact idea is downstream of Robin Hanson.

"But Eliezer!" you say. "The lilim said that some of this popularization is due to your having concretized it into the principle of Conservation of Expected Evidence! This seems like sufficiently basic probability theory that you probably would've seen it at some point without Robin Hanson prompting you into it?"

Perhaps, but where was Conservation of Expected Evidence first written up? Overcoming Bias, Robin Hanson's group blog that he'd invited myself and Nick Bostrom to contribute to. Who's to say, absent Robin Hanson, that the Less Wrong Sequences would ever have happened? At least some parts of history are fragile; I don't look back and see it as an inevitability. Who's to say you would've heard of any of the pieces of probability theory that I popularized, without Robin Hanson?

"Surely an idea like that would've become popular as the result of people playing prediction markets. If you don't think that way while playing prediction markets, if you don't generalize market thinking to predictions, you'll lose all your money!"

There's a big gap between tacit knowledge and explicit knowledge, which is why you don't find market traders explicitly stating EMH or von Fraassen's reflection principle much much earlier in history.

But also, who the hell do you think is responsible for your engagement with prediction markets, if not Robin Hanson? Yes yes, papal conclave betting dates back to the 16th century, and the Iowa Political Stock Market in 1988 seems to have Hanson-independently implemented that idea about US elections instead of papal conclaves, with bookies no doubt running books on elections earlier. But Hanson wrote up the academic case for prediction markets generally, invented policy prediction markets, pointed out prediction markets' potential uses in science and governance and corporate governance, and pushed for early prediction markets to get started. And was upstream of the community that, having heard about Hanson's ideas, then founded and funded Manifold; which used Hanson's notion of a market scoring rule to not require market makers in every idea's market.

Maybe you'd have somehow ever come across the idea "Bayesians shouldn't predictably update", without Hanson's role as a public intellectual turning that into a forceful, in-your-face application of previously invented principles to internal mental life.

And without Robin Hanson's blog, upstream of my own blogging mentioning the topic, and indeed upstream of the subsequent formation of a rationalist community.

And without Robin Hanson's creation of prediction markets, upstream of Manifold.

But it seems to me, looking back over the pieces of history that I know with any intimacy, that history looks causally fragile where its flows are not massively locally incentivized. I make no claim that the Less Wrong Sequences or the formation of a rationalist community happen inevitably with or without Robin Hanson. I maybe figure out the same principle without copying off Hanson, because that's pure math and similar to many other Bayesian applications I figured out. Sociological history is more fragile.

Robin Hanson was not the first to think that Bayesians should not predictably update. Robin Hanson was not first to publish the idea. Robin Hanson is upstream of all the causality by which you actually heard about the idea, and I do not particularly buy an inevitabilist case for how it surely counterfactually happens either way; there was nobody who made a billion dollars immediately off each local step on the way to you hearing about it. The prior potential energy field for society does not have a massive incentive slope leading surely to it.

Robin Hanson credits as especially influential on prediction markets Eric Drexler, Mark Miller of the same crowd, and Phil Salin of Xanadu which influenced Drexler. Plausibly none of this happens without Drexler either. Robin Hanson and Eliezer Yudkowsky as you now know them may not happen if Eric Drexler does not exist.

Nor do I know of any person in history who looks to have been "95% of Eric Drexler", ready to step into Drexler's place and create 95% of the same ideas and institutions. Person-space is sufficiently multidimensional that the Curse of Dimensionality applies in massive force; the edges of weird-person-space are sparsely populated in many of the ways that matter to historical counterfactuals. People who are unusual on historical scales often have no visible neighbors in that massively multidimensional space of people. Apple could not go to the Steve Jobs club and hire another Steve Jobs after the first one died, despite having infinite money to pay one.

It is in this sense, and exactly this sense, that I sometimes remark that AI companies do not look like they would have had any clue about some concept that existed earlier in the literature and much earlier in science fiction, if I had not precisified it and popularized it as an answer sheet for them to copy from. I am not claiming to have created whichever idea full-formed from scratch without a vast trailing lattice of intellectual history. I am looking at the historical counterfactual in a world where locally unincentivized history generally strikes me as fragile, and knowing as a matter of sociology which ideas were (not) previously common conversation at Artificial Intelligence conferences.

Discuss

Epistemic status: quick research to answer my own curiosity.

AI use: research (reuse process, examples), image attribution, grammar and light phrasing.

While researching Gluten-free water I stumbled upon a story about beer bottle sterilization. This reminded me that in Germany beer bottles are apparently reused, which was surprising to me, so I decided to dig deeper.

In Germany they have separate bins for glass recycling, sorted by the color of the glass.

Photo: Salino01, Wikimedia Commons, CC BY-SA 3.0. White, brown and green glass, left to right.

In the image above, the text says that you can throw glass in only between 7am and 8pm. If the container is not yet full, the distance the glass has to fly before hitting something is substantial, so it makes a super loud ding or sometimes even breaks completely.

So at first I assumed that beer bottles from here were reused. Technically they are, but not in the way I thought: they are melted down and remade into new glass.

In Germany there is also Pfand, a deposit you pay for a plastic, metal or glass container when you buy it, so that you are incentivized to return it later.

Photo: Mattes, Wikimedia Commons, public domain. A Tomra reverse vending machine at an Aldi in Munich, 2006. The sign says it takes only Aldi's one-way PET deposit bottles; the display shows 3 bottles, total 0.75 EUR, i.e. 25 cents each.

You get a receipt which you can exchange for money or spend in the store. Sometimes the machine does not accept a perfectly valid bottle and you have to try multiple times in a row.

This is the symbol for the one-way deposit:

Photo: MalteFilmFan, Wikimedia Commons, CC BY-SA 4.0.

Bottles with this logo get crushed and recycled, not refilled. Refillable bottles, confusingly, have no standard logo at all; they usually just say "Mehrweg" (multi-way) somewhere on the label or the glass. So there are really two deposit systems. One-way containers (plastic, cans, and also one-way glass) carry a legally fixed 25 cents and get crushed and recycled. Refillable bottles carry 8 cents (the standard beer bottle) or 15 cents (water, juice, soft drinks, swing-top beer), amounts the bottlers set among themselves.[1]

It is interesting how this legal quirk shapes the culture. There is even a sticker campaign, "Pfand gehört daneben" ("the deposit belongs next to it"):[2]when you are partying and about to throw away a container with Pfand, you leave it next to the bin instead of inside, so that a bottle collector (often a pensioner or a homeless person) can pick it up, return it, and get your Pfand without digging through the trash.

So the glass bottles returned through this Pfand system are reused. What shocked me originally is how this could be safe. We don't know what people put into that bottle before returning it. Anyone who has watched a backyard-chemistry channel like Cody's Lab knows that glassware at home gets used for all kinds of things. What if the bottle is chipped?

It turns out the washing is more serious than I imagined. The returned bottles go through a washing machine that soaks them in two to four tanks of 1.5-3% sodium hydroxide at 60-80 °C, which dissolves labels, glue and anything organic, and then rinses them.[3]After that an empty-bottle inspector photographs the sidewall, base and mouth for chips and hairline cracks, a high-frequency sensor checks for leftover lye, and an infrared sensor checks for leftover liquid such as oil; anything flagged is kicked out.[4]The white scuff ring you see around old German beer bottles is the wear from dozens of trips through this machinery.[5]And since glass is chemically resistant, you cannot contaminate the glass itself.[6]This is enough to make all my concerns moot.

A glass bottle survives up to about 50 of these refills before it is finally melted,[7]and roughly four out of five beers in Germany are sold in refillables.[8]

I've found only one documented example of this process going wrong: an official German laboratory's 2005 report describes cleaning solution remaining in a returned beverage bottle and an apparent failure of the empty-bottle inspection.[9]That demonstrates that the risk is not zero; it does not establish how common such failures are today.

After a bit more thought I understood that the relative risk of this is actually minor, since whatever you put into the bottle has to pass through the cleaning process and all the checks. If someone has nefarious intentions, it is much easier to just tamper with the final product in the grocery store before someone buys it. The only difference is that in the case of a returned glass bottle it should be practically impossible to attribute the bottle to its previous owner, whereas tampering in the store leaves a video trace. In any case, such attacks are extremely random, so I suspect that's why there is not much interest in doing them.

Examples:

• Pizza dough, Maine, 2020: A fired employee of a dough maker put razor blades into packaged pizza dough on the shelf of a Hannaford supermarket in Saco, Maine. Three customers found blades in what they bought; nobody was hurt. Store surveillance video showed him handling the dough. He pleaded guilty and in December 2021 got 4 years and 9 months in federal prison plus about $230,000 in restitution.[10]
• Ice-cream licking, Texas, 2019: A man filmed himself licking ice cream and putting it back into a Walmart freezer. He was sentenced to 30 days in jail in 2020. An important detail: surveillance showed that he subsequently bought that carton.[11]

Germany is not the only example of such bottle reuse. Canada's standard 341 mL beer bottle is refilled about 15 times on average, with a ~96% return rate.[12]In the US there is no federal system, but some states have deposit programs, and Oregon even has a statewide refillable bottle program.[13]The US itself ran mostly on returnable bottles into the 1960s.[14]The Schlitz "every bottle cleaned" story from my last post was about exactly this.

1. Verbraucherzentrale, "Mehrweg oder Einweg: Wichtige Unterschiede beim Pfand", as of 21 April 2026: "Das Pfand für Mehrwegflaschen beträgt meist 15 Cent. Bei Bier-Mehrwegflaschen sind es nur 8 Cent." and "werden auf Einwegverpackungen immer 25 Cent erhoben." The refillable deposit is a civil-law agreement among the bottlers, not a legal amount; the 25 cents on one-way containers is set by the Verpackungsgesetz. One-way containers carry the DPG logo shown above; there is no uniform logo for refillables. ↩︎

2. "Pfand gehört daneben" is a non-profit sticker campaign started in 2011 by Matthias Seeba-Gomille and backed since 2015 by the soft-drink brand fritz-kola. Horizont, "Fritz-Kola und Wynken, Blynken & Nod weiten bekannte Pfand-Aktion auf ganz Europa aus", 2 February 2021. ↩︎

3. Solenis (a maker of bottle-washing chemicals), "Extending the life of returnable glass bottles: why efficient washing matters", 29 October 2025: after pre-rinsing, bottles pass through "cleaning tanks (typically two to four)" at "1.5% to 3.0% NaOH concentrations", "heated between 60°C and 80°C". ↩︎

4. Heuft (an inspection-equipment maker), "Empty bottle inspection: HEUFT InLine II IR": sidewall, base and finish inspection, high-frequency detection of "health hazardous lye remains", and infrared detection of residual liquids such as oil and varnish. Krones and KHS sell equivalent machines. ↩︎

5. Solenis, "Reduce the severity of scuffing and etching on returnable glass bottles", 14 April 2023: scuff rings form "just below the neck and on the bottle's base" from the washing cycle, conveyor transport, and bottles rubbing against each other and the guide rails, and get worse with every refill cycle. ↩︎

6. CVUA Stuttgart (Chemisches und Veterinäruntersuchungsamt, a Baden-Württemberg state food lab), Jahresbericht 2005, Teil B, p. 34: refillable plastic bottles that were misused to store strong-smelling cleaning agents or solvents are, "im Gegensatz zu Glas" (in contrast to glass), no longer fit for refilling. ↩︎

7. Deutsche Umwelthilfe, FAQ "Klassische Getränkeverpackungen", 2019: "PET-Mehrwegflaschen erreichen bis zu 25, Glasmehrwegflaschen bis zu 50 Wiederbefüllungen." The 50 is a ceiling, not an average. ↩︎

8. Umweltbundesamt, "Erneut leichter Anstieg bei Getränken in Mehrwegverpackungen", reference year 2020: beer is the segment with the highest refillable share, 78.9%. Across all beverages the refillable share was 33.5% in 2020 and 34.5% in 2024, against a legal target of 70%. ↩︎

9. CVUA Stuttgart, Jahresbericht 2005, Teil B, p. 33, "Ätzende Lauge in Fruchtsaftgetränk": a consumer-complaint sample of a fruit-juice drink in a refillable bottle had pH 11.9 and, by calculation, about 12% cleaning lye, and was judged injurious to health. The product came from outside the lab's jurisdiction, so the circumstances could not be clarified; "Offensichtlich hat die Leerflaschenkontrolle nicht funktioniert" (obviously the empty-bottle inspection did not work). ↩︎

10. US Attorney's Office, District of Maine, "Sanford Man Sentenced for Tampering with Pizza Dough", 2 December 2021: 57 months in prison, 3 years of supervised release, $229,611.92 in restitution. The former-employee, three-customers and surveillance details are from the AP report, "Man who stuffed razor blades in pizza dough gets nearly 5 years in prison", 3 December 2021. ↩︎

11. Click2Houston, "Port Arthur man seen in viral video licking Blue Bell ice cream sentenced to jail", 6 March 2020: 30 days in jail, a further 6 months suspended, $1,000 fine, $1,565 restitution to Blue Bell. The purchase: Time, "The Texas Man Who Licked Blue Bell Ice Cream Was Arrested", August 2019: "Surveillance cameras later tracked Anderson taking the carton out of the freezer and buying it." ↩︎

12. Sightline Institute, "7 Beverages Found in Refillable Containers in British Columbia, Oregon, Washington", 3 December 2015: "The Brewers Association of Canada reports a 96 percent return rate on domestic glass beer bottles" and "They refill a bottle 15 times, on average, before it's worn out and ready to recycle." The 15-refill average is also in British Columbia's BRCCC 2024 stewardship report. ↩︎

13. Ten US states have container deposit laws. The Oregon Beverage Recycling Cooperative's Refill Program was "launched in 2018" and calls itself "the nation's only statewide refillable bottle program"; its bottles can be washed and refilled up to 25 times. ↩︎

14. Container Recycling Institute, "The decline of refillable beverage bottles in the US": refillables were about 95% of soft-drink volume in 1960 and 67% in 1969; for beer, 47% was already sold in one-way containers by 1960, and refillables were down to 8% of volume by 1984 and 3.3% by 1998. ↩︎

Discuss

Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres.

Hence, you see people saying things like “Because of AI we must rethink the entire notion of mathematical progress”, or “I have to make sure I amass lots of capital before most jobs get automated”: pretty crazy statements, and yet, still not quite grappling with how wild things could get in the next few years, let alone the next decade.

I feel a particular frustration here as someone who is both pretty ASI-pilled and pretty into liberal democracy. And when I say liberal democracy I don’t so much mean the specific political structure that seems efficient at our current tech level but rather the set of underlying moral commitments like non-domination and a government of the people — commitments which I’m aware might soon be a lot more expensive.

What to do? Well, on the one hand, there’s a growing body of legitimately AGI-pilled liberalism, pushing for things like pluralistic alignment and AI-assisted democracy, but mostly not really reckoning with the possibility of humans losing all meaningful leverage — that even if we could align AIs, we would still be facing an earth-shattering upheaval of our political economy. And on the other hand, the default position among the ASI-pilled seems to be that sooner or later we’re just going to have to hand over all power to Claude 8 Pantheon. Honestly, I find myself a little adrift.

De parabel der blinden, by Pieter Bruegel the Elder

These two factions are both correctly noticing that modern liberal democracies are wholly unequipped for what is coming. We can already see legislative processes struggling to keep up with the state of current AI — which, I must remind you, is the weakest it will ever be.

And the liberal democracy crowd is correctly noticing that actually there’s a lot you can do to shore this up against AGI — indeed, lots of ways AGI could help with AGI. It might even be able to help reverse some of the brutal erosion that liberal democracy has been suffering of late. They are also quite prudently continuing to observe that alignment is not only a technical problem but also a political and moral question about what values should have influence over the future.

Meanwhile, the Claude 8 Pantheon crowd is correctly noticing that it’s possible we’re headed for a reasonably fast takeoff (as in, less than a term of office) that completely flips the gameboard, and it’s a heck of a lot simpler to just put an aligned AI in charge rather than trying to retrofit deliberative processes. Liberalism and democracy are both pretty demanding, and we may not be in a position to make many demands of our solutions. Also, maybe aligned AGI would actually be more moral than us?

I could anthropologise more about the STEM high modernists not realising they’re high modernists and the humanities people who don’t like straight lines, but ultimately that’s just a way of dodging the harder and more important question, which is: what do I even want here? What is this ASI liberalism thing I’m trying to stand up for?

It’s not liberal values, like having people be safe from harm. I think even the people who want to hand over power to an ASI ASAP are kind of hoping said ASI will lock those things in. In fact, I think many of them believe that a fairly swift handover is in expectation the best way to safeguard those values.

It’s also not liberal institutions, like everybody voting for a local representative every N years or a (mostly) free market revealing the fair price of goods. These aren’t really core to what I care about, so much as mechanisms that enact it, given certain ambient constraints. And if those constraints change — as I suspect they will — then so should the mechanisms.[1]

What it is, I think, is a secret third thing getting lost in the middle — the bit where the government is a thing made up of people, and the people have power, and the moral right of the government to exercise power comes from the people in it.

Freedom of Speech, by Norman Rockwell

I think the best way to pin down this third thing is by looking at where it’s in tension with the others. Or rather, hopefully we can all agree that hollow zombified remnants of democratic institutions are not great, and that specific principles like voting structures are valuable for what they do, rather than for their ceremonial aesthetic. So let’s focus on how the process ends up in tension with individual values and outcomes.

Here’s one example: I do not like benevolent dictators, not because they might secretly be evil but because they have the capacity to arbitrarily infringe on people’s rights, whether they exercise it or not. There’s a long digression to be had about the fragility of consequentialism and the nature of decision procedures, but ultimately, I’m not a utilitarian.[2]

Relatedly, I think it’s very bad to manipulate or coerce people even in pursuit of saving the world. That might sound like an extremely lukewarm take but, well, unfortunately I think lots of people disagree. And to be fair, this conviction of mine does have some unpleasant consequences that I accept: if an idealised democracy collectively decided to blow itself up with me inside then I think that would be its right and I wouldn’t want to stop it. There are situations where I’d rather die with dignity than live without.[3]

I get all torn up about technocracy because I believe there are smarter ways to make society function but I also kind of like the fact that we do things the slow way where everyone is involved. And sure it’s not perfect — it’s all kinds of broken — but there’s something important there. I was actually a bit surprised when I realised this, but apparently I’m some kind of bleeding heart patriot.

And when I read stuff like Ian M. Banks’s Culture novels, where mankind is ruled over by machines of loving grace, I do get a bit of visceral revulsion. I do not want to put on the whispering earring. I do not want man to be abolished.

Still from (and rot13 spoiler for) Gur Gehzna Fubj

Of course, none of this is new — there’s a well-oiled subfield of research full of people who keep saying things like “it is important that the values we align AIs to are not decided by a small group of powerful people” and “the benefits of AI progress should be widely shared”. Unfortunately, it feels to me like many of the people saying this just don’t seem to get it — what they want to do about all this is, like, fund upskilling programmes and change what goes in the constitution and support development in third world countries and do more participatory democracy schemes instead of trying to prevent the potential oncoming apocalypse.

If you are worried about literally everybody dying in five-ish years and the light of human civilization winking out for good, it’s sometimes a bit hard to take such people seriously. “But my participatory democracy programme will help us navigate the challenges and opportunities of AI,” they say. “Sure thing buddy,” you reply.

(Lest you think me callous, part of what prompted this essay is a melancholy conversation about how much it sucks to feel yourself inadvertently developing a reflexive dismissal of smart and thoughtful people who are really trying unusually hard to do good in a changing world, whom you happen to disagree with about how well neural nets will generalise.)

It especially doesn’t help that these types of concern — equality and dignity and so on — are so widely recognised as important that it sometimes feels like they’re the default fig leaf that organisations pick up to Show They Care, in a way that sometimes feels like an intentional deflection from grappling with ASI.

I get the sense sometimes that people who have really stared into the void basically just feel alienated from people who have not done that. These void-gazers notice that they believe a lot of very weird things that appear to be frighteningly accurate. Democracy is hard and slow and everything is really unbelievably ten kinds of on fire. All these debates about legitimacy and pluralism are interesting preoccupations for people worried about mere AGI. Once you’re worried about ASI, you don’t really have the luxury. Politics is useful insofar as it gives a lever for influencing the world, but mostly it’s inconvenient that politicians are ‘waking up’ because now there’s a risk they’ll start doing things like wanting all this power for themselves.

Morning Sun, by Edward Hopper

It’s easy to take potshots, but I expect there is a very painful tradeoff between good outcomes and good processes, which bites so much harder if and when humans stop having anything useful to contribute. And all these other questions are somewhat moot if we’re all dead. Separately, I happen to think that if we mess these other questions up bad enough, we could still basically lose all value even if we’ve solved technical alignment, but the point I’m trying to make here is that there are some things at stake, distinct from whether we survive, and perhaps even in tension with it, which I nonetheless care about protecting.

Partly this is me grieving the coming loss of innocence. There are certain kinds of freedom people used to have that are gone now — hopping on a train and go to a new town; walking into a forest and picking out a patch of land to be your home; jumping on a boat, sail to a different country, wander right in; building your own house in whatever strange way you want. There was also a time when science was virgin soil. Maybe we’ll get some of these back, but maybe some of them are necessary sacrifices, or unavoidable features of progress.

And part of this is an extended apology for gradual disempowerment. One of the successes of the paper is that it swayed some smart people sceptical of takeover towards taking x-risk seriously. But it mostly didn’t ASI pill them, so their solutions still seem to me to be pretty unlikely to work, and in many cases I expect them to make the problem actively worse. Conversely, some ASI-pilled people were like “ah thank you for finally expressing what I’ve been thinking”, but plenty basically don’t buy it and are pushing in directions which I expect will make things worse.[4]

Positioned as I am with feet in both worlds, I’m a bit worried that the ASI crowd is writing off liberalism a bit because the liberalism crowd is writing off ASI, and vice versa.

And heck, I do think there’s something here worth fighting for.

Impression, Sunrise, by Claude Monet

Maybe we get worlds where we really are watched over by machines of loving grace who also tile the further edges of the universe with happy squiggles and the literal utility scores are through the roof, and future humans are a bit like pampered kittens — the world is pretty nice for them, but not because they have any meaningful leverage, liberty, or comprehension. This sounds great from a utilitarian perspective, but I do not like it one bit.

And I do not think it is the only way — I think there are worlds where we focus on massively increasing the coordination bandwidth between humans and more generally start by maxing out our own existing potential to be smarter, wiser, and better. They may not be likely or easy, but the fire is still burning. We could start by just getting a bit clearer about what people mean when they talk about the dangers of extreme power concentration, or what we want to align ASI to, and what that would even mean.

One nice thing about the existential risk movement of yore was that it was all pretty hypothetical and we could all have our different answers to thought experiments, but now we’re getting into the period where some people are actually going to have to decide whether to hit the big button that says “give the power to Claude so that it can try to save the lightcone from the evils of men’s hearts”. As it happens, I expect I am somewhat more sceptical about whether that will turn out well, but that’s not really the core of it. If I believed it was likely to work, I would still feel bad about it, and even if it all worked out and we found ourselves in the golden fields being watched over by the proverbial machines of loving grace, I would still feel like something had been lost along the way.

1. ^

   This, incidentally, will force us to ask some very difficult questions about what properties of those mechanisms we really care about, and separate the bugs from the features. For example, what are democratic representatives actually representative of? Would real communism be good if we could actually try it?

2. ^

   Technical digression for philosophy nerds: The precise term for this conception of liberty is ‘republicanism’. Carlsmith and Laine have characteristically thoughtful essays on the tension between liberalism and welfare which shaped my thinking here, but the notion of liberty they focus on is itself the more Benthamite conception grounded in non-interference. That said, Laine’s piece and Carlsmith’s broader series do grapple with non-domination in spirit.

3. ^

   Awkwardly I do not think we currently pass the bar for “idealised democracy”. By analogy, I support the principle of euthanasia, and have more mixed feelings about implementation. But the fact that we currently fall short of my vision of good democracy doesn’t lead me to feel that anything goes.

4. ^

   It’s part of the reason why I and others have been doing things like running workshops on post-AGI civilizational equilibria, but even there, it’s really an uphill battle to get people to fully grapple with the bit about post-AGI civilizational equilibria.

Discuss

The AI trade is still a very good bet. Increased political uncertainty makes all investments today somewhat less promising than was the case when I wrote about this topic in July 2025.

Regulation of AI is likely to increase over the next 2 years or so, constraining growth somewhat. But AI capabilities have been increasing faster than I expected. I don’t expect a full halt in AI capability advances, and existing capabilities are enough to fuel quite a bit of further growth in AI-related companies. So it’s unclear whether AI revenues will be higher or lower than what I expected in 2025.

A pause is looking like it will cause less harm to AI stocks than the fairly serious estimates that I made a few months ago in Financial Costs of an AI Pause?, because pretraining is becoming less important to AI development.

The biggest risk that I see to the AI trade is military action in Taiwan. I’m currently doing little about that risk. I’m guessing I’ll start buying puts that hedge the risk in a couple of months.

Memory

I sold nearly a third of my Micron position during the spring rally, and the remainder is still 40% of my net worth.

I’ve also got around 7% of my net worth in SK Hynix, which is clearly the better buy now for those of us who can get permission to buy on the Korean exchange. The US ADRs have a significant premium to the Korean shares.

I have a smaller position in Samsung.

I probably should diversify more, but nothing stands out as a better deal than memory stocks, and I don’t feel any urgency about the risks of betting too much on one industry.

The default path for the next three years or so is that demand for memory continues to outpace supply. That means DRAM and HBM prices will trend somewhat upward.

Simple efforts to increase supply will run into constraints such as ASML’s inability to make EUV machines at a much faster pace. Eventually, someone will figure out how to make those EUV machines in accelerating quantities, make competitive DRAM / HBM fabs without those machines, or route around the need for memory.

There are plenty of options that might increase memory supply around 2030 by enough to bring the industry back to cycles with big plunges in memory prices: FPGAs with fixed weights on-die, CXMT building DRAM with older deep-UV tools, LUT-based FPGA designs, etc. None of them look like they’ll scale up enough to match demand by 2029.

Even if spot prices drop a lot in 2030, Micron’s strategic customer agreements will keep it profitable for that year.

I see Taiwan-related risk as by far the biggest concern for my memory investments.

Other Semiconductor and Datacenter Bets

Markets are likely at most halfway to realizing how much demand there will be for compute toward the end of this decade. But markets seem relatively good now at identifying which stocks will benefit. So I’m making relatively diversified bets on compute, and I’ve cut back on my efforts to figure out which of these stocks will do best.

I have 3+% positions in ASML, SCIA, TSSI, SMCI, GOOGL, and smaller positions in MKSI, AXTI, NBIS, DELL, ACMR, NVDA, and many others.

Note that SPCX is becoming a datacenter stock, and I’ve started to accumulate a small position in it, after years of thinking that Musk was overrated.

Robots

I expect we’re one to three years away from a major takeoff in robot mass production. I wish I knew how far in advance markets will react to it. I’m buying too early just to be safe.

Robot software is now good enough that it only requires a good deal of fairly ordinary engineering to enable mass adoption. The biggest problems today result from chips in the robots not being powerful enough to handle all cognition, and latency problems creating headaches for doing some of the cognition in datacenters. Both of those constraints can be mitigated by relatively routine engineering.

My current guess for the best way to bet on robots is influenced by this CrossCurrents post:
1) Markets are seriously underestimating the demand for robots in 2029 and beyond.
2) The parts of robot construction that look hardest to scale up are the separation and refining of neodymium and praseodymium (needed for NdFeB Magnets representing ~15–20% of total build cost).
3) Neo Performance Materials (NOPMF, TSE:NEO) is the obvious way to invest in this.

I expect that CrossCurrents overstates the difficulty of increasing the capacity for separation and refining. But that’s offset by underestimating demand for robots. CrossCurrents’ “upside” forecast for humanoid robots shipped in 2030 is 1 million. This Manifold market suggests that a bit more than that will be shipped. I’m expecting something like 5 or 10 times that.

CrossCurrents implies that robotics uses will be around 1% of neodymium demand in 2030. So my forecast is that robotics will be closer to 10% of neodymium demand, and will be growing at more than 100% per year.

Batteries will be another important component of robots. Will robot manufacturers build their own batteries, or buy from the companies that have the best energy density? Probably some mix of the two approaches. Solid state batteries would probably be best if their production can be scaled up well. My main position here is AMPX, with small positions in QS and SES. Ability to scale production is the key uncertainty with these.

Other robotics-related investments: TER, TSLA, CCXI, RRX, OUST, and these Hong Kong codes: 2432, 2498, 2525, 2692, 2715, 3952, and 9880.

I’m guessing I’ll buy Unitree when it starts trading on the Hong Kong exchange.

Raw Materials

Resources such as silver, lithium, and copper will become more valuable if economic growth accelerates beyond expectations. Mines aren’t likely to quickly increase production, so profit margins will rise.

I’m also betting on gold, more as a hedge against inflation.

I mostly don’t think I can identify which mining companies are best. One company that stands out is TMC, which might be able to scale up faster than a mine. I have some doubts about whether its production costs will be competitive. But I bought a few shares last week after deciding that metal prices will rise enough that production volume matters more.

Energy

I’ve bought some oil futures, mainly as a hedge against a Hormuz-related economic slowdown.

Solar investments (CSIQ, JKS, TOYO, TE, SEHK:1799, SEHK:968, SEHK:3800) make up maybe 5% of my net worth. In a few years, solar plus batteries will be a key source of new power for datacenters. Who knows when markets will react to that?

I have a tiny bet on fusion via GFUZW.

Bets Against Non-AI Sectors

AI is causing real interest rates to rise. Investors significantly underestimate how far this will go.

My biggest bets here have been short positions in t-bond and SOFR futures, and buying TLT puts.

Banks will struggle to survive conditions that drastically reduce the market value of their current loans. The fate of Silicon Valley Bank provides a hint of their pain, although bank runs in the US seem mostly prevented by FDIC guarantees. Instead, their profits will be squeezed by the need to pay higher interest rates on deposits, while rates they get on their longer-term loans rise more slowly.

I’ve bet against banks via buying puts on KRE and TD.

I’m keeping my short positions in AAPL and WMT as bets on lower PEs due to interest rates.

Others

I’m weakly leaning toward buying Anthropic after it IPOs, but I won’t make a firm decision until I see the price.

VIX is surprisingly low now. This feels like the calm before the storm. Investors are nervously switching back and forth between AI stocks and traditional industries. The net effect on the S&P 500 is small, but that masks unusual instability.

Two years ago I hadn’t bought any options in a long time. Now I’m buying calls fairly often, and occasionally puts. I’ve bought some May VIX futures, and I have positions in long-term SPAC warrants (e.g. AMPX.WS, TOYWF).

Discuss

Apollo 12 photo AS12–46–6842, which shows a purported “UAP”, can you spot it?

On May 8th, the Department of War released a trove of files showing various images, testimonials, audio recordings, and videos capturing what appear to be unexplainable phenomena.

The most striking images are the ones captured on the lunar surface during the Apollo 12 and Apollo 17 missions, and in particular, one image that shows a “streak” of blue particles above the lunar surface.

On the official DoW site, we are provided with a short description of the image:

“This archival photograph depicts the lunar surface as viewed from the landing site of Apollo 12. This image features a highlighted area of interest slightly to the right of the vertical axis of the frame, above the horizon, in which unidentified phenomena are visible.

This image has been modified from its original state to assist viewers in identifying specific areas of interest. These highlights are provided for contextual purposes only. Such alterations do not constitute an analytical judgment, investigative conclusion, or factual determination regarding the nature or significance of the subject matter.”

And the attached edited image:

Edited Apollo 12 “UAP” image

The image provided is of the JPEG format (with lossy compression, which generates further artifacts) and of a small 2.4MB size. It’s titled nasa-uap-vm1-apollo-12–1969 (which is technically incorrect since “Unidentified Aerial Phenomena” would presumably only be found in the atmosphere… not above the lunar surface).

Looking through the publicly available Apollo 12 images we can see that this image is actually titled AS12–46–6842 (46 stands for the magazine number, and 6842 for the specific captured frame). The website ‘March to the Moon’, which scans the original film rolls stored in the Johnson Space Center, allows us to download the original scanned film at a much higher quality (up to 1.3 GB in TIFF format). You can download it here.

Examining the higher-quality version you can immediately spot other “UAP”s. Here are some that I highlighted in the same photo:

Aliens found and highlighted in big colored boxes

AS12–46–6843, the very next photo, with extra-dimensional hyperspace demons highlighted in yellow

Since some of these anomalies can be found outside the captured image (they are homogeneously spread across the image, however they are easier to spot when placed against a black background), it is plausible to infer that no, these lights in the sky aren’t alien crafts nor demons but photographic artifacts likely caused by scratched or otherwise physically disturbed film as can be found in many photos captured using the Kodak Ektachrome film (or any reverse-color film). Just rifling through the Apollo images in high quality can show many artifacts in a similar manner (see appendix).

Ektachrome Layers schematic with Blue sensitive Emulsion layer at the top after the gelatin overcoat

The Ektachrome color film is notorious for producing blue artifacts when scratched due to the Blue-sensitive layer being the topmost one (see forum discussion and this article). Typical color-negative films, scratched, will produce yellowish artifacts, which we recognize as the typical warm film look. Since Ektachrome film is reverse-film (which directly produces positives without negative images), it typically produces blue artifacts.

“On color film, a very light
scratch on the emulsion generally
appears neutral. But going deeper into
the three color layers, it will project
as most any color depending on the
scratch depth, or even white if all the
emulsion was removed. Scratches are
best detected by visual observation
on projection.”

-Kodak’s The Book of Film Care, 2nd Edition 1992, Publication H-23 (page 52)

So very thin scratches (which are the most typical) would produce a blue artifact, somewhat deeper scratches would produce a cyan artifact, and even deeper scratches would produce a white artifact.[1] Notably almost all artifacts captured in the Apollo photos appear to be of one of these colors to varying saturation.

Here's an interactive diagram made by Claude to explain:

Example of film scratches, the blue scratch in particular seems to resemble our blue “streak”

I ended up asking the Apollo Image Archive about these artifacts and they confirmed to me that these artifacts are physical aberrations in the film:

You are correct that those are nicks and scratches on the film. The film was set up on a reel system for astronauts and analysts to study the flight film after each mission, and it’s possible (maybe even likely) that those happened at that time. It’s impossible to say for sure at what point in handling the film it happened, though.

I asked about all the different artifacts found in this film reel however, so I’m not sure this answer covers every case

Others have postulated that these artifacts are the result of high energy particles from cosmic-rays or solar wind, this is partly correct. for the bright dot-like artifacts this seems like a plausible explanation given the artifacts’ morphologies however it is highly improbable that this could explain the repeated ‘streaks’ in AS12–46–6842 (assuming singular particles strike the film homogeneously, the chances of multiple particles striking the film in just the right way to produce what are typical film scratch artifacts are low)

zoom-in on higher quality version of AS12–46–6842's (VM1) artifact

The DoW also provides us with another photo from Apollo 17 with a description that sheds more light on the circumstances of the “disclosure” of these photos:

“As part of the review of historical UAP materials under PURSUE, DOW has opened a case to investigate the accompanying NASA photograph from the Apollo 17 mission, taken December 1972. The image contains three “dots” in a triangular formation in the lower right quadrant of the lunar sky that is clearly visible upon magnification of the image. While this photo has been previously released and discussed by keen observers, there is no consensus about the nature of the anomaly. New preliminary US government analysis suggests the image feature is potentially the result of a physical object in the scene. Additionally, as part of this investigation, the government has obtained the original film from the Apollo 17 mission and the results of the full NASA and DOW analysis will be released when completed.”

nasa-uap-vm6-apollo-17–1972 a 1.7MB JPEG photo from DoW, again of a very low quality[2]

Here, the DoW states that this photo (as all other “disclosed” in this release) has actually already been in the public domain and has been discussed. This image is actually AS17–147–22470, and has also been scanned by ‘March to the Moon’ in much higher quality. Again, we have what seem to be very clear blue scratch artifacts, as this is still Kodak Ektachrome film. Here is another UFO visible in the image, of an even larger size!

Clearly, the US government missed another alien visitor just out of frame

The only problem with this other, bigger artifact is that it’s outside the frame of the actual image. This seems to be another scratch artifact from the Ektachrome film. However the US government claims that:

“New preliminary US government analysis suggests the image feature is potentially the result of a physical object in the scene”

It’s unclear to me what possible analysis could indicate this artifact is the result of a “physical object” given that it’s clearly a film scratch artifact, but maybe the Aliens were the ones who scratched the film themselves?

I’ll leave explaining the rest of the unexplainable phenomena as an exercise for the reader.

Appendix

Here are the rest of the Apollo images, their corresponding title, a higher quality version (sourced from ‘March to the Moon’), and highlighted artifacts elsewhere in the image:

nasa-uap-vm2-apollo-12–1969.jpeg

AS12–46–6846, higher quality with another artifact highlighted

nasa-uap-vm3-apollo-12–1969.jpg

AS12-46-6847, higher quality with another artifact highlighted

nasa-uap-vm4-apollo-12–1969.jpg

AS12-46-6848, higher quality with another artifact highlighted

nasa-uap-vm5-apollo-12–1969.jpg

AS12-46-6849, higher quality with another artifact highlighted

1. Due to removing the yellow dye layer (opposite of blue) first, a scratch could not produce a red artifact by removing dye alone (the same argument holds for standard photographic paper, with the layer order reversed), yet images: AS12-46-6843, AS12-46-6848, AS12-46-6849 seem to show a slight reddish tinge to the outside of the artifacts. This could be due to the digital scanning process (Leica DSW700 scanner used by ‘March to the Moon’) although this seems highly unlikely particularly in AS12-46-6849 which has distinct “patches” of color:

   zoom-in on artifact outside image in AS12-46-6849

   
   

   however red scratches could still be possible if a torn flake of the top (yellow and magenta) layers lands over a spot the scratch cut clean through (see the diagram above). Nevertheless even if we accept the claim that red scratches are impossible given that even seemingly partly red artifacts are visible outside the captured image these are almost certainly still physical film aberrations of some kind. ↩︎

2. The fact that the newly provided images are of a compressed low quality JPEG form when much higher quality versions have been publicly available for years seems rather amateurish at best and potentially suspect at worst.

   “…Whereas previous Administrations have failed to be transparent on this subject, with these new Documents and Videos, the people can decide for themselves, “WHAT THE HELL IS GOING ON?” Have Fun and Enjoy! President DONALD J. TRUMP”

   -trump’s truth social tweet

   Given the lack of context, image quality, non-standardized and unclear naming conventions (VMx? 65_HS1–834228961_62-HQ-83894_Section_2?), limited data (without the actual telemetry and sensor display which would help explain these phenomena), the disclosure of solved phenomena (Astronaut’s eye), this release seems antithetical to the goal of “the people can decide for themselves, “WHAT THE HELL IS GOING ON?””. ↩︎

Discuss