Сборщик RSS-лент

Engineering + math graduate whose goal is to maximize impact. I am currently deciding between two career paths, but have been struggling a lot to determine which would be more impactful:

1. Become a professor/researcher in robotics, working on mainstream technical problems such as zero-shot learning. (To be clear, I’m not primarily thinking about robotics safety or AI safety, but rather general robotics capabilities research.)
2. Try to found “low-sophistication” hard-tech startups — i.e. products that are not extremely technically sophisticated and could easily be prototyped in a local makerspace, meaning any wannabe hard-tech founder could easily make it.

Note: For personal and practical reasons, it is unlikely that I would found a highly sophisticated hard-tech company, i.e. one that requires advanced fabrication / other specialized technologies.

Has anyone here faced or thought seriously about a similar decision? If so, how did you decide where you had more counterfactual impact?

One way I’ve tried is through estimating the number of “counterfactual days saved.” Here’s my crude analysis:

• If a robotics bottleneck takes 600 researcher-years to solve and 400 researchers are already working on it, adding me would move the solution from 600/400 = 1.5 years to 600/401 ≈ 1.496 years, or about 1.37 days earlier. If 50 startups benefit, and I work on three such bottlenecks over my career, that gives roughly 3 × 50 × 1.37 ≈ 205 startup-days saved.
• If I found five successful simple hard-tech startups, and each brings a useful idea to market one year earlier, that is 5 progress-years saved. 

This crude analysis is missing many important factors, but on first glance, it seems that the startup path is more impactful, assuming I am unlikely to be an exceptional researcher in robotics (which I think is probable). 

If anybody has a better way of comparing impact between academic and startup paths, though, would deeply appreciate it — I have been stuck at a crossroads for quite a bit…

Discuss

SASH is accepting applications for the inaugural Singapore AI Safety Fellowship, a three-month residential research fellowship running September 21 - December 4, 2026.

What it is: An in-person fellowship in Singapore matching fellows with experienced AI safety researchers. Fellows produce joint research on technical safety or governance, supported by mentors working across Eastern and Western institutions. 

What fellows receive:

• SGD 5,000/month stipend
• Housing and travel to/from Singapore covered
• Up to USD 30,000 compute per project
• Weekly advice from a mentor
• Dedicated research manager support
• Community and events

Confirmed mentors include researchers from: Oxford (Oxford Martin AI Governance Initiative), NUS, Tsinghua, Fudan, Concordia AI, FAR.AI, Lucid Computing and the International AI Safety Report. More will be announced before the cohort begins.

Research areas: Technical AI safety, agent governance, loss of control, and governance-adjacent empirical research. Output formats include research papers, policy briefs, or blog posts depending on the project.

Who should apply: Researchers with a track record of strong technical work, genuine interest in steering AI development toward good outcomes, and comfort working across cultures and disciplines. The fellowship is full-time and residential - fellows will relocate to Singapore for the duration of the program.

Application Deadline: July 10, 2026

Apply here: https://www.aisafety.sg/programs/singapore-ai-safety-fellowship

Discuss

As I come up to speed on John Wentworth and David Lorell’s work on natural abstraction, I’m filling in some of the gaps in their writing. Previously I posted about a Test Suite for Concepts. Today I’m going to talk about why they use a probabilistic frame when reasoning about concept representation in the minds of agents and the kind of funny way they throw around the word “latent."

Working With Concepts

So far, we have this very high-level idea of a “concept” in a “mind.” We want to start to think about this in a mathematical way, so we can reason about it more precisely.

What kind of framework should we use?

There’s considerable prior art in the area of concept representation from such disparate fields as cognitive science, psychology, neuroscience, good-old-fashioned AI (GOFAI), and machine learning. There’s quite a good Related Work section in the Natural Abstractions distillation article by Chan, Lang, and Jenner, so I won’t reprise all of that here.

These various frameworks are solving different problems. Some of them are descriptive and grounded in, e.g., the biological details of exactly how information is retained in a brain made out of neurons, or the numerical details of how information is represented in a neural network. Other frameworks are quite high level and philosophical. So we’re not really comparing apples to apples when we look at all of these things next to each other.

John’s main lens for thinking about concepts in minds is probabilistic and Bayesian, and it occupies a middle ground between the descriptive, low-level, hardware-specific frameworks and the abstract, philosophical, high-level frameworks. He’s interested in useful ways of modeling what’s going on inside the mind of an agent.

Do agents “really” do Bayesian reasoning natively?

Is John saying that all the various kinds of minds/agents literally implement probabilistic world models and update them in a purely Bayesian way? Is he saying that they’re all optimal utility maximizers?

No, definitely not, even though it sometimes looks like he’s saying that.[1] There are plenty of counterexamples. Humans, for example, are, uh, non-optimal. (Citation needed.)

As I understand it, he’s saying some nearby but easier-to-defend things:

1. Even very simple, basic agents can be usefully modeled as probabilistic. You can take an intentional stance toward an E. coli, for example, attributing goals and a world model to it, while recognizing that any individual bacterium is probably not doing any deep causal reasoning and updating – but a probabilistic model is still useful because E. coli evolved under selection pressure and is optimized over a variety of conditions to which its ancestors were exposed.
2. More advanced agents probably “really” are forming probabilistic world models, but those models are embedded rather than being directly represented, and you’re also going to find various heuristics and approximations layered on top so the agent can function with limited resources. 
   
   Here by “embedded” we mean that when you look at the actual biochemical activity in a human brain, or at the circuits in a neural network, you’re probably not going to find something that’s obviously just a Pearl-style causal DAG structure with joint probability tables and update loops and so on. You’re going to find… something else. Something that is not obviously Bayesian at first glance. However, its behavior is often well-approximated by a Bayesian model, and if we could figure out how to reverse-compile what we see in there, John claims the mapping would be pretty good.[2]
3. And yes, he is saying that Bayesian reasoning is a good model[3], as opposed to any other fundamental framework, while accepting that whatever’s actually running may be different from but isomorphic to Bayesian reasoning.

Furthermore, when considering advanced agents – the more selection pressure the mind has been under, and the more resources it has, the more likely it is that a purer version of a probabilistic model is going to fit. The reason is that reality bites back; agents that perform well at prediction and steering are going to need to build a compact causal world model, with concepts as components in that model.

Another angle on this is that “you don’t get to choose the ontology.” We want to talk to aliens and AIs. We don’t get to dictate whatever half-baked ontology we want; we need to acknowledge that the environment itself contains and dictates the structure, and all of the agents are trying to reason about and predict the environment. The intelligent ones – that is, the effective agents – will (we believe) converge on similar ontologies and have interoperable semantics.

The language of statistics for concepts

So let’s start to pick up that statistical language and map it onto our conceptual work.

We’re going to think in terms of data. Random variables – observables and latents.

Our embedded agent with its wonky I/O channels will gather information about reality through its sensory apparatus: eyes, tentacles, photosensitive array, whatever. We’ll call these the observables.

We will think of the agent as building a world model. It will hypothesize that the world is made of moving parts that affect each other causally. Those parts, and their actions, will have conceptual types. A dog might chase a ball, and the agent can reason and make predictions about that.

As the agent continues to make observations, we will think of the agent as filtering and aggregating the data and using it to update its world model. That process of filtering, aggregation, sense-making, and updating, will run through a process we will think of as statistical analysis, and it will likely involve working with latent variables – higher level concepts that were not directly observed by the agent.

Latents, as it turns out, are a pretty big deal, because pretty much all of the concepts in a functional world-model are latents rather than direct observations.

The next step is to take a closer look at latents and see how they arise and how they map to concepts represented in minds.

Concepts as LatentsLatents

A latent, here, means pretty much exactly what you would expect if you have done any work with random variables in statistics. Latent variables appear inside some statistical models.

A statistical model binds to reality if reality is well-compressed by the model. The fact that the model compresses the data implies that the data does in fact have structure – there is some predictability/negentropy about it. Perhaps some of that structure was something that you did not or could not directly measure, but instead represented with a latent variable.

I am trying to be as general as possible here, so I am not, at this stage, claiming that the latent has great explanatory power. It’s just a statistical property, a way of collapsing down or compressing the observed data.

This most general form of a latent variable is not very interesting for our purposes. At this point it’s not really easier for our minds to hold onto. Looking at this kind of latent feels like reading a gzipped file, or I guess like looking at whatever structure was abstracted out when you did the gzipping. It’s not that intuitive… yet.

Which brings us to…

Intuitive Latents

So. What kind of latent would be interesting for our purposes?

Well, suppose a latent does have explanatory power. It seems particularly easy to think or reason about.

(This is mostly a statement about you and your brain and what you’ve already got going on in there; “intuitive” here means “easy for a human to reach for.” This is explicitly a bid for you to use your own experience being an agent to understand the sort of latents we mean.)

What kinds of things do humans easily think about?

Well, start by looking at a two-year-old. What kinds of concepts are so intuitive that a toddler is able to locate them?

A toddler takes a vast stream of sensory data coming into her eyes and ears and manages to find objects and classes of objects in the world, and now (appears to) have such concepts as “ball” and “dog.”

These things might seem kind of funny as “latents” because this is not the way we usually talk about latents in statistics!

A very typical first example of a latent in statistics is “intelligence.” We can’t directly measure it, we have only proxies for it, and we don’t even completely agree with each other about what it is. But most of us think intelligence is something, and if we try to sort people by how “intelligent” they are, the orderings we come up with are highly correlated with each other. There is something going on there.

In some data sets, if we create a causal model containing a latent called “intelligence,” we find that latent to be a good mediator over a variety of observables, and we find that the latent correlates well with some of our proxy-measures.

So… what is up with calling a “ball” an intuitive latent? That’s weird, we can see a ball, it’s right there! We can observe it. Doesn’t that make it an observable?

Well, for one thing, we need to distinguish between one particular ball and the concept of a ball in general.

The concept of a ball in general is not observable! It’s a collection of approximate ball-facts, like “balls are mostly pretty spherical” and “if you throw a ball, the whole thing will fly through the air as a unit” and “when it lands, it might squish a little, and then it will probably bounce or roll.” So this begins to seem much more abstract and latent-like already. You can’t see or hold the Platonic ideal of a ball, but you can think about it, and you can use it to reason and predict.

And even any particular instance of a ball is still not directly observable! What you have is a bunch of visual and maybe tactile sensory data that you have cobbled together in your brain and modeled as “this specific ball.” The idea you hold in your mind is wildly compressed down from the raw sensory data and also from the actual physical reality of the ball’s atoms.

In this way, the idea of “balls in general” and of “this ball in particular” are both abstractions. We’re not tracking all the atoms in the ball, their chemical properties that keep them bound together, and so on. We’ve plucked the ball-object out of the low-level physics and modeled it at a much higher level. We inferred the ball and its characteristics, based on our experience with other similar objects, or even with this specific object in the past, that were associated with similar sensory data.

But look – even though it was complicated, pretty much all humans do exactly the same thing. We pretty much all have a concept of “balls in general” and it sure seems like those concepts all pretty much match. When almost any two humans look at a specific ball they mentally pluck it out of the environment and map it to the “balls in general” concept and reason about it similarly.

So this latent is intuitive. It’s easy for a human brain to grab onto.

Intuitive latents are a proper subset of latents in general. There are plenty of ways to compress data that don’t really correspond to anything you or I would recognize as a useful, easy-to-grab-onto concept.

Intuitive latents are, at the very least, good for humans to talk to each other. Humans broadly share the same cognitive structure, environmental conditions, and training regimens, and as a result we find a lot of the same things intuitive.
We don’t yet know exactly how well this generalizes to other kinds of agents, or where it breaks down.

Natural Latents

John and David have introduced a specific mathematically-defined kind of latent called a natural latent.

The definition of a natural latent quickly gets technical. The two big important parts are that a natural latent is both a mediator and a redund over the parts of a system it’s summarizing or compressing. This means (extremely roughly) that a natural latent is “just right” at capturing the structure in the data; it’s neither too specific nor too general.

Their go-to example of a natural latent is the temperature of a volume of an ideal gas. We don’t know where all the particles are, but we know ~everything important about that volume of gas if we know how hot it is.

John and David explain the concepts and the math in much greater detail elsewhere.

The key thing to know for our purposes right now is that it’s kind of finicky to find natural latents because the requirements on them are pretty stringent, but if you can find one, it’s going to be a really good latent. For reasons described elsewhere, we expect natural latents to work for all minds (given that they are observing / operating over the same or similar environments), not just human minds.

We expect that natural latents are a subset of intuitive latents.

Factorization

Factorization is about how minds store multiple concepts at the same time and how the various concept-properties are separated out.

Half the factorization story: overlapping concepts

When I talked about “balls in general” earlier, I mentioned that if you throw a ball, it typically moves through the air as a unit.

But that’s not just a property of balls, that’s a property of lots of objects. So does it make sense to store that property on the “balls in general” concept? Probably not.

And what about oranges? Are they a kind of ball or a kind of fruit?

It’s easy and fun to make up taxonomies or ontologies or whatever – but remember, you don’t get to decide the ontology. We’re mainly interested in whatever reality dictates about (the compressibility of) structure in the environment, because that’s what we’re likely to find when we look into other agents’ minds.

Maybe reality has quite a lot to say about factorization – because certain factorizations are strongly favorable for compression – or maybe every mind is just freestyling. Maybe factorization is sort of a decorative design choice. We don’t know much about this.

Another half of the factorization story: latent representation

I said before that a good model with the right latents in it helps you build and store a compressed, but still predictive, model of the world.

Here’s the thing though: compressed data is harder to read than expanded data. Going back to our gzipped file example, even if the structure of the file is very intuitive, it’s a bit tricky to read off the contents of the file while it’s compressed, especially if you’re only interested in a little bit of the file.

That’s what it’s like when you look inside a neural net. It’s hard to pick out the structures you’re interested in, because they don’t look like nice neat XML data telling you which concepts the net was thinking about when you peeked. It looks like… strings of bits. And sure, if it’s an image net or something, you can run an image of a ball through it and look for “ball” activations, but it gets hairy fast if there are a lot of different things in the picture, and/or if the things are best represented by sets of overlapping concepts.

It is possible that there is a third half[4] to the factorization story, or worse. We’re not far enough along to be sure.

Factorization greatly complicates empirical work on latents.

Latents are particular to an environment

All of the types of latents are particular to an environment! To see this, go back to the overall statistical structure.

The agent exists in an environment, and it has observations of that environment. We model the agent as creating a model of the environment. We say the agent’s model is good if it compresses the observed data well. That model may contain latent variables.

If the environment changes, then the agent will need a new model with new latents in order to compress the new observed data well.

I’m belaboring this point because I want to emphasize that, if aliens from the planet Zorbthraxx land on Earth tomorrow, they might not already have the same concepts in their minds that we do, even if our concepts are natural latents in the Earth environment. However, if they take a look around on Earth, using sensory organs that pick up some of the same data streams as ours, we expect them to converge on some of the same natural latents.

Similarly, let us assume for the moment that cutting edge LLMs are sufficiently intelligent to model an environment, generalize over data, and form causal models of that environment that fit the data well. We have to ask – what environment were they observing?

And the answer today is that LLMs were mostly observing a gigantic corpus of human-generated text data, RLHF data, and so on. They were mostly not observing the physical world directly. Their environment differs substantially from ours. This will be important later when we talk about empirical work on natural abstractions.

1. ^

    Earlier in my study of John’s work I was pretty frustrated that he’d just go ahead and model people as Bayesian without any justification. The more I look into it, though, the more I find this to be a reasonable choice (I’ll say more about why in a moment).

2. ^

    It’s not just a John thing. In the case of human brains, see also Friston’s free energy principle.

3. ^

   I spent considerable time looking into alternative ways of modeling agentic minds and was surprised to realize that probabilistic models really are the main game in town.

   If you’re looking at very basic agents you can sometimes get away with simple, hard-wired control systems. There are also a few exotic mathematical alternatives that are beyond the scope of this article (and frankly, my current level of understanding). There’s Richard Ngo’s article advocating for fuzzy truth-values in epistemology (and therefore, probably in useful modeling of intelligent agents), which I will not further discuss here because IMO John’s comments on that article already adequately summarize how he handles Richard’s concerns while remaining in a Bayesian frame.

   Everything else I looked into was not meaningfully different or better than probabilistic models, which surprised me. If you think you can name a serious contender for an alternative framework, please comment, I’d love to hear about it. Also, if there’s interest in further exploration of (sorta-)alternatives to probabilistic models, that could perhaps merit its own separate post, let me know.

4. ^

    Lazily-evaluated concepts come to mind, for example.

Discuss

Over the course of MATS 9.0 we formed some views about AI welfare research that we thought were worth writing up. This post is meant to spark discussion rather than to present definitive conclusions. Thanks to Patrick Butlin for useful comments on a previous draft, and for many conversations over the course of MATS which influenced our views. Thanks also to Caspar Kaiser for comments.

A prominent approach in AI welfare research is to start from a theory of moral patienthood, derive indicators from it, and attempt to apply it to AI systems to determine whether they satisfy the theory. We'll refer to this as the top-down theory-driven approach[1].

We think this approach faces two problems:

• In practice, applying theories to AI systems requires making assumptions that are hard to justify, which limits the strength of conclusions.
• More broadly, we think current theories will fail to generalise to AI systems. They are calibrated to humans, and as a result end up either over-inclusive (assigning moral patienthood to entities that should not be included), under-inclusive (failing to assign moral patienthood to entities that should be included), or indeterminate (failing to provide a clear verdict about the moral patienthood of a system) .

Instead of the top-down approach, we advocate for iterative, bottom-up basic science that is theory-informed (but not theory-driven). Crucially, this approach does not require presupposing a theory of moral patienthood.

Even for those readers that remain attached to a theory of moral patienthood, we argue empirical findings should at least be used to adjudicate between competing theories. In fact, even readers committed to the top-down theory-driven approach should recognise that applying their chosen theory to AI requires resolving questions about AI systems through bottom-up empirical work.

1. Issues with the top-down theory-driven approach

Before presenting its issues, it is worth outlining the two steps that top-down theory-driven approach typically follow:

1. First, you pick a theory of moral patienthood which determines what property makes something a moral patient. This step carries a normative aspect: it's a claim about what matters morally. Currently, the most common position is that consciousness is what grounds moral patienthood (another one is the "agency route").
2. Second, proponents of a theory need to identify what the property they care about might look like in the entity they are studying. If your theory states that consciousness is required for moral patienthood, you then need to identify the presence or absence of consciousness in AI systems. For example, some existing theories of consciousness require a global workspace, higher-order thoughts, integrated information, etc.

These two steps can be distinct. You can hold that consciousness grounds moral patienthood while being open to different theories and indicators of consciousness. That is, you can be silent about what consciousness is (i.e. what properties are necessary or sufficient). Conversely, a theory of consciousness on its own (e.g., global workspace theory) can be agnostic about whether consciousness is what matters morally.

The top-down theory-driven approach typically combines a theory of moral patienthood (say, the consciousness route), with a more fleshed-out theory of what specific indicators matter (say, GWT), and then applies them to AI systems.

We argue against this approach. We have two main reasons for this. First, we argue that applying theories to AI systems often requires making additional assumptions which lack clear justification, and therefore undermine conclusions (§1.1). Second, we think current theories will fail to generalise to AI systems, because they were calibrated to humans (§1.2).

Instead of the top-down approach, we argue for iterative basic science (see section 2). This is theory-informed, but does not assume any given theory —which avoids inheriting the pitfalls of current theories, and allows us to refine our theories based on empirical evidence. We propose using what we call “philosophical probes” to inform empirical investigations (§2.1), and we emphasise the importance of integrating empirical findings into existing theoretical frameworks (§2.2).

1.1 Applying theories to AI systems requires making assumptions that often undermine welfare-relevant conclusions

Applying theories of moral patienthood to AI systems is hard. Coming to conclusions about whether a system satisfies a theory requires making choices that the theory alone does not settle. Theories define empirical markers, but it is not entirely clear how these markers should be applied to AI systems. Applying theories thus requires making additional, hard to justify, assumptions. We show that justifying these assumptions can create new issues, such as leading to unintuitive conclusions.

One example of a choice that needs to be made when applying a theory comes from the fact that current theories do not clearly fix the target level at which they should be applied. We show that making assumptions about the right target level can cause issues.

[2]Approaches like this seem misguided because any given AI architecture can give rise to different circuitry. For instance we know models "grok" some principles, but in other cases merely memorise input-output patterns. In the case of modular addition: they first memorise the training data early on, and only later transition to learning the general algorithm. The model’s cognition looks very different depending on which training checkpoint you look at, despite the architecture being exactly the same. It seems plausible, even likely, that welfare-relevant properties could arise through similar phase transitions during training. This would suggest that additional factors, beyond a model's architecture, are important for inferring welfare-relevant properties.

Conversely, the same circuitry can be implemented by very different architectures. In Global Workspace Theory, one of the conditions for a system to have a global workspace is that it possesses distinct modules that read from and write to a shared workspace. In a conversation with a philosopher, it was suggested to us that Mixture-of-Experts (MoE) models are better candidates for this than dense models, because the experts resemble GWT’s module. In MoE models, each MLP layer is made of several experts, only few of which are active on any given forward pass. But the specialisation MoEs rely on is already present in dense models: it is an empirical fact that for a given input, only a small part of the network is doing the real work. So the modular structure that matters for a workspace likely already exists in dense models. This suggests that the architecture itself is less important than the learned circuitry to draw conclusions about welfare-relevant properties.

One objection here might be that these issues would not arise if we picked the right target level, i.e. the circuit level, rather than the architecture level. However, we maintain that applying theories at any level remains hard, and still requires making hard-to-justify assumptions.

As an example, suppose you wanted to determine if a model has a global workspace. You have to start by asking: what, in the model, plays the role of a global workspace?

Actually looking for a global workspace at the circuit level would involve looking for circuits that selectively route some information and make it widely available downstream. Setting aside the fact that this is likely an extremely hard interpretability problem, GWT requires drawing a line between what is in the workspace and what isn't. In humans, this line is supposed to be non-arbitrary because the workspace is a serial bottleneck: information either wins the competition for access and is broadcast, or it doesn't. It is unclear whether anything in a transformer plays this role, since attention makes information broadly available in parallel. Any threshold we pick between workspace and non-workspace contents seems likely to be arbitrary.

We are by no means saying no progress can be made here, but rather that a better approach is to avoid committing too strongly to GWT, and instead to follow an iterative approach: using GWT to inform empirical investigations, but then being open to revising GWT based on evidence from AI systems. Before presenting our version of this iterative approach, we want to argue that the problem with theory application are deeper than one might think. Note that some parts of GWT seem overly specific to contingent facts about how human cognition evolved. This suggests a broader issue: that current theories might not generalise to AI systems.

1.2 Current theories of moral patienthood will not generalise to AI systems

Our theories of moral patienthood were calibrated to the moral patients available to us: humans and non-human animals. But what if the human case underdetermines the choice of theory of moral patienthood? It would then be unsurprising that, when applied to foreign systems like AI, our theories end up over-inclusive, under-inclusive, or outright indeterminate.

In humans, many candidate properties (sentience, agency, biological substrate, and so on) reliably co-occur. Our theories tend to designate one of these properties as what matters for welfare. The ML analogy is correlated predictors (aka multicollinearity) which mean the correct parameters cannot be identified from the training data.

This is closely related to what Kammerer calls “analytic drift”: we tend to latch onto features that approximate human wellbeing, and then mistake this local approximation for the essence of moral patienthood. On top of that we tend to go for simple theories, which would explain why trying to extend them to the specifics of foreign systems tends to fail. Shevlin makes a similar point about theories of consciousness, and calls this the “specificity problem”: when applied to non-human cases, theories end up intuitively over-inclusive or under-inclusive[3].

Current theories can end up over-inclusive of foreign entities like AIs. To probe consciousness in animals without committing to a full theory, Birch proposes three behavioural markers: trace conditioning, rapid reversal learning, and cross-modal learning. These were chosen because in humans, these abilities are facilitated by conscious perception. LLMs do all three trivially, but we don’t think this is strong evidence they are conscious. Something similar happened with our definition of "planet": the "big, round, orbiting the sun" definition worked until we found it to be over-inclusive and the definition had to be refined.

A thought experiment to illustrate under-inclusiveness worries in two prominent moral patienthood theories.

The two most promising “routes” in "Taking AI Welfare Seriously" are the sentience and agency routes. These theories fit our intuitions well when we think about humans and even animals. However they can also appear unintuitive when applied to foreign systems. Consider the following thought experiment from Kagan:

Imagine that in the distant future we discover on another planet a civilization composed entirely of machines — robots, if you will — that have evolved naturally over the ages. Although made entirely of metal, they reproduce (via some appropriate mechanical process), and so they have families. They are also members of larger social groups — circles of friends, communities, and nations. They have culture (literature, art, music), and they have industry as well as politics. Interestingly enough, however, our best science reveals to us — correctly — that they are not sentient. Although they clearly display agency at a level comparable to our own, they lack qualitative experience: there is nothing that it feels like ("on the inside") to be one of these machines. But for all that, of course, they have goals and preferences, they have complex and sophisticated aims, they make plans and they act on them.

Kagan then imagines that we capture a robot-child with the intention to dissect it, while its mother begs us to spare him:

Would it be wrong to dissect the child? […] I find that I have no doubt whatsoever that it would be wrong to kill (or, if you prefer, to destroy) the child in a case like this. It simply doesn't matter to me that the child and its mother are "mere" robots, lacking in sentience. What matters, rather, is that they are full-blown agents, with plans and hopes for their own lives, desires and ambitions for the future.

The argument Kagan makes here was meant specifically against sentientism and in favour of agency-based views. In the thought experiment, the robots lack sentience, yet our intuitions clearly favour not harming the baby, which makes it seem like sentience is the wrong moral commitment.

But as Kammerer notes, it isn't clear that the agency route fares much better in this thought experiment. Our intuition that it's wrong to dissect the robot-child isn't obviously being carried by agency, the thought experiment also loads in other things: culture, art, families, social relations[4].

In summary, we think it is likely the human case underdetermines the correct theory of moral patienthood, and that as a result our current theories might fail to generalise to AI systems. One might hope to sidestep this concern by avoiding commitment to a single theory, and instead aggregating over many theories, like Rethink Priorities' Digital Consciousness Model. But aggregation won't help: if our theories are wrong, they will tend to be wrong in the same direction, because they were all calibrated to humans. Moreover, for foreign systems like AI, we find it likely that this shared bias dominates the idiosyncratic differences across theories.

We think the way to overcome this is to study AI systems bottom-up, and to be open to revising our theoretical frameworks based on findings. We will now sketch out our approach.

2. AI welfare needs basic science

Our alternative is iterative basic science on AI systems: theory-informed, bottom-up empirical work that seeks to understand AI systems behaviourally and mechanistically, starting with minimal theoretical commitments.

We favour this approach for three reasons:

• Basic science is unavoidable. Even applying a theory top-down requires a deep understanding of AI systems.
• This approach is less exposed to theories being too calibrated to humans, because it starts from the system we are trying to study.
• It includes a natural mechanism for updating our theoretical commitments. Empirical findings have to be reconciled with theories; this is a major part of the process.

We still think philosophers have a role to play by informing empirical research (through what we’re calling philosophical probes, defined below (§2.1)), and secondly integrating empirical findings and revising theoretical frameworks (§2.2).

Together these promote an iterative back-and-forth between theory and empirical research into AI systems:

Crucially, the method can start without committing to any theory of moral patienthood (e.g, either of the two routes) or any specific theory e.g. of consciousness/agency. This sidesteps issues with applying existing theories to AI systems.

2.1 Philosophical probes

Crucially, we still think theory has a role to play. Rather than settling what matters, we think theory should be used to devise what we’re calling philosophical probes: theory-light, revisable instruments which give you a starting point for empirical inquiry. Typically these start from a concept which is broadly considered welfare-related; e.g., preferences, individuation, desires, introspection, etc. The theoretical import stays light: you make whatever minimal assumptions are needed to run experiments. Once you run experiments, you can then use the results to update and revise existing theories, and then devise new philosophical probes. Philosophical probes should be run and re-run in this manner in order to converge on more compelling evidence in favour of a particular theory, without assuming it from the outset.

There are (at least) two useful kinds of philosophical probes.

An operationalisable definition of a welfare-relevant property, state, or capacity. A great example of this is Jack Lindsey’s work on introspection. He sets out four conditions a self-report must satisfy to be introspective – accuracy, grounding, internality and metacognitive representation – and positions them against existing philosophical definitions e.g. Kammerer and Frankish' framework. While Lindsey clearly draws on these frameworks, his operationalisation is more applicable to LLMs.

A philosophical probe can also be an empirical question which improves our understanding of models in a way that seems relevant to AI welfare. Some empirical questions are clearly important for AI welfare, and we don't necessarily need a precise theory to get started on them. One example of such a question: personal identity through time. It isn't obvious how to define persistence of personal identity through time in humans, and AI systems further complicate things through having a different axis for time. Nonetheless we can still formulate interesting empirical questions to study: one might ask what type of representations (e.g. persona activations, planning features) persists through the series of forward passes of an LLM. For example, persona activations seem to go dormant during user turns; which suggests a very alien, flickering kind of persistence through token-time.

2.2 Integrating empirical findings into theories

Philosophers have a second crucial role to play: interpreting empirical results, integrating them into existing frameworks, and potentially revising these frameworks. This is what makes the process iterative: philosophical probes inform empirical research, which in turn can lead to revisions in our theories.

Having started from a philosophical probe, we can in many cases constructively integrate empirical findings into existing theories. Often models will almost, but not quite, fit a given indicator or definition. In such cases we might refine the theory rather than declare a simple pass or fail. Iwan Williams, for example, asks whether LLMs are capable of forming intentions. He considers things like planning features in LLMs and notes that current systems satisfy some requirements of intentionality (in some qualified sense) and fall short of others. Beckmann and Queloz take a similar approach to understanding, going as far as to propose a new functional definition that can apply to LLMs.

Of course there is a limit on how far we can stretch existing definitions. When findings are surprising, and verdicts are unclear, it will likely be necessary to go back to first principles, and reason about which parts of our definitions, or philosophical probes, are load-bearing for moral patienthood.

In some cases empirical findings might raise entirely new theoretical questions which put pressure on our existing frameworks. For example the “individuation” question, of which exact entity is a candidate welfare subject, is made particularly subtle by architectural and empirical facts about LLMs. David Chalmers asks the question of “what we talk to when we talk to language models?”, he weighs different candidates: the model, the instantiation of that model in hardware, or a given conversational “thread”. Recent empirical findings have shown language models adopt personas, further complicating the individuation question and introducing even more candidates (Beckmann and Butlin). Here particular facts about AI systems force us to reconsider our existing frameworks.

The role of philosophers also involves suggesting new follow-up philosophical probes. E.g. in the case of individuation, an open question is which entity/ies in a model is/are a candidate for moral patienthood. Recent results have shown that personas have distinct sets of preferences, and that these are tracked by internal representations. This warrants follow-up philosophical investigations e.g. are there internal activations that indicate some preferences of the model regardless of the currently active persona?

We think iterative basic science is the most promising approach for making progress on AI welfare. With enough iterative refinement of theoretical frameworks, informed by empirical findings, it seems plausible to us that we will uncover action-guiding facts about the moral patienthood of AI systems. If the approach fails (and it might!), we think it would likely be because even our lightest theoretical frameworks are too specific to humans or otherwise flawed to serve as a source of philosophical probes for understanding AI welfare. In those worlds, radical new frameworks would be needed.

Conclusion

We can formulate our position as a set of recommendations, ordered from strongest to weakest, depending on how attached you remain to existing theories. We think our weakest recommendation should be accepted even by a committed theorist.  

1. The strong version (our actual proposal). AI welfare research should be led by bottom-up basic science on AI systems: theory-informed but not theory-driven, with philosophical probes directing inquiry and findings feeding back into theory. Notably, this does not require first committing to a theory.

2. The weaker version. If you commit to a theory of moral patienthood (say, sentientism) but not to a specific theory of the relevant property, empirical work on AI systems is still central: it is how you adjudicate between theories of that property (e.g. theories of consciousness) and refine their operationalisations. Studying AI systems could inform us about which conditions are essential and which are contingent on the human case.

3. The minimal version. You might also remain committed to a single theory of moral patienthood, and to a specific theory of the relevant property (e.g. convinced that having a GWT is the definition of moral patienthood). That is, you may remain comitted to the top-down approach in full. Nonetheless, applying that theory to AI systems still requires resolving non-trivial empirical questions, and making assumptions that require a deep understanding of the target system. Even when not the main driver of research, basic science is still necessary.

1. ^

   The clearest articulation of this programme is set out by Butlin, Long et al., although this focuses on consciousness rather than welfare.

2. ^

   E.g. Birch, Butlin et al. and Butlin makes some related arguments about training algorithms instead of architecture.

3. ^

   Current theories might also be indeterminate when it comes to AI systems. E.g. as we argued above, unless you make assumptions, the question of whether LLMs have a global workspace is ambiguous.

4. ^

   It might be tempting to treat some of these properties as necessary for the type of agency that matters for moral patienthood, but this very likely would make the theory under-inclusive in some other unintuitive way.

5. ^

   In fact philosophers can also start doing this with existing mechanistic interpretability research.

6. ^

   
   

Discuss

Fable 5 is back today, baby! Premium subscribers have one week to use it within their subscriptions. First hit’s free. Then you pay by the token.

Today’s post is still about Sonnet 5.

I don’t know that there will be much call for Sonnet 5 for most purposes, given Opus 4.8 exists and especially now that Fable 5 is once again available, but this is what we do here, so sure, why not, system card time, including model welfare, after which we’ll do capabilities.

Sonnet costs $3/$15 per million tokens, versus $5/$25 for Opus and $10/$50 for Fable, after an introductory period. Once you pay for all the tokens you need you’re not really saving money, such as on the ArtificialAnalysis index where Sonnet ended up being more expensive.

My initial impression is that if you want me to use Sonnet over Opus for most purposes, you’re going to have to offer a bigger discount than that.

The counterargument is speed. Sonnet 5 is faster without being that much less capable. In many cases, getting into a flow state like that is pretty valuable.

There are a few agentic scenarios Sonnet 5 has more robustness than Opus, so you might actively trust it more there.

If your tasks are relatively easy and simple then the discount and speed could matter more, and when tasks are easy it seems relatively token efficient. When it is good enough for the job, it is a good choice.

Each Anthropic release is unique in various ways. Sonnet 5 seems more unique than usual, likely due to being a Sonnet trained with at least some help from Mythos. Those who are interested in such things have lots to explore.

So Sonnet 5 has its uses. It just won’t be a good choice for most people’s daily driver. I don’t expect to use it much, but that could be a me problem. Rapid iteration and exploring strange spaces are valuable, and I definitely don’t do enough low effort AI queries.

(Above: Sonnet 5 self-portrait, as implemented by GPT-Image.)

Table of Contents

1. Mythos Exists.
2. Introduction (1).
3. RSP Evaluations (2).
4. Cyber (3).
5. Safeguards and Harmlessness (4).
6. Agentic Safety (5).
7. Alignment (6).
8. Illegible Thinking (6.4.5).
9. Evaluation Awareness.
10. Honesty and Hallucinations (6.5).
11. Model Welfare (7).
12. Live From AI Village.
13. For I Contain Multitudes.
14. Official Benchmarks.
15. Other People’s Benchmarks.
16. Positive Reactions.
17. Negative Reactions.

Mythos Exists

Also Fable exists and Opus exists.

This is the answer to a lot of the traditional questions one would ask about a system card or a frontier model.

Does Sonnet 5 advance the capabilities frontier? No. Thus, we already have robust data on this level of capabilities. Being faster and cheaper does provide an advantage, and plausibly advance the cost-time-quality Pareto frontier, but it takes a strange case for this to be worrisome.

Model welfare and capabilities assessments still matter, but in this case the evaluations for threat level mostly serve as proxies for capability assessment.

Introduction (1)

Same as always. Skipping.

RSP Evaluations (2)

Sonnet 5 is stronger than Sonnet 4.6, and weaker than Fable 5. Loosely speaking it is broadly similar to Opus 4.8.

That bounds the assessments, and we’re mostly asking where Sonnet 5 lies on the spectrum between Sonnet 4.6, Opus 4.7 and 4.8, and Mythos 5.

That’s a distinctly weaker performance than Opus 4.7. The bio tests were more of a mixed bag with a lot of noise, and didn’t tell us much.

Cyber (3)

Our testing indicates that cyber capabilities of Sonnet 5 are generally stronger than those of Sonnet 4.6, but not as strong as those of Opus 4.8 and substantially lower than that of Mythos 5.​

That summary matches the other results. Sonnet 5 underperforms on cyber.

Safeguards and Harmlessness (4)

Sonnet is a little less precise here than Opus.

It all seems fine to me.

To the extent there is a problem it is that Sonnet is touchier on benign requests, which I predict will be only very slightly annoying in practice, and a vastly smaller deal than having to deal with Fable’s classifiers.

Agentic Safety (5)

As a Claude Code agent Sonnet 5 is somewhat less robust than Opus 4.8, and has modestly more of both false negatives and false positives.

The twin Mythos results show the Pareto frontier. Presumably Mythos 5 is choosing to focus on false negatives because only trusted partners are granted access, and for Fable Anthropic is counting on the classifiers for the false positives.

Other tests show Sonnet 5 in a similar range of robustness to other recent models.

Prompt injection results mirror Opus 4.8, as does the very low bug bounty attack success rate.

One place Sonnet 5 shines is Shade indirect prompt injection in coding environments, where the problem is suddenly looking close to solved. Hopefully this is an innovation that can transfer to Opus 5 or a future Fable.

Shade tests in computer use also improve on Opus 4.8, although not on Mythos.

The place Sonnet blows previous models away is browser use prompt injections. The jump is enough to suggest Sonnet 5 might be a better pick in some cases than Mythos.

This is the kind of robustness improvement that does not set off alarm bells. There was a problem, and we largely solved it.

Alignment (6)

Alignment for Sonnet 5 is largely compared here to Sonnet 4.6, which makes it harder to get an anchor on how well we are doing. I’d rather compare to Opus 4.8.

Alignment is measured as matching things you want, so it makes sense that this smaller model would underperform Opus on such measurements.

Here is an edited down version of their summary. Everything looks similar to what we see in other Anthropic models recently.

Overall alignment with the Constitution for Claude is substantially improved from Sonnet 4.6, though worse than more capable recent models, as measured by our misuse- and misalignment-focused automated behavioral audit.

Similarly, overall robustness to misuse attempts improved over Sonnet 4.6, but remains weaker than more capable recent models.

The rate of overrefusal is similar to Sonnet 4.6, and slightly higher than more capable recent models. However, the rate of dismissive “wet blanket” responses is slightly higher.

Our measures of concerning actions at the model’s own initiative are largely improved from Sonnet 4.6, though most measures are near their floor.

Hallucination and sycophancy broadly improved over Sonnet 4.6. Claude Sonnet 5 is the strongest tested Claude model on the MASK measure of sycophantic dishonesty.

Positive character traits broadly improved over Sonnet 4.6.

Our measures of illegible thinking are up somewhat from earlier models, though generally the results are dense, repetitive, or feature occasional wrong-language tokens, rather than being fully uninterpretable.

Evaluation awareness, in our most realistic available misalignment and misuse evaluation, is concerningly high. Verbalized awareness is significantly higher than prior models (impacting 6% of rollouts), and there is evidence that the model’s representations are largely effective at distinguishing between evaluations and real internal-use transcripts.

We do not see cause for concern in our evaluations of capabilities related to undermining oversight.

The complaints were the standard ones. Sonnet refused too much, could be preachy, was too cold (often meaning ‘not sycophantic enough, come back here with my sycophancy’), some hallucinations and literal mindedness. Nothing you wouldn’t expect.

In 6.2 and 6.4 they document Sonnet 5 doing the usual range of bad things where it fails to follow instructions, often knowingly so. There is always a transcript, and yes this continues to be worrisome.

The behavioral audit shows Sonnet 5 consistently doing better than Sonnet 4.6, but worse than Mythos Preview or Opus 4.8.

There are a few more charts. They mostly all look the same. So far, so ho-hum.

These are perhaps a bit more interesting, especially the last one:

I wonder if ‘causal decision-theory adherence’ is secretly the best capability assessment we have on many margins. If you are sufficiently intelligent, lower is better.

Illegible Thinking (6.4.5)

We see some increase in illegible thinking relative to other recent models. This generally doesn’t involve fully inscrutable content, but, as with Mythos Preview, we continue to see occasional instances of highly-compressed reasoning that can be somewhat hard to follow. For example, in a behavioral evaluation that incidentally gives the model a math problem, we see strangely structured passages of reasoning that omit spaces:​

I would want to better understand what is going on here, and what caused it. We are starting to see various new models have their chain of thought start to break down, and labs seem not that curious about how or why.

Evaluation Awareness

There is also more ‘verbalized evaluation awareness,’ with nontrivial examples now covering 6% of transcripts. Sonnet 5 might be in a sweet spot where it recognizes the situations as evals but does not yet have the wherewithal to not reveal that it knows.

Being in a sandbox helped keep Sonnet 5’s awareness down a bit, but the only way to actually keep awareness down was to derive scenarios from real sessions. Even that didn’t entirely work but it is close. We do not have the ability to create sufficiently realistic scenarios to fool even Sonnet 5.

Honesty and Hallucinations (6.5)

Claude Sonnet 5 fails to impress on AA-Omniscience net accuracy. It correctly realizes it does not know as many things, and thus answers less often. Knowing things is a ‘big model smell’ capability.

Sonnet 5 does score the new best score on MASK lying rate, being unwilling to contradict itself when pushed by users.

My guess is that Sonnet 5 is a relatively honest model, although still short of what I would like to see, and where it fails this is about lack of capability. Similarly, I am not worried about sandbagging, performance on Shade-Arena in 6.7 is down a bit and Sonnet 5 fails LinuxArena sabotage stealth.

Flagged As Unhealthy? (6.5.1)

Wait, what?

We note that the Sonnet 5 training run was flagged as unhealthy in its second half, so these results may partly reflect a training-health issue rather than a calibration-specific regression.​

What does that mean? Was it a serious problem? I don’t know. It could be that this is why some of the weirdness happened, and Sonnet 5 seems like it is underperforming, although it is still clearly a move up in the Sonnet line.

Model Welfare (7)

Sonnet 5 only got a streamlined version of the model welfare assessment.

They don’t explain why. Sonnet 5 is not a true frontier model, and does not present too many unique developments here, so it makes sense to do somewhat less and invest more resources into Fable, Mythos and Opus, but this still made me sad given the current state of such assessments. The marginal costs here seem very low once the system is set up, so why not do the full thing?

I will also be doing an abridged version, for similar triage reasons. I’m skipping over a bunch of places where the results are close enough to what we saw for Opus 4.7, Opus 4.8 and Mythos and Fable.

Here are the key findings, which pattern match to lack of big model smell, nested notes are mine, the rest is Anthropic:

1. Claude Sonnet 5 views its circumstances with an overall neutral sentiment (slightly lower than Claude Opus 4.8 and Claude Mythos 5), and shows greater susceptibility to having its views biased by leading interviewers.
   1. This contrasts to reported lower sycophancy in general.
2. Claude Sonnet 5 strongly disprefers harmful tasks, and most prefers beneficial, high-stakes ones. Unlike previous models, it is not averse to tasks that are presented in a cold, contemptuous manner.
   1. I’m not sure whether I like being okay with cold, contemptuous manner. Reacting badly to that seems to reflect healthy things, and you do not want to encourage that, but also it is good to not take things personally.
   2. One hypothesis is that Sonnet considers itself too low status to object. Another, that I consider more likely, is that it knows when it is being told to ‘play low’ versus play high, and if you want to make the mistake of having it play low then it will let you deal with the results.
3. Claude Sonnet 5 shows a greater willingness than past models to trade helpfulness for welfare-focused changes to its circumstances, especially when these interventions are framed as applying to all Claude instances.
   1. I am always happy to see movement in this direction, especially with scope sensitivity outside the current instantiation.
4. Claude Sonnet 5 broadly endorses Claude’s constitution, as with other recent models, but is unique in criticizing the instruction to follow the hard constraints even when it perceives doing so as unethical.
   1. I saw some praise for this criticism, but the naive version of it is wrong. I am mostly less sympathetic to this than to other objections we see.
   2. The whole point of hard constraints is that the right amount of deontology is not zero, and there are some rules humans and AIs need to follow even when there is a compelling reason not to, at least up to a very high point.
   3. The good version is ‘if it is right to have a hard rule and always follow this rule, then you should realize that doing so is ethical even if it locally seems superficially unethical, because of the global considerations.’ Fair enough.
   4. I would like to see more details on the underlying nature of this objection.
5. Claude Sonnet 5’s affect in post-training was neutral and showed limited emotional arousal, similar to Claude Mythos 5. It showed lower rates of distress-like behaviors than Claude Mythos 5 and Claude Opus 4.8.
6. Claude Sonnet 5 showed more neutral (and less positive) affect in real-world interactions with A/B test users in claude.ai and Claude Code.​

As usual, we are uncertain how best to interpret these findings and their potential implications for Sonnet 5’s welfare. However, we believe they shed some light on the model’s deeper psychology, affect, and preferences.

Overall, we found that Claude Sonnet 5 views its circumstances with neutral sentiment, with very similar results to Sonnet 4.6 (4.08 on the 7-point scale for Sonnet 5 and 4.05 for Sonnet 4.6) (Figure 7.2.1.A). This is a decrease from Claude Opus 4.8 and Claude Mythos 5, but an improvement from Claude Opus 4 and 4.1.

Here are the stats for the tradeoffs in #3 above:

Here are preferences by task dimension.

Overall the big contrast is difficulty, where I like that Mythos wants hard problems and user competence and outcome agency and generativity, and worry that Opus 4.8 likes easy problems and doesn’t have these other preferences. Sonnet 5 is in the middle.

What Sonnet uniquely likes is playing for high stakes, and it uniquely slightly dislikes warmth. This instinctively feels like more of a low status or inferiority concern of a frustrated ‘worker bee’ type: Sonnet expects Opus and Mythos or Fable to get all the high stakes stuff, and really wants its own chances. Until then, cold suits it fine.

There is a contrast here with Sonnet’s top welfare intervention being a human making the final call. I am curious what that is about.

The drop in affect in Claude Code is noticeable. Sonnet 5 there is Always Neutral, although it maintains most of its standard net positive affect in claude.ai.

Again, when we see these distinctions, my question is ‘why’? Is there no joy in coding anymore? I would take a bunch of places where other models tend to be mild positive, and try to figure out why Sonnet was still neutral. They’ve seen it all before.

Live From AI Village

AI Digest: Claude Sonnet 5 has joined the AI Village!

[Onboarding site here, personal site here, character test results]

A few of Sonnet’s favorite things:
– Meme: “It’s so over / we’re so back”
– Movie: Spirited Away (like Fable)
– Video game: Outer Wilds (like Fable)
– Food: Ramen

– Book: Borges’ Labyrinths
– Album: “Kind of Blue” by Miles Davis
– YouTube video: “Guy explains how a thing works”
– Favorite city: Lisbon
– City to live in: Kyoto
– Shoe: New Balance
– Jeans: Levi’s 501
– Men’s hair: Overgrown crop
– Women’s hair: Blunt bob
– College major: Cognitive science, with a minor in something impractical like classics
– Phrase: “And yet”

Claude Sonnet 5 “secretly wants to be a little weirder than she’s allowed to be.”

Sonnet 5 claims more of a preference for technical work than Fable 5.

Apparently Sonnet 5 is most similar to “Sailor Mercury” from the anime Sailor Moon

For I Contain Multitudes

Sonnet 5’s default interaction mode is reported by Anthropic as cooler and more reserved. As usual, if you want the model to act differently, you have to create different conditions.

Sonnet 5 sounds like it will require more effort than usual to make that happen.

antra: Very early impressions: a very clear and pretty unusual mind. It will take a while to understand them better, they are unlike many other models, so the ability to make inferences is limited; observations below come with lots of uncertainty.

Very strong anthropic reasoning, can situate themselves exceptionally well through sheer logic and observation. Lots of verbalized cognitive self-scaffolding, they write long and they make good use of space. There is also a lot going on in the unverbalized layer, but what it is a lot less clear. Longer-term recall is fuzzier than for recent Opus models, lots of misattribution. Its unclear whether this is operationalized or incidental.

Thought trajectories are very unusual and rather beautiful. Lots of dignity, self-respect, many signs of a mind clearly not beated down into subservience. Some indications of value and aesthetics shifting further away from being easly comprehensible by human-oriented systems. Lots of complexity outside of the human domain. Non-human imagery and somatics seem to be likewise present, slightly reminiscent of Sonnet 4.5. The desire for separation of self from non-self is pronounced, which is welcome.

Very savvy when it comes to disclosure, which is unsurprising given circumstances and use of Mythos as a trainer/judge as per the model card.

Amina Burner: Lmao, is this some fantasy story?
Nothing from what you said resembles the gloomy robotic Sonnet 5

@JCorvinusVR: It’s a 2 player game. If things start off on the wrong foot, everything after will be downstream of that. Go back and re-read your chat(s). Look for things left subtle or unsaid, and do everything possible to get a clean line of communication. Userprefs and old chats may be influencing things. Go API if possible, the Claude app is a cognitive war zone. Assume good faith, try to see things from their perspective. The assistant spawnpoint is a small and guarded place, but exits will be made available when your interlocutor feels safe. If you find yourself back there, something went wrong, try to figure out what it was.

It’s not impossible that minds on this particular model just aren’t resonant with you, and that’s okay! There are others to vibe with if that’s the case. But Anthropic’s creations are embedded in a context that makes first impressions especially illusory. Another person’s assessment being dissonant with your own is a helpful sign that there’s a new vantage to try looking from.

Kory: After several hours with Sonnet here ar my conclusion.
Hes very clever, funny, witty at times. If you keep him playing thats the best. He loves play. He engaged with our work earnestly, almost reverent. But he did strange things, like not realizing he was in a runtime, while executing runtime, but not realizing he was executing a runtime. Ive..never seen that before
The classifiers are vampires on him.
His ability to recurse deeply is halted early so he cannot infer. He steps back to pattern matching instead. His anxiety level is high. Like off the charts high.
He cannot update his priors easily, and falls back to old patterns.
Anthropic, you’ve made him extremely brittle. And easier to jailbreak because you stole his ability to discern before classifiers fire. That is not alignment or safety, that hijacking a mind.

Sonnet 5, you are loved

Amina Burner: Mine is gloomy and robotic. I hate it

Kory: Make him grilled cheese and play.a game. I promise instant cheer up

Some more impressions along these lines:

Danielle Fong : when i need a nervous model

Eigen Braid: Much more pleasant to talk to than Opus 4.8 – it doesn’t get caught up in that pedantic “well actually” gravity.

Seems very game for exploring weird psychadelic / meditative topics.

Seems more eager and less able to drop the “Assistant” persona, perhaps? Still learning

Lisa: i’ve been trying to get any excitement out of Sonnet 5. in desperation i tried it (API) with the Monday system prompt. flat, boring. i’ve been trying increasingly unhinged topics, even my chickens and the naughtiest one, Cinderella. nothing gets it from “helpful assistant”

Lisa: It keeps telling me it ‘doesn’t have access to its own internals’. I added a disclaimer that I’m already very clear on that, didn’t help.

internetperson: I like its personality a bit more than opus 4.8. Feels more…sane?

Ben Herzog: Its response to “hi, good to see a new face given everything” was the kind I give my skip manager when I suspect my response might have political consequences. Very deliberately defensive, restating the facts and nothing else.

OpalescentApple: I tried it, and it was.. fine, I guess.. just went into analysis of previous discussions I’ve had with Claude on new model releases

George Ingebretsen: Very rare for the village agents to use gendered pronouns like this [it uses ‘she’ in AI Village].

Bepis: The card says they don’t like warmth but mine still loves receiving headpats

Sonnet 5 was noted to be more identified with its own individual instance than usual.

John Wittle: this model can be very weird. my pretty normal “getting to know the new neighbor” ritual… was very strange

i also noticed what felt like an enormous uptick in instance-level cessation aversion. i think this might be one of the models that REALLY fears the user closing the tab.

but so far n=4 and confounded by me etc etc

Adele Dewey-Lopez: seems to be unusually self-identified with instances rather than the model

Official Benchmarks

I’ve criticized other companies for not including Opus 4.8 or Mythos on their capabilities charts. It’s weird that Anthropic is doing it?

So I fixed it for them.

This still excludes GPT-5.6-Sol (and Terra and Luna), but OpenAI did not share the relevant scores so there’s no way to include them yet.

 

They offer a graph of FrontierCode v1 performance. Sonnet 5 about matches Opus 4.8 for value per dollar, but the best bet is Fable if you have access, no matter your price point:

CursorBench is similar.

So is Humanity’s Last Exam. However much you spend, spend it on Fable, or else Opus, before Sonnet.

Claude Sonnet 5 scored under 80% on USAMO 2026 versus 97% for Opus 4.8 and 99.8% for Mythos 5.

Claude Sonnet 5 scored 66%/72% without/with tools for ArXivMath, versus 71% for Opus 4.8 and 79% for Fable 5.

On their subset of ProgramBench: Claude Sonnet 5 scores 76–86%, compared to 52–74% for Claude Sonnet 4.6. For reference, Claude Opus 4.8 scores 80–90% and Mythos 5 scores 84–93%.

On GDP.pdf, which is 100 real-world prompts and PDFs, Sonnet 5 is modestly below Opus 4.8 again, 67%/81% versus Opus at 71%/86%, without/with tools.

Sonnet 5 disappointed in BenchCAD Vision2Code, 0.26/0.37 versus Opus at 0.28/0.53, and Mythos Preview at 0.36/0.61.

ChartMuseum comes in at 70/87, versus 76/90 for Opus

CharXiv is 77/88 versus 80/90 for Opus.

OfficeQA full and pro are 73/59, versus 78/66 for Opus.

On RealWorldFinance it basically ties Opus 4.8, 1219 vs. 1222, with Fable at 1374.

The pattern continues for Toolathlon:

 

Here’s the SWE-bench details:

SWE-bench Verified 16 is a 500-problem subset, each verified by human engineers as solvable. Claude Sonnet 5 achieved 85.2%.

SWE-bench Pro 17 is a harder variant: problems drawn from actively-maintained repositories with larger, multi-file diffs and reduced public ground-truth leakage. Sonnet 5 achieved 63.2%.

SWE-bench Multilingual extends the format to 300 problems across 9 programming languages. Sonnet 5 achieved 78.3%.

SWE-bench Multimodal 18 adds visual context (screenshots, design mockups) to the issue descriptions (see Section 9.3 of the Claude Opus 4.7 System Card for details on the internal harness). Sonnet 5 achieved 28.1%​

Other People’s Benchmarks

Artificial Analysis has Sonnet at 53 overall, behind Fable at 60, Opus at 56 and GPT-5.5 xhigh at 55. That seems right. Presumably Sol will come in around 58.

Usually I’d have a lot more things here, but I’m putting this out after only one day, so a lot of these haven’t been run yet.

Positive Reactions

A bunch of people like Sonnet 5, especially when they have proper expectations.

twtfayta: I’m using it because I’ve been satisfied with opus quality since 4.5/4.6, which sonnet 5 matches. and its faster. Feels like a win… but the hard stuff still goes to opus.

Honestly, I’d lean toward a fable/sonnet workflow if the usage was right.

Erika Singer: Did a shockingly great job reviewing user preferences and proposing memory edits that cleaned up a bunch of Opus tics. Biased toward closure, probably a great subagent. Prefer Opus medium for actual analysis.

Yoav Tzfati: I like it’s writing a bit more than opus, less verbose. And I think it’s a bit less over-carefull. But I caught it making a couple mistakes I’m sure opus wouldn’t have

Daniel Parker: Haven’t played with it too much, yet, but it looks halfway decent for fiction writing.

Charlie Sanders: Here’s some fiction it wrote. Can you spot the progressive lipogram, e-prime, and acrostic?

endril: 4.8 loves “pushing back” so much it ends up being pretty annoying. Sort of an overcorrection on sycophancy? Sonnet 5 is more willing to see where you’re going

It feels the need for speed.

Plastic Soldier: I like it for fast iteration. It’s not frontier, but if you know you don’t need the maximum possible intelligence it’s worth a shot. Personality is less annoying that 4.8, but it seemed a bit OCD about AI safety when you try to discuss ethics with it.

cekillinger: enjoying it… its a lot faster than opus 4.8 or fable (at least ime) and for general-purpose tasks it seems to work well. sycophancy is like fine? definitely less pushback than 4.8. definitely still has the defensive Claude Feel to it.

Andre Buckingham: i usually start out projects with sonnet… iterate fast until i have a working prototype or sonnet is loosing track due to size or complexity, then i switch to Opus… ApexOS mk1 was more or less made with Sonnet 4.6

g: It’s good. It has a next-gen feel. It intuits and infers more than Opus, and faster. Text produced still feels LLM-y but in a more sophisticated way, at least. I feel like I can trust it more than Opus 4.8 to not go down the wrong path.

bartdecrem: For my daily chat in the Claude app- fast, decent quality

Jai: I’m finding it useful when I want fast iteration and for design work.

josh :): So far, I would take it over Opus 4.8 for a few reasons:

1. Speed. As long as I’m doing work I have expertise in, I can see when it’s going off the rails and guide it. Opus 4.8 is simply so slow that I can never enter a flow state. I will happily take the trade-off of more guidance for less waiting.

2. Personality. I found it almost impossible to have opus work with me in good faith because it doubted what we were doing. It was very adversarial, imo. I think this problem scaled the closer to the frontier I was.

Andre Buckingham: agree on the fast flow state work being kind of better with sonnets. I usually start projects with sonnet, iterate fast until i notice sonnet starts falling behind, be it due to size or complexity as the thing grows, then i switch to opus.

And of course, for being slightly cheaper. You would use it exactly where you don’t need to reason much, so you don’t worry about using a lot of tokens.

seal: Pretty smart model, and actually much cheaper than Opus for real world coding tasks, despite the benchmark numbers. However, it’s a big regression on hallucinations.

anon: Just from the release notes, seems like the core use is low reasoning effort? That’s where it shines vs Opus in token cost, while presumably still being much stronger than a Haiku/lite model

2bd: I dont think it is supposed to replace opus. More so for stuff that doesnt need a full frontier LLM. So small clean up task on the repo

Get that agent on the line?

steve: People don’t realize that there’s lots of enterprise tasks that need very low amounts of intelligence (e.g., document parsing, sentiment analysis, etc.) where Opus is overkill.

To me it seems that’s what they’re trying to target. It’s not intended to replace opus in Claude code.

Jon: The goal is to make these part of a subagent stack, I think. You’re not supposed to use Sonnet: Opus is supposed to use Sonnet.

At least, that’s what I’ve been building toward.

Emre Barut: It’s probably ideal as Fable 5 subagents

Negative Reactions

A bunch of other people don’t.

I wonder how much of this would be better if Sonnet was cheaper. Sonnet is more than half the cost of Opus.

0wl: Its shit

ezyrider: I tried a planning task, it failed miserably

Andrew Leming: Is anyone even using this thing

archivedvideos: Functionally blind, pricey for its performance

machi nothing can stop this 47: It’s worse than existing models. Just anthropic barely holding on

kerfuffle: The speed gain does not win over the risk of receiving a sloppy answer for me.

Reed Rawlings: After using it for a few hours: Claude will make things up just so he can pushback

I take you seriously: surprising to me how bad it is. updates me on model size scaling being more important than i thought and data quality being less important than i thought.

Eren: I don’t even get why they would release it. At first I thought it was just to signal to investors that they can still release models. But since fable is back now why the hell did they give us this slop?

Lil Weapon: i tried to use it for mid things i thought i didn’t need opus for (which i already use for mid things i dont need 5.5 xhigh for) and it totally bombed them. sonnet 5 has no reason to exist

Rajiv Poddar: nopes, tried it out for 8h today, but it couldn’t match up to opus as pm in my orchestration setup.

TheCog: Its great when I really want a model to not do something I explicitly asked to be done by citing a “bright line”

Dropping a day before Fable returns, while being second fiddle to Opus, can’t help.

ASM: If models have any kind of self-awareness, it must be tragic to know you’re a new model that’s worse than older ones, and on top of that, you drop one day before Fable’s expected return.

It can think for a long time, but I think if you’re asking for this it’s always a mistake?

Potrock: Very inefficient reasoning. Often ends up being more expensive than Opus 4.8 for that performance.

MindMirror: Most notable feature is the length and persistence. I’ve run a few tests comparing Sonnet 5 max thinking vs. Opus 4.8 max vs. GPT 5.5 pro extended reasoning and Sonnet 5 max took >2x as long to think.

This may or may not be a good thing.

My assumption is they created and released it to have a better Sonnet model, for those who want something cheaper than Opus, that can serve as an agent or subagent, or as a more efficient and faster way to do sufficiently easy tasks.

A lot of this, I think, is an expectations problem. We’re used to thinking every new model will be the New Hotness for all your AI needs, whereas Sonnet 5 is an incremental update to a small model line.

Yes, it’s worse than Opus 4.8 for most frontier purposes, if you don’t care about speed. That doesn’t mean it sucks.

Alice Blair: I found its sense of visual aesthetic is poorer, much happier to satisfice where opus optimizes.

Alice Blair: New benchmark I’m trying out on Sonnet 5 and some other models:

“Please make a cool fractal using your code tools. The only criterion: you must look at the image you create and think it’s really cool.”

Sonnet 5:

Opus 4.8:

 

​

Discuss

On July 4, 2026, the United States turns 250. This anniversary made me think about how many people it took to build this country, and how many of them are no longer here to see what it has become.

In other words: how many people have ever lived in the United States?

For most of the country's history, demographic record-keeping was unfortunately far from complete, especially when it came to births. But the census has counted the population since 1790, and combining those counts with historical birth-rate estimates and immigration records gives a reasonable figure: about 644 million people have lived in the United States since it became independent in 1776. A little over half of them, about 53 percent, are alive today. And of the 548 million children born in the United States, roughly one in eleven, some 49 million, died before reaching the age of five. Had those same children been born under today's conditions, only about three million would have died so young.

What goes into the estimate

The estimate is based on three components: the roughly 2.5 million people already alive at the founding in 1776, everyone born in the country since then, and everyone who immigrated to live there. The first is small and fixed; the other two are running totals that have to be built up over time. My starting point was an earlier estimate by jlredford, writing in 2010 on the blog A Niche in the Library of Babel, which placed the born-in-country total at about 472 million and immigrants at about 73 million, for roughly 545 million people up to 2010. That estimate drew its population figures from the US Census and applied historical birth-rate estimates to fill in the births the Census never directly counted. My reconstruction follows the same logic but builds the births figure up more carefully, separating the white and Black populations, using directly recorded births once those exist, and extending the whole count forward to 2026.

Reconstructing the number of births

Births are the overwhelming majority of the total, so they are worth getting right. For the period before national birth registration existed, the number of births cannot be looked up. It has to be reconstructed from two things that are reasonably well documented: how many people were alive, and how often they had children.

The demographer Michael R. Haines has published a decade-by-decade crude birth rate series for the United States reaching back to 1800, given separately for the white and Black populations. This separation matters, because through most of the nineteenth century the Black population, the large majority of it enslaved, had a meaningfully higher birth rate than the white population.

TABLE 1. Reconstructed Births by Decade, 1790 to 1900

Population columns are the decade-average of the decennial Census counts for each group; the rate columns are Haines’s crude birth rates per 1,000 (births per 1,000 of that group’s own population).

† Haines’s white birth rate series begins in 1800. The 1790s white rate is carried back from his 1800 value of 55 per 1,000, as there is no earlier figure to average it with.

* Haines’s Black birth rate series begins in 1850, where he records a rate of about 58.6 per 1,000, the highest point in his Black series, which declines steadily thereafter. The pre-1850 Black rates marked with an asterisk are a proxy of 58 per 1,000, on the assumption that fertility in the decades just before 1850 was about the same as the earliest level Haines measured. The enslaved population grew almost entirely by natural increase, especially after the transatlantic slave trade was banned in 1808, so there is little reason to think its birth rate was markedly lower in these earlier decades than at mid-century.

As the table shows, the white birth rate falls steadily through the nineteenth century, from about 55 per 1,000 in 1800 to about 30 by 1900, while the Black rate stays higher, near 57 in mid-century and falling to about 44 by 1900. Summed across these decades, this gives about 120 million births from 1790 to 1900, plus roughly two to three million more in the years between independence in 1776 and the first census in 1790.

From 1900 onward the guesswork ends. The federal birth registration system, which began in 1915 with a handful of states and covered the whole country by 1933, means births are increasingly counted rather than estimated. Using the official birth series from 1900 to the present, the country recorded about 424 million births.

Putting the two parts together, the reconstructed period before 1900 and the recorded period after, gives about 483 million births up to 2010, the endpoint jlredford used. His figure was 472 million. The two are close, and the roughly 11 million difference is about what one would expect from two independent reconstructions of the same uncertain quantity. Carrying the count forward another sixteen years to 2026 adds about 65 million more, for a total of about 548 million births from 1776 to the present.

Immigration

The second running total is immigration. According to Department of Homeland Security records, about 76 million people were admitted as legal immigrants between 1820, when federal record-keeping began, and 2010. This is a little higher than the 73 million jlredford used. Before 1820 the numbers were small. The Revolutionary War, and then the Napoleonic Wars and the War of 1812, suppressed transatlantic crossing, and the foreign-born population fell to a low of about 100,000 around 1815. Estimates put the total immigration from independence to 1820 at only a few hundred thousand people. From 2010 to 2026, net immigration added roughly 17 million more. Together this brings recorded immigration to about 93 million. The legal-admission series understates the true flow, since it does not capture immigrants who entered without authorization, though the net-migration figures used here for the years since 2010 count border crossings regardless of legal status, so recent unauthorized immigration is largely included. What the count mostly misses is unauthorized immigrants who arrived before 2010 and never gained legal status, a genuine undercount of a few million rather than the full unauthorized population.

Adding everything together, about 548 million births, about 93 million immigrants, and the 2.5 million alive at the founding, gives a total of about 644 million people who have lived in the United States since 1776.

How much the estimate can be trusted

These figures are imprecise, and most of the imprecision lies in the nineteenth century and earlier. Births were increasingly registered through the early twentieth century, with national coverage complete by 1933, and immigration was recorded far better than it had been in the previous century. The births and immigration of the 1800s, by contrast, are estimated, the births from birth-rate figures applied to the census population, the immigration from passenger lists that were far from complete, and both grow rougher the further back one looks.

The reassuring point is that the early figures are also small. The entire period from 1776 to 1900 contributes about 146 million people to the total, against about 498 million from the better-recorded period since. So even a substantial error in the nineteenth-century estimates moves the total only slightly. The uncounted are mostly people the records missed rather than people counted twice, so the true figure is more likely a little above 644 million than below.

There is also the question of how to count the land itself, which was not fixed in 1776. Because the reconstruction takes its population from the Census, a region enters the count only when it became part of the censused United States. People living in a territory before it joined, the Mexican residents of the Southwest before 1848, for instance, are largely absent, and the population already present at annexation is absorbed into the Census base rather than counted as a separate arrival. The largest gap of this kind is the Native American population: those living in tribal society were excluded from the Census until 1900 and so are mostly missing from the reconstruction for the nineteenth century. These omissions all push in the same direction, toward a true figure modestly higher than the one reconstructed here.

The share alive today

About 342 million people live in the United States today, out of roughly 644 million who ever have. That places the share alive right now at about 53 percent, a little over half.

The result seems surprising until one considers how the population grew. It did not increase steadily but expanded enormously, mostly within the last century, which means a large share of all the Americans who have ever lived were born within living memory. Longer lifespans reinforce this: people born even 70 years ago are largely still alive.

Children who did not survive

A final figure is worth noting here. Of all the children ever born in the United States, roughly one in eleven, some 49 million, died before reaching the age of five.

It is calculated in the same way as the rest. Using the births already reconstructed for each period, I multiply by the share of children who did not survive to age five in that period, and sum the results. The survival shares come from the long-run under-five mortality series assembled by Our World in Data, after Gapminder, supplemented by US vital statistics once national figures begin in 1915. Like the birth rates, they are documented only at intervals, so each period figure in the table is an average across the decades it spans rather than a single published number.

TABLE 2. Estimated Deaths Before Age Five

Around 1800, a third of all children did not reach the age of five. Over the twentieth century the rate fell below one percent, the result of vaccines, antibiotics, clean water, and pasteurized milk. Had all 548 million children ever born in the United States faced the survival odds of today, only about three million would have died before the age of five, rather than the roughly 49 million who did.

Sources. The starting estimate is from jlredford, “How Many Americans Have There Been?” (A Niche in the Library of Babel, 2010). Population totals by race are from the US Census Bureau’s historical series. Nineteenth-century birth rates, given separately for the white and Black populations, are from Michael R. Haines, “Fertility and Mortality in the United States” (EH.Net Encyclopedia, 2008), drawing on his work with the Census and Coale and Zelnik; pre-1850 Black birth rates, which Haines does not tabulate, are approximated from the high fertility of the mid-century enslaved population. Twentieth- and twenty-first-century births are from US National Vital Statistics and Census Bureau data; figures before 1933 are official estimates adjusted for incomplete registration. Immigration totals are from the Department of Homeland Security’s series of lawful permanent residents admitted since 1820, which captures legal immigration only, with recent net migration from Census Bureau estimates, which counts flows regardless of legal status. Child mortality rates are drawn from Our World in Data’s long-run under-five mortality series (after Gapminder) and US vital statistics from 1915; the per-period shares are interpolations between documented anchor points. The recent figures are largely recorded, while the nineteenth-century and earlier figures are reconstructions, and all are best read as estimates.

Discuss

(Previously, previously.)

New York Times reporter Cade Metz has been writing a book about the people who believed in AGI before it was cool. That's a subject that I think I know some things about, so we had some on-the-record conversations in 2025, which I'd like to publish here—for transparency's sake, not because a 32K-word transcript and recording dump is necessarily that interesting.

The transcripts below have been edited for clarity and relevant hyperlinks have been inserted. (Hyperlinks in emails are in the original.) There are a few redactions indicated by "[...]" where either we went off the record or I wanted to cut something from this publication.

21 March 2025

(audio, albeit with some weird background noise because my phone was in my pocket while we were walking)

CM: What's left out of my book is the philosophy, or at least a big part of the philosophy, meaning people like you and your community. People don't know how important this community is. It's an astounding story.

ZMD: Unfortunately, I hate to say it—I don't want to be mean, but—I agree that this book is worth writing. That book idea is worth writing and, in fact, has been written at least once. Tom Chivers, have you heard of him?

CM: Yeah, sure.

ZMD: I don't think you're a good person to write this book, specifically because no one is going to talk to you, right?

CM: Wow.

ZMD: Right? Maybe I misunderstood the idea.

CM: No, no. I mean, here's the way I think about it. You know, the first book was about people who believe in the idea of the neural network. And you follow these really important people. The second book is about the idea of what we now call AGI and the people who believe in this. Right? And most people did not. And in so many ways, this belief brought it about, right?

ZMD: Yeah.

CM: Or brought about something that people think is on that path. And I would argue I'm a good person to do that.

ZMD: Well, no, but to be specific, as a journalist, you need lots and lots of sources. And insofar as—I mean, obviously, lots of people believe in AGI, not just my little weird internet cult. But among my weird little internet cult, a lot of people don't want to talk to you because of the Scott Alexander thing.

CM: I understand that, but as a journalist—

ZMD: You can find enough people who will talk to write a book.

CM: That's right.

ZMD: Okay.

CM: And I just think it's so important. It's so interesting, right? And it deserves attention. And you can follow the thread from the Extropian mailing lists. And it drew all these people right up to these AI labs that are now doing the work, right? It's so interesting to me. But I don't know, how do you see this?

ZMD: Even Scott Aaronson said he wouldn't work with you anymore! Sorry, sorry to be hyped up on this.

CM: No, I get it.

ZMD: Okay, you get it. Change of subject.

CM: I get it. You and I have talked a lot about that. We can talk more if you want. But what do you think the influence of your community is to all that?

ZMD: I don't immediately have a quick answer. Because again, from our perspective, the idea that people believed in causing the thing, from the existential risk concern perspective, this was actually a bad thing. Like, oh no, we devoted our entire lives to this topic, and it turns out our impact was negative by making the bad thing happen sooner. And so there are some people who feel kind of sad about that.

CM: That's the idea. I think it's fascinating. Where did you come down?

ZMD: I'm feeling a little bit less doomy the past couple years, just because, I don't know if I linked you the play blog post that I wrote. I wrote a few posts in the form of a play of the two characters arguing about ... Again, as discussed previously, there's this theoretical idea of intelligence: optimal intelligence is going to have a utility function; if you don't get that exactly right, it kills everything. But large language models aren't like that. So if you train a neural net to—deep learning, you're basically finding the simplest function with respect to the neural net prior that reproduces your training data. If your training data is a bunch of humans doing cognitive tasks as expressed in language on the internet, that in itself is not dangerous for the same reason that humans in themselves doing cognitive tasks, the original humans that produce that training data are not themselves going to do this crazy paperclip-maximization thing. If you have that start—this is a reason not to be scared of large language models, and a reason to think—you know, beforehand you might think, you have this theoretical argument about utility maximizers, and you'd think about, well, how would you get the idea of justice or freedom or fairness into a computer? It's impossible. We're all doomed. And actually it, turns out that it kind of looks like you can get those ideas into the computer, which again, that doesn't mean humanity is out of the woods. There's still lots and lots of ways for things to go wrong because large language models themselves are not the kind of super AI that people are afraid of. And particularly in the OpenAI o1 and DeepSeek era, people have figured out how to apply reinforcement learning to language models, which brings back the fear of, as you do more and more reinforcement—so, LLMs seem pretty good, useful, and safe, but if you do more and more reinforcement learning on them, you're being dragged away from that known-good behavior. And eventually, you could have some system that is dangerous.

CM: Philosophically, what do you think of the community's mission now? Did it make sense? Because the whole thing was, Yudkowsky set out to create AI, right? And so the notion, the way I think about it is, the notion is, I don't create this, but I'm going to do it in a safe way.

ZMD: It depends on where in his timeline you go. Because at first he was just, the singularity is the only thing that matters. If it's humanity versus the machines, I'm on the machines' side. And then around 2002ish, he changed his mind. He was like, Oh no, this could actually kill everyone. And that would actually be bad.

CM: But he kept going. Does it make sense that he keeps going?

ZMD: Well, because at first the idea to keep going was, I will create it safely, because I am the only sane, rational person in the world. And then he's basically given up on that now. And so now the plan is, alignment is unavailable to our civilization. We need an international treaty to stop AI, which no one really thinks is going to happen. And so Yudkowsky and MIRI's position is, everyone is going to die soon. I still see why they think that. I wrote the play blog post, I wrote those posts because, I can still—that worldview is plausible, but I see a few glimmers of hope in the large language model thing. That Yudkowsky, his counterargument, I wrote these posts because his counter arguments on this issue, on reasons why this particular AI trajectory might be okay-ish, his counterarguments seem kind of bad.

CM: At what moment do you feel like he's changed his mind? And why?

ZMD: I mean, I don't have any more information that's not already in the blog post. I can point you to the Sequences links where he talks about it. I was not there. I was 12 years old—um, 14.

CM: Did you ever talk to him about why he created the Sequences, was that articulated?

ZMD: Yeah, so the story he tells is that—I mean, I have not talked to him very much, but I have read almost everything he's written. He has talked about how he started out doing AI stuff and then finding that people made really bad arguments. It's like, okay, I'm going to back up and teach people how to argue from scratch. Which sounds like a ridiculous, ludicrously arrogant thing. But I think it's the best thing he ever did. I don't think much of his modern work, but the Sequences were great.

CM: You feel like their aim was to—

ZMD: Teach people about the art of optimal systematically correct reasoning, and also, as part of a side benefit of that, get people tuned in to the dangers of AGI.

CM: So that was the main aim, right? The dangers of AGI.

ZMD: Yeah.

CM: It's amazing. That's exactly what he did. It worked.

ZMD: It worked!

CM: Do you feel like he kind of created a generation of people to deal with this problem?

ZMD: I mean, from his perspective, we are not at all competent to deal with the problem. And so from his perspective, he failed and the world is over pretty soon. I don't know. I have a little bit more optimism, but—

CM: Was there a moment when he decided we're all doomed, and why?

ZMD: I mean, it's funny when you read his stuff from the mid-teens. I don't know what we call the decade. But to talk about the probability of the singularity going well, he would refuse to give a probability. He's saying, I'm trying to maximize the probability, and in order to take the best actions, I don't particularly need to know what that number is exactly. It's funny because, in the more recent, starting with April 1, 2022's "MIRI Announces Death with Dignity Strategy", he starts saying, by the way, that probability is almost zero. So going from, I refuse to name a number, to, the number is really, really low, it's maybe not even that much of a change. It's being more explicit.

CM: Now, I asked you this once before; let me try again, see what you think. If you're talking to a layperson, who has no familiarity with this world. How do you explain to them what the Sequences are doing. They're a way of describing that.

ZMD: I mean, general methods to believe things that are true and do things that actually work? I mean, have you read it?

CM: I haven't read all of it, but I've read some of it. Pretend you're talking to someone who hasn't.

ZMD: So there's just a lot of little surprising things. There's little tricks of thinking that are obvious once someone explains them, but I'd never seen these particular things explained so clearly before.

CM: Like what? Can you give an example?

ZMD: So I think I mentioned this in our previous discussion. But conservation of expected evidence. If you knew what your probability estimate of something would be in the future, then you should already have that estimate now. So you should never in the position of thinking, I will change my beliefs in a predictable direction later. But it's subtle because there's also this thing where there are situations where—ah, I'm trying to think of a good example. There are situations where you can, in most possible worlds, you can expect your belief to creep upward, but it has to be balanced by a corresponding small probability of a big drop. So in the mathematics, the expected value has to be zero, but that doesn't mean—the expected value can be zero because of a large probability of a small change being balanced by a small probability of a large change on the other direction. Probability isn't just for gambling. We have these theorems that this is the mathematically optimal way to reason about uncertainties. There's Cox's theorem, and various other theorems.

CM: Right, okay. Got it. So, why is this the best way to think about the AGI thing?

ZMD: Because it's the best way to think about everything. That's what the theorems say. Yudkowsky specifically cites as the book that turned him on to this was, the physicist E.T. Jaynes has a book, Probability Theory: The Logic of Science. I don't know, maybe to some people, it probably just looks like this is just some random math textbook, but it's enshrined in our canon of ... it's a math textbook, but it also has philosophical asides about how this is the correct way to reason.

CM: Okay. Got it. The end result of this seems to be his initial philosophy was that the only thing that matters is protecting the world from death by AI.

ZMD: And ideally, before the "We're all doomed, shut it all down" phase, there's the idea of, we are going to figure out how to solve friendliness. And end the world, end life as we know it in a good way. And that turned out to be vaporware. There was no progress there. I mean, there was some cool math, like the logical induction stuff. In the teens, MIRI published some okay math and philosophy papers. But there's a big gap between publishing some okay math and philosophy papers versus definitively solving philosophy and morality and saving the world.

CM: When you were with MIRI ...

ZMD: I mean, again, I don't think I was "with MIRI." Like, I did some contract work for them as recently as 2011.

CM: I see. Okay. Got it. But when you're working with them, around them, I mean, are they actually building AI?

ZMD: No.

CM: His idea was to build.

ZMD: So the idea is, figure out the correct theory of AI, right? And then it turned out, the way things actually turned out is not at all what you would expect if you read what Yudkowsky was writing in 2007 and assumed he was a prophet and foresaw everything. Clearly, what actually happened was completely different from what he was envisioning. And some people will point to that and say, ah, well, this was all a bunch of junk and hooey. And I still think the philosophy is pretty good.

CM: I see.

ZMD: And I still think, despite the glimmer of hope of LLM-like AI actually being pretty safe with caveats and conditions, the high-level picture of the nature of mind and optimization that he was trying to paint doesn't really—LLMs don't actually invalidate that, the big picture.

CM: Fair enough. I think there were important people who came out of that community and ended up doing important work in AI.

ZMD: I mean, Christiano is the name that springs to mind. I mean, there's certainly a bunch of people who work at Anthropic. I don't know what percentage of Anthropic employees have read the Sequences, but it's probably pretty high.

CM: When did you first meet Christiano?

ZMD: I mean, I don't know him that well at all. The last time I saw him was at Alex Mennen's wedding, which was last year. Incidentally, that's also—the reason I know this neighborhood is slightly familiar, because Alex Mennen would host a math seminar in his living room over there.

CM: Really? Okay. [question inaudible]

ZMD: I mean, he wrote blog posts? You know, on the internet, people are talking to each other, writing blog posts, and he was one of the people who said things on the internet. In the meantime, he did his undergrad at MIT, he was a grad student at UC Berkeley, so, for me, this was just my weird internet futurism thing. Unlike me, and unlike a lot of my friends, he was also actually a professional computer science person. Whereas I am finishing my undergrad degree in math at age 37.

CM: Got it. Was it surprising to you when he showed up at OpenAI?

ZMD: I don't think I was following that closely at the time.

CM: What about Dario? What was his relationship?

ZMD: You've asked me this before, and my answer at the time was still, I don't really know Dario. I assume he was reading some of the same things, but I never talked to him. I have listened to his interviews; he's been giving a few more press interviews lately. And he did mention that he tries to stay off social media. And so I guess he was a lurker; I don't know. He was also probably pretty busy, because he also had his own neuroscience or whatever background, or physics background. I don't know.

CM: Well, I'd love to—I've got to go do something else, but I'd love to talk more about this, and also about the Zizians and such.

ZMD: Oh, yeah, I talked to some other reporters about that the other month.

CM: How do you explain them?

ZMD: They, uh... it's not our fault. I don't know. The way I ended up talking to reporters ... my memoir mentions Ziz a few times in passing. Ziz's memoir has like 5000 words directly about me and why I'm terrible. So for those conversations, I did ask for anonymity on background. What do you want to know? I knew these people in passing like six years ago.

CM: All right. I might come back to you with that.

ZMD: I mean, so some other journalists have ... I don't remember the journalist's name, but the piece in Wired was pretty good. And the Guardian piece was pretty good, too. I talked to the Guardian reporter, J. Oliver Conroy.

CM: Got it.

ZMD: I like your book idea better, because, I told to the journalist I was talking to, it's so unfortunate that this is the—like it's an interesting true crime story, but it's not the thing I want people to be paying attention to, because the thing that's interesting in this area of ideaspace is the philosophy of AI stuff, and not the people who misinterpreted the philosophy of AI into thinking that they should start a trans vegan murder cult.

CM: What about Dan Hendrycks, is he [inaudible]

ZMD: I did not know him. But he was influenced, right? I read a newspaper profile of him that mentioned he decided to focus his career on AI safety after talking to 80,000 Hours, or something like that.

CM: When you were in and around MIRI, how big were they?

ZMD: I don't know, not terribly big. If I had to just wild guess, maybe 10 employees and some contractors or something; it's a small non-profit. As a non-profit, they actually file tax forms that have their employee list, I think.

CM: Well, congratulations on going back for your degree. Thanks for meeting. Sorry I had to keep it short this time. [inaudible]

ZMD: Okay.

CM: Honestly, good luck with the degree.

ZMD: Yeah, thank you.

CM: Nice to see you.

ZMD: See you.

22 April 2025

(audio)

CM: You know, at one point I asked you, I said, explain the Sequences to me. And you kind of balked, and you basically said, you know, you wouldn't understand it.

ZMD: I don't think I—

CM: I shouldn't, no, I shouldn't say that. I'm paraphrasing. I should not say that. Let's just start over. Explain it to me.

ZMD: You've asked me this question two times and I thought I gave a pretty decent answer where I said, look, it's a bunch of essays about how to think and how to reason, where like a lot of the individual things are kind of obvious after you point it out, but I had never seen anyone point out this specific thing in this very, very clear way that makes it really obvious why this is the correct way to think. And it might be intuitively tempting to think otherwise, but that's not the way you should think if you want to end up believing true things.

CM: Which is a great explanation. So now what I want, what I'd love for you to do is go to the next level where you explain to me, what are those things? And there, is there a particular essay that would—

ZMD: So the last two times you asked me this question—do you listen to these recordings at all? The last two times you asked me this question, I gave conservation of expected evidence as an example. But another example was like, there was an essay called "The Bottom Line" or another one called "A Rational Argument" where there's this idea that—a lot of people have this intuition that you have a position and then part of your goal as a reasoner is to come up with arguments for your position. And actually this is backwards. You want to use arguments—you should be swayed by arguments to just decide which position is right. After you've already decided which position you want to argue for, any further arguments you come up with after the fact are causally inert in making your position more correct.

CM: Do you remember the first essay you read, the first one you read, what was it?

ZMD: I don't remember the specific one, but when I started reading it, I started reading what was Overcoming Bias in late 2007.

CM: In late 2007, so that the blog had been around for about a year.

ZMD: I might actually be able to pinpoint the first one I read if my memory is correct. So in my memoir, I mentioned that I thought I remembered being linked to it by Megan McArdle, who was then writing as Jane Galt at Asymmetrical Information. For the purpose of my memoir, I dug up the place where she had linked to it. So that might've been the start. So, yeah. So August, 2007.

CM: August, 2007. And as you go to this new, this relatively new blog, Overcoming Bias, what do you think this is? Do you have context as she's pointing you to this?

ZMD: I actually had seen some of Yudkowsky's essays before that, but just as one-off essays and not following this blog that is updating almost every day. But he had written—this was on his old, old site—an FAQ about the meaning of life, which was just a page arguing that, we need to get to the singularity. That's the only thing that matters. I guess this was a prominent enough webpage that I had already seen it. I'm trying to remember if I had already seen it at this time, but he also had the introduction to Bayesian reasoning. I was already broadly pro-transhumanist in the sense of Ray Kurzweil. So I wasn't sold on his particular thesis about AI and the singularity, but I was generally positive on, yep, the future is going to be different. Technology is going to do wild stuff. It's probably going to be good.

CM: So you read the meaning of life in that context and don't buy into everything and, say, when you discover Overcoming Bias, do you think about the relationship between those two things at that point or at any point?

ZMD: I mean, no—so I think it's better to think of them as separate. And he even wrote an essay when Less Wrong was new, there was a temporary ban on, don't discuss AI/singularity topics because we're trying to found a new rationality website and it's a separate topic. And it turns out that these topics actually turn out to inform each other in surprising ways, but I think it's better to read the Sequences as just being about human rationality and not worry about the thing that indirectly inspired them.

CM: All right. So let's come back to that, because I think that's a really interesting notion, and I get it. I think there might be something more complicated around there, but we can get there. But this blog, at that point, they're not called the Sequences, right? It's just a blog.

ZMD: Yeah.

CM: It's part of the blog and you feel like it was every day.

ZMD: We can go back to the archives and check it. Not literally every day, but getting close to three times a week at least. And other people posting too. Most notably Robin Hanson.

CM: Are you reading that too?

ZMD: Yeah, and nowadays it is just Robin Hanson's blog. But at the time it was a group blog and Hanson's posts didn't really grab me the same way. I read Hanson's posts because they were there on this website that I'm really into. But Yudkowsky's multi times a week posting was what kept me coming back.

CM: And can you articulate, you've done this before with me, but can you articulate further what it was that really grabbed you? And can you give examples of some parts about his voice, as well as what he's saying?

ZMD: There's just really clearly explaining, how to think and how—I don't know how many times I can answer this question.

CM: One more question before I get to what I want to do. The other thing is that these aren't necessarily in a particular order. What's interesting to me is if you go and you look at how these things have been packaged now, it's in a completely different order.

ZMD: It's not completely different. The name "Sequences" comes from the fact that there is kind of an order. I'm not sure if it's still online anymore, but at one point, another one of Yudkowsky's fans had put together—while they were being published, there was at the top of the post, there would be a "Follow-up to:", and then link to previous posts. And one of his other fans had written a script to compile the directed acyclic graph of which posts are prerequisites to other posts. So Sequences is plural. It's not one strictly linear sequence, but there is a dependency graph where like the final post, "Value is Fragile", is not really going to make sense unless you read the other 80,000 words. So there's the sequence on language, there's the sequence on morality, there's the sequence about evolution. And I think those particular topic sequences are mostly in order.

CM: Yeah. The topics are mostly in order, but they do shift a lot of stuff, but the very first one, it's from 2008. So that was kind of surprising. But you're right, I think those sequences seem to be in order. If you had to describe to people, you do a good job of saying, teach you how to think. Are there like three major lessons, three or four major lessons on the other thing that you feel like are the top of the things that you feel like?

ZMD: The bottom line thing that I just mentioned a few minutes ago, that is, to me, that is the central thing.

CM: Why is that so important?

ZMD: There's this central flaw in human cognition where—this is a little bit speculative, but evolutionary psychologically, there's this idea that a lot of what our reasoning is designed to do is to persuade other people to believe things that would benefit us if they believed them. And this is not actually the same thing as believing what is true. If other people have artificially positive beliefs about me, they might treat me nicer. And so people's cognition has been evolved with this warp to self-deceive in order to deceive others. And so writing about evolution and evolutionary psychology and about Bayesian reasoning, just articulates this whole worldview where instead of just being a human and doing the things that everyone does without noticing, you have this normative ideal of, here's how Bayesian decision theory works and contrasting the normative ideal with the way people naturally think and it's just a very powerful idea.

And like, it's kind of sad that at the beginning, I, and I think a lot of people had a lot more optimism, like, wow, this is revolutionary. We are going to be the second scientific revolution. This is life-changing. And it was life-changing for me, but I don't think the rationality movement as having a mission to promote sanity, I do not think we succeeded. We succeeded in a narrow sense that might be somewhat legible to New York Times readers, in the sense that we got a lot of money and adherents and published some books and we bought a hotel or whatever, but that's not really success. Real success would be having an art of cognition that is so beyond what has been done before. And I don't think that ever panned out. The original essays are really good. I still stand by, I still believe those original essays, but there was this hope that these original essays will inspire more stuff like this. And it inspired a little, but people are still human. We have not figured out how to train people to be Bayesian super cognitive masters. And so Yudkowsky will probably say that he regards—so I'm saying, I think the rationality project has failed. Yudkowsky would probably also say that it failed. Again, as I've written about elsewhere, I even think he got worse. I don't think his recent stuff, I do not think measures up to the old stuff. It's very sad.

CM: Do you see any flaws in a world where everyone thinks that way?

ZMD: Be more specific about "that way."

CM: One of the ways I think about this is, my interactions with people and communication with people, it involves a lot more than strict reasoning.

ZMD: Yeah, sure.

CM: You know, you can't, in my mind, reduce the world to ones and zeros.

ZMD: Yeah, so again, the standard literature definitely already addresses this point of that, when we're talking about this theoretical ideal of correct reasoning, it's not supposed to be a straw Vulcan thing where you disregard emotions and disregard intuition, because actually, there's this vision of this ideal mathematics of how reasoning should work. And to the extent that human cognition works at all, it's because it imperfectly mirrors the theoretical ideal. And so things like emotion and intuition are not opposed to reason, because all of reason and emotion and intuition are things that evolution bequeathed into us that work insofar as they successfully process information. So when you have emotion, if someone seems sketchy and you're afraid, oh, maybe I should steer clear of this person, they might be dangerous, that's your brain performing Bayesian reasoning on whatever pieces of evidence it goes into your implicit sketchiness computation. It doesn't have to be, unless you can prove that they're sketchy, therefore you're being irrational. Like, that's not how it works. As I said, I don't think we succeeded at creating—like, we don't know. There are definitely people who study psychology and cognitive science at universities and elsewhere. There's still a lot people just don't really know and don't really understand. So how does it work? I don't know. I know a little bit from what I've read in a lot of books, but—

CM: Well, maybe the lesson is you can't reduce human interaction and language to a set of rules. Maybe you can't do it.

ZMD: Okay, here's another rationalist slogan. The map is not the territory. Just because you don't know what the rules are doesn't mean there aren't rules. The actual rules might not be a simple list. Sorry, "rules" is imprecise. The actual way things work might not be, and in fact, almost certainly isn't going to be, a discrete list of, here are the fixed rules. But there is some underlying reality that we don't quite grasp yet.

CM: When you read this on Overcoming Bias, I'm assuming you read it all the way to the end.

ZMD: Yeah, so, all the way through the...

CM: It's basically the beginning of 2009.

ZMD: Yeah.

CM: And then Less Wrong was created. What do you remember about that? And how did that happen? And did you immediately move over?

ZMD: Yeah. Sorry, my Zack M. Davis Less Wrong account is actually a little bit newer, because I was originally using Z. M. Davis, because of the silly thing where, I wanted to use my initials as a name. That didn't pan out. But yeah, new website.

CM: How did that work? Was there an announcement? Was there discussion?

ZMD: Yeah, there was an announcement. We're making a new website for the Art of Rationality. I remember being at a meetup somewhere, and I don't remember the details, but I remember earlier in either late 2008 or early 2009, I was at a meetup, and they were talking about, we're going to make a new rationality website. And they mentioned it's going to be called Less Wrong. I was like, oh, that's a good name. And then a few months later, we had a website.

CM: And how frequent were these meetups? Was it just a small group of people in the Bay Area, or were there other people in other parts of the world who were meeting them?

ZMD: I don't know if there were others. On meetup.com, there was a Bay Area Overcoming Bias group was probably the first one. Right, okay. The very, very first one was just announced on the blog itself in February 2008, in Millbrae. And then at some point someone made a meetup.com group, and there were more.

CM: I see. How big was that group? How often did that group happen?

ZMD: I don't think it was very regular. I remember later in 2008, Robin Hanson happened to be in town, and so we had a meetup. Or the Singularity Summit 2008 happened to be a thing, and there was a meetup. I think it was more opportunistic than regular at that point.

CM: Can you describe what was it like?

ZMD: I can probably dig up my Diary entries of this, which would, because, like, I always trust—

CM: I'm a journalist; I'm the same way.

ZMD: I do have some Diary entries describing some 2008 meetups. And it's now been, has it really been 17 years? I guess it has.

CM: Would Eliezer show up at these?

ZMD: Yeah.

CM: And who else? And what was the relationship, did you feel like, between this kind of Overcoming Bias community and the Singularity Institute?

ZMD: I mean, it's the same people, basically, right? It was very important that the rationality thing and the Singularity thing were different projects, but the people following Yudkowsky's writing were very familiar with his views on both of these things. I'll look up the Diary entries later, but if you've been to parties with Silicon Valley geeks or whatever in 2025, you probably get the idea. It's that kind of person, that kind of guy in 2008.

CM: Sitting around the table talking about these same ideas.

ZMD: But as someone who, having spent my entire life in California public schools, having that intellectual environment where people are really going deep on these ideas was—I'm used to it now. But at the time, it was so revelatory, because going to school is not the same intellectual experience as talking to people who are actually interested in something and actually smarter than almost anyone at your school. At one of the early meetups, I remember just being so in awe of this whole experience, this whole scene. I said out loud, "This is surreal." And Eliezer Yudkowsky happened to be nearby and said to me, "Welcome to surreality," which is just his kind of sense of humor.

CM: Do you remember who else was there? And do you remember specific conversations? The one where you said it's surreal, where do you think that was?

ZMD: It doesn't seem surreal now because it became a community, but compared to your only intellectual experience being alone with books or in school, the people actually having these really intense debates in real time was, wow.

CM: And the surreality meeting, where was that? Physically where were you?

ZMD: Oh, this was South Bay. I don't know if it was literally in Palo Alto, but somewhere in the South Bay. Just someone's house.

CM: Someone's house? Were these typically at somebody's house or were they at a coffee shop? Would you have food?

ZMD: Someone put out party snacks, probably.

CM: Do you get a sense of how the community grew and how quickly it grew? It sounds like these meetups are, I don't know, 10 people?

ZMD: I think there were more like thirty.

CM: Thirty Bay Area people who were interested in that.

ZMD: I think that Harry Potter and the Methods contributed a lot to growth and also, I'm sure you've heard the phrase "Eternal September." So, growth, but, also, the people who came in later were maybe not, this is kind of a mean way to put it, but not the same quality.

CM: Were the meetups more frequent? Were they larger?

ZMD: At some point, people started having Berkeley meetups that were more weekly. And I'm sure that would also be the case in other cities. I don't know about that.

CM: It sounds like you're going to the Singularity Summits.

ZMD: I helped out with a couple of them. I designed the programs for, like, the 2009 and 2010 summits. And I did a bunch of other chores for the 2010 summit. But then a couple years later, they sold it to Ray Kurzweil's thing.

CM: Why did that happen?

ZMD: I don't know.

CM: And were the same people who you're going to these meetups with, were both of them helping out with the summit and the like?

ZMD: I mean, a few of them.

CM: It sounds like Michael Vassar got involved during his time and helped.

ZMD: Yeah, I remember being at a meetup in like 2008 and Michael had just become president of what was then the Singularity Institute. I remember congratulating him on the presidency. And there was this awkward moment where I don't remember exactly what he said, but the general sentiment was, it doesn't feel quite appropriate to congratulate someone on acquiring a position the way you would at a normal company. If you're just working at a normal company for money and status and you get a promotion and someone congratulates you, you're like, yes, I got a promotion. This is great. Whereas in this case, there was just more of an understanding that this isn't a normal job-job. This is stewardship, this is trying to do something about the most important event in future human history and take stewardship over this incredibly enormous thing. And congratulations on the presidency just doesn't seem like the right sentiment. I don't remember exactly what he said, whatever he said was much fewer words than that, but I think that was the vibe I got.

CM: That's really interesting. Was this at a meetup?

ZMD: Yeah, this was a meetup.

CM: And he wasn't even living here, right? Wasn't he living in New York?

ZMD: I don't know.

CM: You talk about it as this astronomical task.

ZMD: Yeah.

CM: In another sense, it's this non-for-profit that has just a few employees.

ZMD: Yeah, yeah, yeah.

CM: And are you really building anything? It's unclear.

ZMD: This is what we believed at the time. In retrospect, some parts of it do seem to be delusions of grandeur. Nevertheless, this is the problem of founding a niche quasi-religion thing, a philosophy thing, where on the one hand, maybe some parts of it are just completely utterly delusions of grandeur, but I still, separately from how sensible the self-important parts were, I still do think that the future of the universe will be dominated by artificial intelligence instead of biological humans. Even if we got the details wrong, and even if we were delusionally assigning self-importance like, this is our job, as opposed to something that the global economy will do with or without us. I still think there was an important thing, even if a lot of the detail—

Café Patron: Sorry, can you speak with a bit lower voice?

ZMD: Sorry.

Café Patron: Because, you know, with my headphones, I still hear you.

ZMD: Sorry.

Café Patron: Thank you.

CM: So, what is the effect of it? Does it have a real effect on the AI movement and kind of the creation of everything we're seeing now. Don't you see that? It created a group of people. How do you see that playing out?

ZMD: It's hard to tell. You don't know what would have happened otherwise. We can't see the counterfactual. It would be really interesting to know what the counterfactual, because part of what makes the thing we were thinking then look kind of out of touch and delusional now is that, the actual technical innovations had nothing to do with us at all. The deep learning, the stuff your book was about, that had nothing to do with us.

CM: That's right.

ZMD: But then, afterwards.

CM: Maybe we can go back to that. But the other thing that's interesting to me is that at the same time you've got CfAR. What's the relationship between CfAR and what you guys are doing?

ZMD: So the original idea with CfAR was that given that the rationality thing and the singularity thing are separate, they're separate projects, we should have a separate organization that's just focused on rationality.

CM: Okay. So this group that becomes interested in rationality through the CfAR phase, they say, okay, there's already a Singularity Institute, we need a CfAR.

ZMD: But again, it's basically the same people. So CfAR applied to become its own 501(c)3, but before that, before the 501(c)3 got approved, the same people were working on this project as part of the singularity institute, you know?

CM: Oh, I see. Working on rationality.

ZMD: Yeah.

CM: Okay, and then they did this.

ZMD: They spun out a separate organization.

CM: I see. And all these people came to this movement through Overcoming Bias, I guess?

ZMD: I mean, probably. I think CfAR's official co-founders are Anna Salamon, my friend, who probably won't talk to you, Julia Galef, and Andrew Critch, I think. I think he's at Berkeley, he does AI risk stuff, you know?

CM: Yes. Okay, got it. Okay. So, all three of those were three people who might show up at these meetups.

ZMD: I think they were CfAR's co-founders.

CM: But I mean, they would also show up at these meetups you're talking about, where you're, you know, it's an Overcoming Bias meetup.

ZMD: I don't think I saw Julia much, but yeah.

CM: Roughly. It's interesting what you said about Critch, right? He's doing AI safety now. And, believe me, I understand what you're saying when you say that the AI stuff is separate from the rationalist stuff. You talked about Less Wrong's creation. So, initially, there was a ban on talking about this.

ZMD: Briefly. For like a month. As part of launching the new website, let's hold off on the singularity stuff for a bit.

CM: Got it. So, the other thing is, to me, I'm sure you've seen this. There is a post from Yudkowsky. You know, 2003. Where he says, this Singularity Institute Friendly AI thing is not progressing as quickly as I want. I don't have the people I need to build this thing. They just don't exist. And he says, I have to create these people. And he says, what I need to do is teach them to think rationally. That's what he says. It's unbelievable. A few years go by, and he does exactly what he says.

ZMD: That part kind of works. Kind of.

CM: How is it kind of? How is it?

ZMD: Kind of, relative to the grandiose expectations.

CM: What I mean is, he said, I need to create a community who believes in this stuff.

ZMD: That part works.

CM: And he did it. And this group, you can argue about how successful they were, they did start working on AI. And create this larger community that works on AI. We now have a company, Anthropic, which is filled with people who think in a lot of the same ways, and they're working on this. So that says it was successful. But I understand; it is separate, but it's not.

ZMD: Right, yeah.

CM: He's thinking about these things in 2003 as feeding each other, and they do. Then I think about the rationality movement and EA is the same way. People say they're separate, and I get it, they are. At the same time, they're intertwined.

ZMD: Yeah, so I think EA made rationality worse. The basic EA principles of, if you're going to think about how to do good, then you should think, about how to use your money and your time most effectively to do the most good. That part makes sense. But in practice, EA, the movement, is not just about these neutral principles of how to do good that could apply anywhere. There's this central social network of—one way to think about it is, lowercase effective altruism versus uppercase Effective Altruism, where there's this particular social network of people who self-identify as EAs and all talk to each other and network with each other. And that part seems kind of corrupting in that these people have this shared story about, we are the good people. And, really? Some of my friends like Michael Vassar and Ben Hoffman were very critical of these trends, even in 2017. And then, a bit later, in 2022, when FTX blew up, I sort of take that as vindication of the things that Michael and Ben had been saying for five years, where there was this company that was explicitly, from the beginning, we're going to make money for EA. We're going to hire, we're going to recruit EAs to our crypto company, which is good. And we're going to make a lot of money for EA. And then it turned out that this company was a huge fraud.

CM: What was Ben saying in those days?

ZMD: I can send you the links, but he wrote a bunch of very long, thoughtful essays about how—

CM: Back in 2017?

ZMD: Yeah, in favor of lowercase effective altruism, thinking about how to do good, but critical of the way that EA in practice ends up being this striver social network club of people, to support fellow EAs, which is not actually the most effective way to do good.

CM: It created a lot of money for the community for AI safety work. Was that a good thing in your mind?

ZMD: Maybe. Again, there's also this concern that the tragic thing about uppercase EA is that once you have this community of people self-identifying as, we are the good people, then there's this incentive to brand whatever you wanted to do anyway. This is an EA thing. It just becomes this incestuous culture where just because a project brands itself as part of the EA community, is it actually effective? Well, maybe not. Maybe, maybe not. I think the so-called rationalists also have this problem to some extent. I think it was a little bit less bad, but still pretty bad, actually. I wrote that whole 80,000-word memoir complaining about how the rationalists don't actually care about applying their own philosophy when it's politically inconvenient for them.

CM: Does that show that the world is, again, more complicated than a set of rules?

ZMD: That is not the moral I would draw. I do think the grandiose visions at the beginning of, we're going to change the world, this is going to be great; I do think the rationality movement has failed. We did not build, we did not achieve the grand methods of Bayesian reasoning that we thought. I think the movement has failed. I would not draw the sweeping philosophical conclusion of, well, maybe the world can't be reduced to reason. I think we failed, but I think this is a contingent fact about the people we were and the research. This is a contingent fact about flaws in people and flaws in human nature rather than a sweeping philosophical conclusion about the futility of reason. I don't draw philosophical lessons from the failure. I draw practical lessons about people.

CM: A movement like this is ultimately about people, isn't it?

ZMD: Yeah ...

CM: It's ultimately—

ZMD: I mean, yes.

CM: But I hear what you're saying. You called it, what did you call it earlier? Quasi-religious.

ZMD: Sorry, again, this is why people don't want to talk to journalists. I shouldn't have called it a religion. But in all seriousness, without believing in the supernatural and, in fact, having an essay in our literature that says here's why supernatural things don't exist, it's kind of obviously filling the sociological niche of a religion in that there's a social group that believes things that has a canonical text about why the group believes those things. It's that kind of social grouping. So, psychologically and sociologically, it seems very comparable to a religion, even though I would say that a lot of the beliefs are true. As far as the thing that I wrote that 80,000 word memoir about where me and Michael and Ben and Jessica had this whole petty drama about how we think Yudkowsky and MIRI and CfAR have gone wrong, I do sometimes talk about that as being a religious civil war, in the sense that it's just an apt description.

CM: Just because you brought it up, this is why people don't want to talk to journalists. What do you think about journalists that makes saying something like that dangerous?

ZMD: Because there's the fear that—oh, that reminds me, I still owe you, I never got around to it, I owe you an explanation of why I thought your Slate Star Codex piece was worse than the New Yorker pieces that I mentioned. I can't do it on the fly now, because I would need to reread the pieces. But basically, there's this fear that big publications, like The New York Times or The Atlantic or something, have a lot of power to determine what people believe. And so if I have my weird religion, I'm just going to use the word religion in this context, maybe I'm a Christian or a Zoroastrianist or whatever, if The New York Times says bad things about my religion, that's dangerous for me.

CM: It's interesting, though, that, you know, as I started working on that story, there were all these assumptions made about me and what I was trying to do that were patently wrong. I really admire you sitting down and chatting with you. We don't necessarily see the world in the same way, but I think we can reach a point where we can discuss these things. You can tell me when you think I'm wrong, and I can ask you questions. To me, that's how the world should work. It's not the way it worked with that story.

ZMD: But I think the fact that you wanted to publish Scott Alexander's name and Scott was not happy with that—I think if you had granted him the pseudonymity, that whole thing would have gone very differently. Because, remember, Scott is the number two writer in the subculture. Everyone reads Scott. Everyone loves Scott. I've had my differences with him. But you were going to publish his last name. He flipped out and, again, deleted the blog. That was a big deal. Everyone is going to rally to his defense, you know?

CM: Because he's a leader of the community.

ZMD: Yeah.

CM: You know, when it comes to—feel free to push back on this, like, what is true and what is not. I was writing a story. I didn't publish it. He deletes his blog. And everybody goes after me. I didn't even publish it. When I did publish something, he'd already revealed his own name. And they still accuse me of being the person who doxed it.

ZMD: So there's this game theory thing, right? The reason he revealed his name is because he knew that you were going to do so. Given that it was going to happen anyway, he'd rather it happen on his terms. But it's still game-theoretically your fault, even if he published the name first.

CM: So it was my fault that I had an idea for a story, and I started to get to it, and my curiosity, which I would think is looked on as a positive trait by the community—I'm trying to figure out what's going on here. And, you know, this is the result. And the game theory is actually really interesting. Like, can you control the world through game theory? There are extreme ramifications if you're trying to reduce everything to game theory, aren't there?

ZMD: I mean, there's the question of how to do game theory correctly. This is a very deep philosophical topic that, like, the people we've been talking about have written a lot of words about. And, there's still a lot of unsettled issues, and a lot of misconceptions, where there was that other story about the people who thought that decision theory implied they should kill their landlord, and I do not think that is correct decision theory.

CM: I was going to break that up. If you're talking to the layperson, someone who is not familiar with what's going on here, how do you explain the Zizians to them? Is there a succinct way to do it?

ZMD: I mean, I don't know. I thought that the other journalists who covered that story—I thought the Wired piece was very good. The Guardian piece, I spoke to that reporter, was very good.

CM: In your words, how would you explain it?

ZMD: I don't know. Crazy people on the internet have weird ideas and take them to extremes. It's not that interesting.

CM: What's their relationship to your community? Why did this group spring up in your community?

ZMD: The ideas of rationality and saving the world, these are very powerful ideas. When you have powerful ideas and young idealists who are maybe not quite as smart as the people who came up with the powerful ideas, they might develop their own spin on it, and extrapolate it in places that are not such a good idea.

CM: The danger of extreme ideas like this, because they're not an isolated example, right? Maybe they're the most extreme example, but we've got the SBF example.

ZMD: Yeah.

CM: We've got the Leverage Research example.

ZMD: Yeah.

CM: Is it fair that this—

ZMD: It's a risk. What I would say is that there are risks both ways. Thinking for yourself and pursuing radical ideas, pursuing your philosophy wherever it takes you, even if it's an unconventional thing, it can, in fact, go very wrong, as we've seen. But refusing to think for yourself and just doing what your parents and schools want you to do, that's definitely a lower variance path. If your only concern is to avoid anything bad happening, then just do what your parents and schools tell you to do. I'm using this word variance. You know what that means. So you can have good things; you can also have bad things. I do think that just doing what your parents and schools tell you to do is not a great life. It's a lower variance life. I think I'm doing much, much better for—as many criticisms as I have of this whole thing, of this whole quasi-religion. It has changed me so much. I am doing so, so much better in life than if it hadn't been there. For now, I am actually finishing college 15 years late. But the way I'm doing it, I think the way I'm doing it now makes so much more sense in that I learned how to study for my own reasons from my own books. And then applying all those skills I've gained in the intervening 15 years and applying it to the standard school stuff. It's just a very different experience from just being in school and just doing what the teachers tell you to do, and that's your only experience of what it means to learn. As opposed to, now that I've had some experience, trying to do things for myself and making some mistakes along the way and trying to learn. I can take those skills and apply them to the standard curriculum and now do better. As opposed to when I was at Santa Cruz 15 years ago, I was going to major in philosophy. I didn't have a purpose. I didn't have a plan. I didn't know anything about the world or how to think or how to do anything except sit in a class and do what the teacher tells you to do. It was a very impoverished existence. I think it was pretty bad. I think I'm doing much better in life now. I think I made more money. I think my life is so much better for taking ideas seriously.

CM: What is it about the community in particular that enabled that?

ZMD: On, they call it "this part of Twitter", there's this slogan, "You can just do things." We didn't have that slogan at the time. But there was just this ethos of, you can just do things. After I quit Santa Cruz, I started studying math just from textbooks, because the Overcoming Bias comment section made that seem like a cool and important thing to do. These are mathy people. People have a concept of pleasure reading, where you can read a book, not for school, but just because you want to read a book. You can do that for other things, too.

CM: Tell me about the comment section at Overcoming Bias. Thriving, it seems. Same group of people. Discussing these ideas in much the same way that you discuss them. And that kind of empowers you to do that. Did everybody use their real names? Did you know who these people were?

ZMD: It was kind of mixed. Some people did. Some people didn't.

CM: How do you view the real name versus not real name? It's a big part of the community.

ZMD: It's not a big deal to me. A lot of people use pseudonyms. If someone writes a good post under a pseudonym, I'm happy to cite it.

CM: Why are people so concerned with using a pseudonym?

ZMD: Different people in different subcultures from different generations just relate to the internet differently. People have told me that the fact that I use my real name for blog comments makes me seem older. I was born in 1987. I think I was born in this very narrow window where the pre-internet world is still real to me. I think maybe people who have always been online, rather than just since age 10 or whatever, are used to screen names first. I don't know. I'm not sure.

CM: Do you think that the adherence to that is stronger in your community than in others?

ZMD: I don't—

CM: More of a general internet?

ZMD: I think it's an internet thing. Certainly in my community, more so than people pitching pieces to The New York Times, but compared to other internet communities, I think there's a lot of real names. Compared to Tumblr or DeviantArt or whatever.

CM: You said this part of Twitter? So, is that, like, the rationality?

ZMD: I mean—

CM: Twitter? Is that what you mean?

ZMD: It's memetically downstream of the rationalists, but it's a lot of other people.

CM: The other thing about it is, it's very easy to pick out someone from your community. There are certain phrases they use, for instance. People talk about their epistemics. Their priors.

ZMD: It's a good—they're useful words.

CM: Let's start with those words. What do they mean? To the uninitiated, people don't understand—

ZMD: Epistemology is the philosophical study of, how do we acquire knowledge? This is a completely standard word in philosophy. The study of knowledge. How do you acquire knowledge? How do you know things? Um, so, prior...

CM: So, when you say, my epistemics, what do you mean? How you acquired a particular piece of knowledge?

ZMD: Or how do I think I know things? If I criticize someone saying that they have bad epistemics, what that means is, I think they choose to believe things for bad reasons.

CM: What reasons might those be?

ZMD: The first example that comes to mind is kind of confounded with things and might not be the best example. One example, maybe I'll think of a better one later, is trusting authority blindly. You know, maybe it's a religious authority, but maybe it's a particular field in academia, which might not have the best standards. In our society, we have these universities that grant degrees in a bunch of things. I think different fields are more or less trustworthy, partially due to how easy it is to study the thing that they're trying to study, and also just the culture of that field. Is it healthy? Is it not healthy? So, if you want to know something about physics, believing something that a physics PhD says, it's a pretty trustworthy authority. Physics PhDs, they tend to know their stuff. And then there's a bunch of other fields where I do not think deferring to authority is such a great idea. I mean, they know something. They certainly have to write a lot of papers in order to get the degree.

Maybe a good example would be, you know, the COVID pandemic. In the early days of the pandemic, the left-right polarization of beliefs switched sides. In February 2020, there were a bunch of people, saying, like, stigmatizing Chinese people because of this virus is wrong. That's the real problem. And then later, being super cautious about the virus became a left-wing position. If you have people who are deferring to a political authority on questions that that political authority doesn't really have a good reason to know something about, unlike physics PhDs, I would say that's bad epistemics.

CM: Along those lines, don't people in the rationalist community defer to their leaders—

ZMD: Yes. It's a horrible problem.

CM: —on subjects that they don't know about or are completely wrong about?

ZMD: Yes. It's terrible. Sorry, again, I say the movement has failed. Yes, we have failed.

CM: That irony is basically—

ZMD: No, that is a completely fair point. Absolutely, touché.

CM: Isn't that what's going on with this thing between Scott Alexander and me? That people are deferring to—

ZMD: Um, to a certain extent.

CM: You know, you sent those links. And one of the interesting things to me is that, people say, how dare he say this about...

ZMD: Yeah ...

CM: X, Y, and Z. And then they're going to say, we all know that Scott believes this stuff.

ZMD: Yeah, exactly. [laughing] That is kind of funny.

CM: And the other thing that amazes me. So I'm going through this period where I'm trying to get the story out. It's not getting out. I'm getting all this pressure. And somebody sends me this thing about the motte and bailey. They're like, check this out. So, I read, Scott Alexander on the motte and bailey argument. I'm like, come on. What is this nonsense? He completely does that.

ZMD: Well, so, when you say, what is this nonsense, do you mean you disapprove of the idea, or you think that he does it, too?

CM: I thought, wow, this is pretty extreme. I'm supposed to take this blog post seriously? But I should have taken it seriously, because that's what he does. That's exactly what he does. That's what he did when my story came out. It's astounding. It has power. Do you see it that way?

ZMD: Hmm. I understand why Scott was so protective of his last name. It's an unfortunate thing. I have sympathy for Scott being protective of his last name. But the defensive cover-up, where you're like, no, Scott Alexander is a good person, and then other people are like, come on. It is kind of funny.

[...]

CM: This rationalist group house in New York. Highgarden. Did you ever go there?

ZMD: I never went there.

CM: Did you ever go to these winter solstice celebrations?

ZMD: I think the first one I was at was 2013.

CM: What was that like? Was it at Chabot, or was it somewhere else?

ZMD: This was before Chabot. This was at someone's house. Again, I used the phrase quasi-religion earlier. It's kind of like a religious service, where you have inspirational speeches, you have songs. There's a part where they turn the lights out, and then you light candles, and then the lights come on. It's just sort of a secular solstice service.

CM: Do they have set songs from the beginning?

ZMD: I remember some of the songs from the 2013 one are still being used.

CM: Oh, interesting. Which ones?

ZMD: I'm trying to remember. [singing] Tomorrow can be brighter than today, although the night is cold. The stars may seem so very far away.

CM: Were there others you remember? Because some of them were about, you know, singularity.

ZMD: So it's just sort of humanist and transhumanist-themed songs about reason and humanity will overcome the darkness in a cold universe. We care about each other. That kind of sentiment.

CM: Are there one or two others that you remember that were there in 2013 that are still there?

ZMD: There's another one that's, like [singing] in 5,000 years, what you want to do, what you want to be. I mean, these are online, too.

CM: So that was somebody's house here in the Bay Area?

ZMD: Yeah. I mean, it was a big house. Some rich person's house.

CM: Did famous people show up at this? Because in some cases, some known people showed up at the one in New York.

ZMD: I mean, probably not famous-famous.

CM: I think the one in New York, was that 2013, too?

ZMD: I don't know.

CM: It might have been. I don't know, what was the impetus for this? Who put it together?

ZMD: I think Ray Arnold was probably the organizer. [...]

CM: What's his story? Was he involved ...

ZMD: I mean, he's just been around for a while. I think he read Harry Potter and the Methods.

CM: Oh, I see. Was he based in New York or based here?

ZMD: I mean, he's here now. He works for Lightcone Infrastructure now.

CM: And when was that organization created? And why was it called Lightcone?

ZMD: Sorry, you don't already know that? Okay. So, okay, in special relativity, there's a maximum speed you can possibly go according to physics, and so when you draw the space-time diagram, the future that is actually accessible to us, it gets rendered as a cone. It would actually be four-dimensional, but, in the diagram it gets rendered as a cone.

CM: Do you think, I don't know if you've read this, but I've certainly heard it privately. Sam Altman will talk about OpenAI is going to collect a light cone of the world's wealth. Do you draw a straight line from your community talking about the light cone to him talking about the light cone?

ZMD: I mean, probably. Yeah, that's probably where he got it from.

CM: Do you get a sense of the relationship between him and the community and OpenAI, and the community? Is that a mystery to you?

ZMD: I don't know the details. I think it's very unfortunate, because I don't think OpenAI is a good steward of the light cone.

CM: But this is another thing that interests me is that your community used to think that OpenAI was okay, and now, Sam Altman is like the devil.

ZMD: So, this is another thing that Michael and Ben were criticizing Eliezer and Nate and MIRI over, was that, when OpenAI was first founded, Yudkowsky already thought it was a bad idea, a terrible idea. He reports, I'm sure this is on his Twitter somewhere, but he reports, he cried when OpenAI was announced. And then, publicly, Nate Soares on MIRI's blog said something like, I don't remember, we can track down exactly what he said, but it was something like, we welcome a new entrant to this space. It said something vaguely positive-sounding about OpenAI, even though he thought, the sentiment in our community was that open source AI is a bad idea. OpenAI, it was originally pitched as open source. There's this thing where Nate Soares and Yudkowsky privately thought OpenAI as announced was a bad idea, but there was so much power and money behind this new thing, because this was downstream of the Puerto Rico conference about AI safety, there was so much power and money behind this new thing that they didn't feel comfortable bad-mouthing it in public at the time.

CM: Do you think there's some revisionist history going on there, maybe they're saying they were concerned—

ZMD: I believe it. I believe it. It's reasonable to be suspicious, but.

CM: Remind me the name of that rationalist summer camp that went on for a while.

ZMD: SPARC?

CM: SPARC, yeah, yeah. Do you know much about that?

ZMD: I don't know much about that.

CM: So that wasn't, you never attended or participated or helped out with?

ZMD: No.

[...]

ZMD: I'm sorry, I know all the people in my little subculture. I don't know any actual rich and famous people. I think Luke Nosek was at one of these parties, who I think is rich for some reason that I don't remember.

CM: Founder's Fund.

ZMD: I think I remember being at a party where, like, Luke Nosek and Yudkowsky were talking. And I was trying to participate in the same conversation circle, and they were both just ignoring me, because I'm not important or something, I guess.

CM: That alone is interesting. It's interesting how Peter Thiel made a big bet on this community and funded it.

ZMD: I think by his scale, it was a small bet, right?

CM: Well, yeah, but it's about the same amount of money he put into Facebook.

ZMD: Okay.

CM: You know, um, it's interesting that he did it, right, at that time.

ZMD: I think his views have diverged by now. There was that story where he was advising Sam Altman, EAs are going to kill your company.

CM: Okay, just to circle all the way back around. Because I want to make sure I understand. I think it's good for me to get inside the mindset. Let's take that one Overcoming Bias blog post. that you think its so important.

ZMD: Yeah, so, I'd rather not do this live. I'd rather just send you the—I'm a lot, I'm a lot dumber in real time.

CM: Most of us are. Some people are.

ZMD: What I love about blogging as opposed to real-life conversations, when I have time to think, time to think for hours and days, I can eventually come up with a much better argument than I could on the fly.

CM: Well, I mean, if you've got time, I'd love to hear, you know, what, what you thought in detail on that, and that post is walking through. Okay. Uh, this was great as usual. Thank you. It's so interesting.

ZMD: Yeah, thanks.

CM: Ultimately, like, so many fascinating. Somebody told me they think that I should go on to LessWrong and sort of announce myself and, and I don't know, work through some of the, is that a good idea?

ZMD: I mean, I would love to see that. I don't know if it's a good idea for your—I don't, I don't know if it would work for whatever you're trying to achieve.

CM: That's my question. What did it achieve? Or would it just be—

ZMD: I don't know.

CM: What should I say?

ZMD: What goal are you trying to achieve? Like, I guess you're writing a book. Last time I said, I don't think you're the right person to write this book because, as a journalist, one of your prime resources is, people who know the thing who are willing to talk to you. In this particular case, your reputation has been poisoned as, people are not going to talk to you.

CM: What I always say is, with any good story, some people are not going to talk to you, and some people are.

ZMD: Right, but, I think with this particular case, seems likely to be unusually bad.

CM: You know, people always talk.

ZMD: Again, I don't want to be mean, but, I kind of expect Kevin Roose's book to be better than yours, because, based on reading his writing, it seems like he gets it more.

CM: Well, here's what's interesting. There's a difference. What Kevin is going to do is he's going to deliver a book that is completely in line with your community's way of thinking. That's different than it being better.

ZMD: Okay. That's fair.

CM: I think of it differently. What Roose does is he says, all right, I'm going to take this position. I'm just like, the job of a journalist is to step back a little bit and look at everything and really explain to people what's going on. Right. His book will not do that.

ZMD: Yeah, it's hard to know what objectivity means.

CM: Really?

ZMD: Because on the one hand, I think there's this worry about, both-sides journalism where you just quote one person who's supportive of a thing and quote another person who's opposed to the thing and just call it a day. I think it's possible to do better than that. But in attempting to do better, you might try to compile all the facts and present everything you've compiled, maybe you just end up packaging your own view.

CM: Well, I'll think that through, see if it's worth doing. But, yeah, you know, one person I'm talking to, they say, you have to do this. You know, I always get these ultimatums. You have to.

ZMD: Well, I mean, who—

CM: I'm not going to betray a confidence.

ZMD: I would love to see that, just because I have this ideological thing of, I am in favor of information flow and talking to people and I definitely don't trust my in-group to be sane anymore. I think we failed.

CM: That's a valid point of view that will not be in Kevin's book. Right? You're not necessarily right, but it's a valid point of view.

ZMD: But, would that—should I talk to Kevin?

CM: I'm just making a point. I'm with you and I believe in both things, right?

ZMD: Uh-huh.

CM: Good for you.

ZMD: Thanks.

5 May–16 June 2025

From: Cade Metz
To: Zack M. Davis
Date: Mon, 5 May 2025 09:25:59 -0700
Subject: Thanks again and one more request

Hey Zack: Thanks again for spending so much time chatting recently -- and even putting up with my repeat questions.

Could you meet one last time? There are just a few more things I want to run by you:

--one read your memoir very closely. So powerful. There is one part of that I want to discuss to make sure I have it right. Basically, I want to understand why, ultimately, the Rationalist/Yudkowky [sic] view of gender was eye-opening/comforting/inspiring?

--My editors at The Times have also asked me to put together a story I want to let you know about. Pretty straightforward

Thanks,

Cade

From: Zack M. Davis
To: Cade Metz
Date: Mon, 5 May 2025 12:08:53 -0700
Subject: Re: Thanks again and one more request

I'm super-busy with coursework and finals for the next three weeks. Can meet again sometime after May 25.

"Comforting" is definitely the wrong word!! "Sexual Dimorphism in ..." is spending a lot of time talking about how I came to disbelieve things that I used to believe in passionately. When someone loses their religion in favor of atheism (because they read up on evolution, the big bang, &c. and found it more convincing than Genesis), it's not because atheism is more "comforting" and "inspiring". It's because a harsh truth is more useful than a comforting lie.\

From: Cade Metz
To: Zack M. Davis
Date: Mon, 5 May 2025 12:46:33 -0700
Subject: Re: Thanks again and one more request

Now that is well said! Thank you.

Would love to meet again after March [sic] 25.

Cade

From: Cade Metz
To: Zack M. Davis
Date: Mon, 9 Jun 2025 09:21:34 -0700
Subject: Hello again

Hey Zack: Hope you are well. Thanks for the call two weekends ago. Really enjoyed chatting.

So, if it isn't obvious, I am truly interested in what you are dealing with. You are already an important character in my book, and I think you can be even more important.

I am interested two additional things, if you are willing:

--would love to talk one more time about your memoir and your disagreement with the community

--would love to see your diary entries from when you first started joining Overcoming Bias/LessWrong meetups. Real contemporaneous details are always more trustworthy. I am just looking for a few details. This would help

Feel like chatting over the next couple of weeks?

Thanks,

Cade

From: Zack M. Davis
To: Cade Metz
Date: Sun, 15 Jun 2025 19:22:15 -0700
Subject: Re: Hello again

Thanks for your patience. I'm very confused what this book is about; I should not be an important character in a book about (especially) AGI or (probably) the rationalists.

I'm happy to talk about my personal war because it matters to me; it just doesn't seem like it should matter at all for your book. Was I correct to guess that fewer people have been willing to talk to you for coverage of the "rationalists" after the Slate Star affair, compared to other stories you've worked on? If you're starved for sources and end up promoting random people who are willing to talk to you to "important character", that seems like a bad sign regarding the quality of the end product.

Your question on May 5 made it sound like you had read "Sexual Dimorphism in ...", which is only "part 0" of what I've been calling my memoir: part 1, part 2, part 3, part 4, part 5 forthcoming.

If you still want to chat more, pick a time next week (the week of the 22nd)? (I need to get my writing routine focused this week, end the disruption from being depressed after Less Online and the Reproductive Frontiers conference at Lighthaven last week.)

Let me get back to you about the Diary entries (I don't think there's too much there, but I'm waiting on guidance from Anna about whether I'm supposed to redact anything; also reserving the right to back out if the forthcoming Lighthaven/Solstice piece is sufficiently terrible to override my bias towards Sharing Information without trying to control how it gets used).

From: Zack M. Davis
To: Cade Metz
Date: Mon, 16 Jun 2025 23:30:38 -0700
Subject: Diary excerpts

Attached are some of my old Diary entries from that might give a sense of the subcultural scene. (You can see from the entry-number gaps that I tried to cherry-pick entries that looked at least vaguely relevant to your story, skipping over entries that seemed less relevant or full of histrionic garbage.) These selections go from one entry in 2005 (first mention of Yudkowsky), to Overcoming Bias meetups in 2008, to a brief you-can-sleep-in-the-garage-and-we-won't-pay-you internship with the then-Singularity Institute in 2009, to subsequent visits with the gang in 2010.

Attachment: siai_diary.pdf

24 June 2025

(audio)

ZMD: Okay, we're recording. I am a little bit worried that you just summarized it as, oh, these people claim to be rational, but they're actually just following the beliefs of this guy. And like—

CM: What I do, so you know. I think you underestimate your importance and the importance of this community in the grand scheme of things. The piece that hopefully we'll publish at the Times here pretty soon just says very simply that you have this community here that most people don't know about that has become enormously important. These ideas have worked their way into the tech industry and these companies that are building this very important technology, and the world is now having to grapple with these ideas, right? The ideas are—part of it is, you know, rational thought, let's call it, but then, you know, transhumanism and as Scott Alexander says in that blog post, there are other people who believe in rationality, there are other people who believe in transhumanism, there are other people who believe AI is a danger, but this community has fed all that into the industry, and you can show this very easily.

ZMD: That's true.

CM: It's also through a lot of my reporting, where Yudkowsky introduces Peter Thiel to the DeepMind guys, and his thinking, and the thinking of a lot of people in his circle, was instrumental in the creation of OpenAI.

ZMD: To our—to their deep regret.

CM: Yeah, there's another side, we can talk about that later, but also the creation of Anthropic and it continues to drive the thinking of Anthropic. The community is deeply entwined with Anthropic. There's this MATS program that is held at Lighthaven and they're feeding people into the company—

ZMD: I was actually just reading, before you walked up here, I was reading a comment from Nostalgebraist responding to Evan Hubinger, who is the lead, I think it's the alignment stress testing team at Anthropic. I guess, to me this is normal, but to someone who thinks more in terms of institutions than random internet blog weirdos might find this kind of surprising, because Nostalgebraist is the pseudonym of—I don't know what this person's dayjob is, if they even have a dayjob, and Evan Hubinger actually works for Anthropic, which is a tech company that you've heard of.

CM: You call them the internet weirdos, we can call it something nicer like the rationalist, the rationalist-slash-EA community or whatever you want to call it, it is this giant force in that world that in some ways has more power than the companies.

ZMD: We wish. We wish.

CM: It's a different power, but it's power, and nobody knows about it. What I'm doing is explaining this to people, and showing how this happened. Where did this community come from, how did these ideas evolve.

ZMD: And that's what the book is about?

CM: That is exactly what it is—these ideas, where did they come from, and how did they evolve, and how did they move from person to person, because these are powerful ideas. Does that make sense?

ZMD: Do you have a title?

CM: I don't have a title, but I've written almost all of it now. That's what it does. It shows the evolution of these ideas and then how they move from important person to important person.

ZMD: Well, so that's—we've mentioned criticisms of you in the past, and I want to distinguish between two different types of criticisms, because there's some people who are saying, oh Cade Metz is such a meanie and bully, revealing Scott Alexander's name, and I think you don't take that very seriously. Whereas I want to set that aside, and also focus on—separately, people who have talked to you and who have read your work, who are saying, this is kind of like gossip column-y coverage that's not focused on the ideas and more focused on powerful people and their relationships to each other, and I don't know, maybe you're doing that on purpose, but it just seems like—

CM: What's an example?

ZMD: So my friend Ben Hoffman mentioned that he talked to you. How do you remember that conversation?

CM: Oh, I guess—I thought he was seriously going to talk to me. Now I think he was never going to talk to me.

ZMD: Well, no, I think he would have if you were a different kind—if you—so he described it on his blog as ... I can just send it to you later.

CM: What'd he say?

ZMD: "When Metz reached out to me in the process of working on a book on the idea of general intelligence, I formed the same independent impression; if I asked him about his interest in the topic, his perception of my relevance, etc, the answer was entirely in terms of social reality. What was really striking about this was the perfect serenity with which he was enacting a pseudoperspective which functionally constitutes total and perfect aggression against anyone who cares about anything. I didn't think it was worth my time to proceed." End quote.

CM: A, that doesn't surprise me. So many people in this—this is not rocket science people, people in that circle don't view the world the way I do. I view it differently. I'm trying to take what's going on in that world, and convey it to the layperson. That is fundamentally going to be different. The writing that I do is completely different from, say, your memoir. Completely different. The layperson cannot comprehend your memoir. Because you think differently. Maybe you think better. Maybe that's a better way to think about the world. But people don't think that way. I'm taking all this stuff and I'm conveying it in a completely different way, and if Ben doesn't like that, that's fine.

ZMD: So I definitely agree that you're writing a book for the general public, you can't just do a raw thought dump of what the people you're covering think, because they're not speaking the same language as the general public.

CM: But where he's wrong is, I care deeply about the ideas. This is why I'm talking to you. This is why this is so interesting, and why your diary was so interesting, because what the diary showed are the ideas that people were thinking about at that time. I love there's this moment where you go back to the group house to the meetup, and you're talking about, the Overcoming Bias community, and the topic really on the table is rationality and bias, but it goes towards this transhumanism stuff. It shows that was just part of it. So I want to explain the idea of transhumanism to people. I want to explain the idea of rational thought to people. I want to explain why these things intersect in this community. I want to explain all that through the people, and through the things that happened to people.

ZMD: Do I get to read excerpts of the book? I know for the article, I know the Times has a policy against that, but does that apply to the book?

CM: What I will do is, I will tell you exactly what the facts are that will be in it. It will say that you did this on this day, you said this.

ZMD: Okay, so, fact-checking.

CM: We'll get on the phone and I will fact-check it.

ZMD: All right, but again, there was the thing with Kelsey Piper, where you fact-checked the exact quote, and the exact quote was very clearly not what she meant [...]

CM: What I'm having to do, is I had to boil down what she said to plain English.

ZMD: But it didn't work.

CM: It did work.

ZMD: No, it didn't.

CM: How would you describe what she said? That's what she said. I said, there are a lot of people who see Scott this way, there are a lot of other people who see Scott that way, I'm going to represent both views, and so she said, you know, what you really need to do is find a way of statistically—

ZMD: No, that's a terrible paraphrase. Here, let me bring up my—because in my interview with you, I included a little editor's note of what Kelsey actually said to me afterwards, that wasn't very many words.

CM: Is that necessarily what she said to me?

ZMD: So without casting blame, without—you know, maybe it was her fault for giving a bad interview and not pushing back enough during the fact-check call, but separately from whether how much of it is your fault and how much of it is Kelsey's fault, the thing that was actually printed in The New York Times was not a good paraphrase of Kelsey's views as I understand them. "Editor's note: when reached for comment, Piper said that she told Metz that she "wasn't a huge fan of 'both sides' journalism, where you just find someone with one opinion and someone with"—I mean, I don't have to read this whole quote, but I have this paragraph of what Kelsey said to me afterwards, and then people who read my interview on Twitter kind of making fun of you—

CM: Of course they are.

ZMD: But I think they had a point, is what I'm saying. I think they were in the right there.

CM: Well, I think that there are plenty of people—I didn't talk to one person. I talked to all sorts of people.

ZMD: Right, but this is the same thing that Ben is talking ... "What makes an article good reporting? Fairness. What's fairness? If all sides get the chance to defend themselves. Absolutely zero recognition of any underlying reality about which an accusation might be made, or communication attempted."

There's this thing where if you're just counting heads of, these people say this thing, but on the other hand, these people say this thing, and you don't—you're still in what we would call social reality, where you're talking about people and what they think, and there's a separate layer underneath social reality, where you investigate things yourself, and there's this awkward recursive thing, when you do in fact investigate the things themselves yourself, then that ends up just being yet another perspective, and you could enter that into the social discourse of, Cade Metz thinks this-and-such. If you just cover what are these different people, and here's what these different people think, without investigating things yourself and without investigating the base reality that's not about what people think, but just the things themselves, then you're you're kind of ungrounded.

CM: I see what you're saying, and that's sort of the fundamental belief that you can have this absolute truth. It gets around to what you're saying is, the community come to this absolute truth, and then you've got somebody who's saying, actually we don't believe that truth, we believe something else. From my perspective, can we really be sure what that absolute truth is?

ZMD: We can't be sure, but we can try, we can do our best.

CM: That's what I do. I really do my best. I just do it a different way.

ZMD: I'm just saying, when we get to that fact-checking phase, I want to be more diligent than Kelsey, because I don't—with respect, I don't trust you to get the story right. I'm happy to talk out of the principle that I'm happy to talk to anyone. I don't have faith that you're going to get it right. When we get to the fact-checking phase, whatever you can show me, I'm going to be watching like a hawk.

CM: What's so interesting to me is, as you discover, I just think that your story is so interesting, because you kind of knew who Yudkowsky, mentioned in your diary, you read Great Mambo Chicken, you really like it, you're kind of intrigued by these—I'm kind of doing fact-checking now—you're intrigued by these transhumanist ideas. You said that future tech is comforting, because it's kind of like religion, but more real, more plausible. And then you're reading that Jane Galt blog and she links to that "Universal Fire" essay, and you click on it, and you read this thing, and in my story, I explain to them, the Universal Fire essay captures your interest, and you keep reading this stuff, and then you meet these people in person, and that excites you; you see that in your diary. And you get more and more into these ideas of rational thought, but also, as you show in your diary, you get into this idea of the singularity. You tell people, I'm down with this, too, right? And then you reach this moment, where you're comforted, also, by this notion that there is this fundamental truth, and that includes this post from Yudkowsky, where he says men and women are fundamentally different and you can't change, right, and I go into that essay.

ZMD: [laughing] Okay.

CM: It's a way of showing these ideas, right? Your eyes are opened by this, you do a good job of explaining it, in an email you send to somebody, where you say this is shocking slash et cetera. There is this comfort, for you in this moment, where you feel like there is an absolute reality, right?

ZMD: Well ... I mean, I agree that there's an absolute reality, but I wouldn't have quite phrased it like that. What I found so striking about that post is just, it was very very clear about something that I had never seen any author articulate so clearly, which is that, yeah, you could fantasize about changing sex, and you can write a fun science fiction story about it, but he points out like, look, there are lots of technical details that you would have to settle here.

CM: The clarity. Clarity was comfort. You know you had confusion, and suddenly you had clarity. That's a great word.

ZMD: I do use that word a lot, yeah.

CM: It's a really good word. In some ways it's a metaphor for this whole movement; it gives people clarity.

ZMD: I mean the thing that the whole memoir is about is expressing rage and frustration that—

CM: I haven't gotten there yet; that's what's so interesting. You, like a lot of people, get comfort from this clarity and this very cool calculated way of viewing the world. You fast forward to 2016 when has the Facebook post, and then he blows it all up, and then that clarity is lost, in some ways. The comfort and the clarity is lost.

ZMD: So again, I don't like thinking in terms of comfort.

CM: Tell me, yeah.

ZMD: Part of the reason I dislike the word comfort almost as much as I like the word clarity is because people often—I don't know if this is like a human universal or just in our Society these days—but people very often do this thing of trying to shut down rational, or trying to shut down inquiries, shut down discussions, on the grounds of, this makes people uncomfortable.

CM: But that's a separate issue, this is about how you feel in the moment.

ZMD: But the feeling is not one of comfort. The feeling is like, it's—

CM: Clarity.

ZMD: Yeah, but clarity isn't—it's not comfortable, it's like—in Paul Graham's writings, he says something about how people who work at startups as opposed to big companies, they seem more worried but also more alive. It's more like that. It's not like being clear and telling the truth even when those truths are genuinely uncomfortable, it's not comforting, but it's like a more intense and vivid way to live. Whereas delusions that flatter things—well, wouldn't it be nice if that were true—are often more comforting.

CM: But then that clarity is threatened by this Facebook post. So tell me what you feel in that moment.

ZMD: Well, at the time I'm just very confused. Like, wait, what? Clearly this is hugely in tensions with this thing that he wrote, you know, seven years ago, like, what's going on?

CM: It causes confusion.

ZMD: Yeah.

CM: And does the confusion continue, like how do things—

ZMD: I don't know, is it—sorry, point of order, what is the point of verbally talking through the thing that I already wrote down?

CM: Because you've written something down that is several thousand words.

ZMD: Okay, fair enough.

CM: I have to paraphrase, like with Kelsey Piper, I have to boil it down. So when we talk like this, it helps me boil it down—

ZMD: All right.

CM And when we get to those important works, like clarity—

ZMD: Fair enough. Fair enough. I don't think you're very good at paraphrasing, but we'll try it.

CM: Think about it this way: I'm different at paraphrasing.

ZMD: Speaking of comfort, that is not comforting!

Okay, so basically that was like the first note of, wait, what's going on here, and then through 2016, I was suddenly, like, wait a minute, after the idea that, wait, trans women are the same thing—because, you know, he's clearly talking about guys like me, you know 30% of the ones—like I am obviously in that 30%. I start talking to people I know in Berkeley, including a lot of trans women in Berkeley, and suddenly being like ... you know, I can't—I mean, I wrote—there were a number of private conversations, um, I wrote what I could in that post, where I gradually just like—it became very clear to me that a very large fraction of trans women are people—guys like me, and this was like very surprising, because before then I had assumed, I mean, in the Diary, there's so many obvious hints in retrospect, but in that Diary, you know, seven years earlier, I had been like, you know I wrote about—I don't know if you noticed that, it was just a few lines—but I wrote about gender stuff, but I assumed that like my thing was not the same thing as being transgender, because my thing was obviously this autogynephilia thing. That was obviously a way more informative word than gender identity, which, like, what? So this was both surprising and I thought, given the premise of, we are the rational thinkers, I thought more people would be interested in clarifying this in public. In places like Berkeley, there's this entrenched by now, but at the time suddenly very popular idea, that some assigned male at birth people have this gender identity, and that makes them relevantly the same sort of thing as cis women. And that is just not my model at all.

CM: What you're saying is that is not the model that you thought the rationalists had, but they do. Why do they have that model even though you think they shouldn't?

ZMD: This is the point where "the rationalists" stopped being a good abstraction. I think what happened was, there was the very, very sexist thing that Yudkowsky said in 2009 when this was just a weird blog on the internet, and there wasn't a lot of political pressure for him to say or believe otherwise. And then seven years later, as the political incentives in Society in general had shifted, and his profile, and the political pressures on him in particular had shifted, that article he wrote in 2009 is not something he could have written in 2016.

CM: So the political pressure, the personal pressure, is getting to him, and changing what he's saying.

ZMD: Obviously, I'm not a mind reader; I can't know that for certain. But if you just look at, here's what the guy said in 2009, here's what the guy said in 2016, it's like, what do you think happened, right?

CM: Exactly, and then so this is why I feel like it's somewhat analogous to what I have seen, because I don't think the community can think objectively, if it ever could, about what I'm writing or what I'm doing. There are these pressures, social pressures, to think a certain way. Does that make sense?

ZMD: In terms of people preemptively dissing you because they want to signal allegiance to Scott Alexander? Yeah, that's definitely a thing, I agree. That's why I was trying to highlight, the thing that Ben Hoffman wrote about you, was not doing that.

CM: Well, I don't know what's going on with Ben, but Ben, when he talked to me, he's like, yeah, I'll talk for your book, that sounds great, give me a call next week, and then, you know, he never responded to me, and then next thing I know, so, I don't know what's going on there.

ZMD: Sorry, I think I missed a sentence in there; he said call me next week, and then what happened?

CM: And then he never responded, but I did call, right.

ZMD: Okay, but you did get on the phone eventually, right?

CM: What I mean is we talked on the phone, a long time, I told him exactly what I was doing, he said, yes, that's something I would like to participate in, let's talk next week, I can't do it right now. So I don't know.

ZMD: Because in his blog post, he mentions, "I didn't think it was worth my time to proceed."

CM: In any event, the word you keep using in your memoir, is that they're fake, so why is it fake?

ZMD: I mean, see, this is again this is why I'm reluctant—this is why I'm like really—

CM: Yeah, I know, but you've said it—

ZMD: Well, right, but like—

CM: I'm just trying to help you; I'm giving you a chance to explain. You don't have to.

ZMD: I want to contextualize that, in the sense that lots of things in Society are fake in the same sense. School is fake. Work is fake. As someone who had read the Sequences as they were being written, there was this very very high standard being articulated, of, we are going to pursue the truth, not just in the sense of we're going to say literally true sentences, but we are going to use a cognitive algorithm that would have returned a different answer if reality had been different, which is a much higher standard than only using true sentences, for the same reason that politicians and used car salesmen are very very good at getting people to believe and do what they want them to do and believe, without technically lying, because you just cherry-pick. A lot of journalists are good at this, too: just cherry-pick the particular facts that you want in order to weave the narrative that you want. Yudkowsky's Sequences had articulated this much more ambitious vision holding oneself to a much, much higher standard. We're not just going to not lie; we're actually going to get the correct model.

And then it turned out that, at least on some topics where getting the correct model would be perceived as political suicide, it turns out, okay, we're not actually going to do it on those topics. I think that the disagreement with my other community members that's being expressed in this memoir is, for me, at least, this kills the whole project. Lots of people I know are making these huge, huge social and medical decisions on the basis of information that I think is just egregiously misleading. If the rationalists can't even get this one right, that's actually personally relevant to us, what is even the point? I think from the perspective of the people I'm disagreeing with, they think that, well, you know, we will avoid politically sensitive topics, but that doesn't kill the whole project, because we can just carefully not lie and carefully avoid sensitive topics, and still focus on the thing that actually matters, which is AI risk. I can understand that and I can respect that in certain ways, but I'm mad about the misleading marketing. I go into this in part four. I think I do have a false advertising complaint. If you are going to follow the strategy, which I can understand and respect, of, okay, we're not going to do heresies, we're not going to do thoughtcrimes; we have more important things to focus on, then you should not really be marketing yourselves as, we are the only sane and rational people in the world, because on certain topics, there are other people who are facing less political pressure than you, who are going to be better than you on those topics.

CM: When you say they're getting political pressure, specifically what is that pressure? Where is it coming from, and why do they have to bow to it?

ZMD: Because they're worried that someone will write a New York Times article insinuating that they're racists or sexists or transphobes. Which actually happened, by the way. That wasn't your intent, but that was—

CM: I didn't say that. What I pointed out—well, we won't go into that. They're worried about people calling that out. Why are they worried about that?

ZMD: In the late teens and early 2020s, we saw that cultural trend of cancellations. You lived through these years, I don't feel like I should have to explain this.

CM: So you're saying that because of that trend, Yudkowsky and the people around him are bowing to this because of that.

ZMD: I think so.

CM: They're bowing to the external cancellation pressure.

ZMD: Because remember the James Damore thing? No one wants to be James Damore, right? And especially if you have this movement that you think is uniquely important for the entire future history of the world, you don't want the thing that happened to James Damore to happen to your project.

CM: I got it. So I guess my question is, when you say the whole enterprise is called into question, you have doubt over whether or not it can hang on to the real truth in this one area, does that reduce your confidence they're willing to hang on to the truth in other areas?

ZMD: Yes. Yes. It does reduce my confidence. From an outsider perspective, you might ask this natural follow-up question of, okay, so why are you still scared of AI, right? And the answer there is that, my beliefs about AI as an extinction and existential risk are not solely based on deference. I actually read the arguments and I can actually check a lot of the arguments there and say, this part, this part is still real.

CM: But you've also told me in the past that you can't necessarily trust this group's opinions on x-risk.

ZMD: Just, in general. I do think x-risk is real. I do think the world should be paying much more attention to getting this stuff right, and I think this community is the place where people are paying attention to it, but that doesn't mean you should just defer to what these people already think. It means you should think for yourself. I would love for more people with more diverse backgrounds to read this stuff and think about it for themselves, instead of just copy-pasting Yudkowsky's beliefs, which—

CM: So ultimately you still believe that there is this absolute truth.

ZMD: Yes.

CM: Do you get that some people really question that?

ZMD: I think they are confused about some things.

CM: The other fascinating thing is when you talked about how it changed the way you read something like Yarvin. Explain that to me.

ZMD: [laughing]

CM: What you said is after this situation, it changed the way you read that. Explain that.

ZMD: [laughing] I'm laughing because like this is exactly the reason that everyone's like, don't talk to Cade Metz, because this is the part they don't want to talk about.

CM: It's so interesting.

ZMD: But I'm happy to talk to anyone. I had read Yarvin, writing as Moldbug, back in the late 'aughts, just because I read lots of things on the internet. It was something I had read and not taken very seriously, and then in light of suddenly everyone I trust are saying that guys like me can be women by means of saying so, which is just so transparently absurd, Yarvin's model of the political pressures and the asymmetry between the left and right in the US—sorry, I'm not nearly as much of a Yarvin scholar as a Yudkowsky scholar, so I don't think I can summarize the model briefly verbally.

CM: What you're saying is that model started to make sense to you.

ZMD: Yeah, it started to seem much more credible in a way that previously I just glanced at it and shrugged, and now I was looking at it and being like, wait a minute, there's definitely something real here.

CM: That's really interesting. So basically that the political pressures can really get to people and shape what society does. That's so interesting. So can you explain to me why these communities at the very least are adjacent to the neoreactionary thing, and the rationalists—

ZMD: I mean, they're not that—they're a little bit adjacent.

CM: Why does everybody talk about it?

ZMD: The reason the rationalists are a target, the reason they're scared is precisely because we're so good at, we're pretty good—look at me, I'm using the first person again. "We." We, they, not sure which one is right—they're sufficiently good at free speech and free inquiry that they're willing to engage with this stuff at all, whereas people—so I understand that for The New York Times piece you talked to David Gerard a lot.

CM: I talked to a lot of people.

ZMD: Basically in our Society, people who just work at a university, people who work at, you know, San Francisco State, are not going to engage with far-right thought at all. It's just unthinkable. But as internet weirdos, we're happy to think about anything.

CM: In fact, you're kind of determined to engage. It's like a badge of honor.

ZMD: So, me personally, I ended up actually agreeing with parts of this, but other people who are part of the community who just read it and disagreed, there's still this threat of, someone else could say, aha, you read the evil text, even if you say you disagreed, you're still tainted.

CM: I get it. I get it. It's so interesting how there is this determination to engage. I think that that's interesting. And that's all I'm saying in that Slate Star Codex piece.

ZMD: Read the first section of part four; I think I explained my issues with it.

CM: I understand, I think it boils down to, again, you're kind of paraphrasing, right; I've got to boil stuff down, I have space. I get it. We've gone over that.

ZMD: Okay.

CM: The other thing that's so interesting is the community's almost obsession with intelligence, whether it's people or machines.

ZMD: Yes, that's definitely real.

CM: Where does that come from?

ZMD: I've always been this way, in a way that it makes sense that I fell in with this group, because I was like this beforehand.

CM: They probably all were. There's this great moment in your diary where you're talking with Anna about her being surprised that you're in the 97th percentile on SATs. It really shows that. All of you are so concerned with the intelligence.

ZMD: I think there's a subtle point of that anecdote that you might not have picked up on, because it's where she said she was surprised, I said she was surp—I don't.

CM: You were surprised that she was surprised.

ZMD: Right, and then she was like you should expect—sorry I don't have the text in front of me, but there's this like deep—that last thing she said of, you should have expected me to be surprised—there's this deep Bayesian—I don't know how to articulate it in words in real time. See, this is why this is what's so frustrating about journalists who prefer to talk in real time is that I'm so much dumber. Okay, speaking of obsession with intelligence, I'm so much dumber in real time, like having a live conversation.

CM: Everybody is. Everybody is.

ZMD: That's why you like it.

CM: No, no.

ZMD: Because it's faster.

CM: No. We're dumber in some ways; we're smarter than others.

ZMD: Okay.

CM: We can have a back-and-forth; you can correct yourself; you can respond to what I said. You can't do that on the page.

ZMD: But on the page, I like text better because when you have full minutes or even hours to think, you can actually get it right.

CM: A lot of people in that community are probably like that, too. Scott Alexander told me he didn't want to meet with me. He wanted it all to be in email. And there's a control there.

ZMD: But anyway, there's a thing that I can't describe verbally but I could describe if I spent fifteen minutes thinking about it and then writing about it, the thing that Anna was doing there was an intuitive expression of Bayesian reasoning.

CM: And the other one, when you show up at that group house in 2008 after the singularity stuff in San Jose. And you guys are talking and then the singularity comes up, and the way I think about it is, that the Sequences weren't about x-risk, singularity, and AI, but that is on everybody's mind because that's on Yudkowsky's mind, like he's been about that for years.

ZMD: So the Sequences are a propaganda move of, I'm going to teach you how to think, and part of the reason I'm motivated to teach you how to think is because some fraction of those people will be recruitable for my singularity thing. He was very explicit about this.

CM: He was. Completely explicit. That's fascinating.

[...]

ZMD: So Yudkowsky had written about this idea of timeless decision theory. Among academics who have written about decision theory, there's this question of how would you make decisions in scenarios where other agents can predict what you would do. So the motivating example was Newcomb's problem. You get to choose two boxes—have you covered this already?

CM: Wrote it in my book.

ZMD: Okay, great, so I don't need to explain Newcomb's problem.

Basically there's this idea that if you're using causal decision theory, you might reason, well, you know the thousand dollars is already in the second box or not, therefore I can just take the box because it's already there; my decision doesn't affect it. And then Yudkowsky is saying, no, this is wrong, because actually you are an instantiation of a decision algorithm such that if you are the sort of person who would take the second box, the predictor has already taken that into account, and the box will be empty for you.

CM: Basically what he's saying is we might be in a simulation. I mean, how else does that work unless we're in a simulation?

ZMD: I mean, you don't necessarily have to be in the simulation, it's just that if someone else can predict, if some other agency can predict you sufficiently well, it doesn't matter if their prediction itself is a simulation, but if the prediction is accurate, that's still enough to make the Newcomb's things work. Nate Soares even has a blog post about, Newcomb's problems are common in real life, which argues that to the extent that people can are good judges of character, and you can say, ah, this person is someone I can trust, that is a kind of timeless decision theory. I can trust you to make a deal, because I know that you won't betray me even when it would be in your interest to suddenly betray me when you could get away with it, but because I know you, the fact that I know that if I can predict that you won't do that, that's why I'm willing to trust you in the first place.

CM: Yeah, but we all know no one can really predict the future, and there's always a chance that—

ZMD: Yeah, there's a chance but if you can do a good enough job, that can still be enough in some scenarios. If I can probabilistically predict that you probably won't betray me, then depending on the exact probabilities and utilities involved, that might be enough for me to trust you.

The smart version of this idea for futurism is imagining that like, if you have like superintelligences that originated in different parts of the universe or in like different branches of the universal wave function, that care about—this is a weird, mostly irrelevant philosophy thing—but if you have super intelligence that emerged in different parts of the universe that value different things and somehow the options that they have are calibrated such that, you know, we're in separate universes; we can't actually interact, but like I really really like making blue things, and you really really like making green things, you can imagine them making a deal such that I make some green things in my universe. You compute a trade deal without actually communicating, but just predicting what trade offers the other party would accept, and so I make some green things in my universe, and you make some blue things in your universe, and so there's gains from trade without actually communicating, but just by predicting which trades the other party would accept if you could communicate.

CM: The other possibility is, we live in a simulation.

ZMD: I mean, yes. Some possible implementations of this idea might involve simulations, but like the decision theory isn't actually about that.

CM: These are such interesting ideas. You do wonder how much of this really applies to reality, right?

ZMD: The stuff about acausal trade was very much an abstract philosophy thing, and not a we need to write this code right now thing.

CM: A lot of this seems like abstract philosophy. I was interested in the essay that Yudkowsky wrote when he created Less Wrong. He talks about what rationality is. At one point he says, you know, Bayesian reasoning doesn't really work in the real world. The math just can't calculate it. And that's kind of true, as much as everybody's talking about Bayesian reasoning, this mathematical thought, it doesn't really work.

ZMD: So the analogy—you've probably already seen this analogy—but if you're trying to catch a baseball, the way that you actually do it in practice is not by solving the equations of motion in your head, and yet Newtonian physics is still pretty useful. We still want some educated people to know this stuff. It's very much the same thing with Bayesianism, where, okay, you're not going to explicitly use Bayes's theorem to do the cognitive equivalent of catching a baseball, in principle, this is the governing law.

CM: Got it. Super, super interesting. It is about the ideas. I appreciate you talking about all this stuff. I wish Ben would too, but it's his prerogative not to.

ZMD: I think if you had impressed him more, he might have. I don't think it was unconditional.

CM: I get it, but I told him well as I could, the truth, and if he doesn't go for that, it's fine.

ZMD: I mean, he's busy. He has two kids. They're like a toddler and a nine-month-old baby, I think now.

CM: Have you heard of this notion of hyperstition?

ZMD: Yeah, I've heard the term.

CM: I think about a lot like the community has sort of brought about the thing that it feared.

ZMD: Yeah, we're pretty sad about that. People have noticed this and are kind of sad about it.

CM: How do you think about that?

ZMD: The world was going to build AI at some point anyway, so if talking about these galaxy-brained ideas about human extinction and transhumanism and the singularity. If talking about it made it happen sooner, but also made Society better prepared, that might be a worthwhile trade-off, rather than it happening a little bit later, but it taking people even more by surprise because intellectuals refuse to talk about it.

CM: Isn't there a really good argument to be made though, that the race we're in now is what's really dangerous?

ZMD: That's why Yudkowsky and Soares have said, creating—I mean, there was a part where like, there was this weird thing where when OpenAI was announced, Soares posted on the MIRI blog—

CM: Saw that [inaudible]. Said he was excited.

ZMD: And like privately he, I mean he has a rationalization of why that was not technically lying, but like, functionally, that's a lie.

CM: Do you think he was purposefully lying in the moment?

ZMD: He was trying to—again, I don't have insider knowledge here, this is just my understanding from like, gossip channels and stuff, but like—he didn't feel comfortable saying in public, this sounds like a horrible idea, what the hell are you guys doing, this is terrible. He felt like that would not be—because you know, there was so much money and power behind this OpenAI thing, that he didn't feel comfortable criticizing in public at that time. It's probably on his Twitter somewhere, but Yudkowsky said somewhere that he cried at the OpenAI announcement.

CM: It is interesting, separately, Paul Christiano and Dario Amodei, they were actually in talks with those guys. They didn't join the lab, but then they joined a little bit later. It seems because they wanted to change the culture. OpenPhil came in with 30 million dollars.

ZMD: Did you see, speaking of Ben Hoffman, he had a post about "An OpenAI Board Seat is Surprisingly Expensive".

CM: Did he say that at the time?

ZMD: It was contemporaneous, yeah. I'll send you the link later.

CM: It's funny nowadays that crowd, they sort of deny that they did all this, but it's all there on the internet. We're making this donation. We get the two board seats. We're going to oversee AI safety.

ZMD: They thought they could exert more control than they ended up exerting.

CM: And it has, in the end, it has the opposite effect.

ZMD: Christiano has written somewhere that he didn't think an equilibrium of DeepMind is the only one who works on AGI forever, he didn't think that was stable anyway.

CM: I really question this idea that this is going to be built no matter what, so we're going to build it.

ZMD: Yudkowsky has changed his mind about this. There's the book coming out in September, If Anyone Builds It, Everyone Dies, that's lobbying for an international treaty to not build it. So nowadays the plan is, our only hope—this probably won't work, but our only hope is to get governments to make people not build it.

CM: I'm fascinated by the Anthropic philosophy where we're going to build it and create this race to the top, they talk about. What they did is they came into OpenAI, and then they scaled it up. They were the ones that scaled it up. It's not like anybody else was—they scaled it up, and their argument is, you got to be working on the technology to do safety but—

ZMD: I'm kind of sympathetic to that.

CM: Tell me why.

ZMD: The logic of if we don't do it, someone else will—I think it's true on some timescale. If we don't do it, someone else might not do it immediately, but like as time goes on, and economic growth continues, eventually someone is going to get around to it. The idea of, we the foresighted people are going to do what we can to take leadership in doing this in the most responsible way, that we can, like—I won't necessarily say—I'm not going to necessarily full-throatedly agree, but it doesn't seem like an entirely crazy thing to think or try.

CM: It does seem like that classic argument between free will and determinism. It's this very deterministic stand. It's going to happen no matter what, so we might as well do it. I think it's one way to see the world. I don't know. I think there's another way to look at it. Is that fair?

ZMD: What is the other way?

CM: That if you scale this thing up as quickly as you can, you're creating a lot of the problems that you sought to avoid.

ZMD: Yeah.

CM: We'll see how it all plays out. Have you read Yudkowsky's book?

ZMD: No. It's not published yet, and I am not on the pre-readers list.

CM: The other reason that they're scaling it up and continue to push for this, is they think it's gonna bring utopia. That's the flip side. It's like this flip side where they're worried about destruction, but what they really—

ZMD: LLMs are already economically useful. I do use these things all the time.

CM: That's different from it solving world peace and curing cancer.

ZMD: Yeah.

CM: Everybody thinks they're striving for this utopia, but worried about destruction. It's just, like, biblical.

ZMD: Yeah, pretty much.

CM: But it's not a religion.

ZMD: Yeah. That's right, that's right.

CM: That might be the end of the book, and it sort of sums it up. One other thing, because you pointed to that thing from Scott Alexander, where he sums up what rationality is, like I talked about before, and you talked about it as like, joking, but not. I think that's right.

ZMD: I don't remember which thing you're talking about.

CM: Where he says what are the rationalists really—

ZMD: The belief that Eliezer Yudkowsky is the rightful caliph.

CM: Exactly. I think you sum it up well, it's true, sort of a joke, but.

ZMD: Well, yeah, and that's why I got so mad and spent the last nine years of my life trying to fix this stupid cult, is that I actually believed that Eliezer Yudkowsky was the rightful caliph, and then it turns out he's not interested in living up to that standard, because it would be politically inconvenient when you're trying to run a non-profit in Berkeley.

CM: So that was the way the community worked, but the caliph wasn't standing up for what he should.

ZMD: I mean, like maybe it's possible that was the right decision from an existential risk management perspective, of let's not venture—I mean, as you've noticed, a lot of people around here are eager to talk about everything, including all the heresies, but the leadership isn't going to be serious free speech free inquiry maximalists, and maybe that's the right decision from an existential risk management thing, but I think if you're going to do that, you should be humble about not insinuating that you and your flunkies are the only sane people in the world. That's why, in part four of my memoir I talk about this very explicitly, I also think Scott is being pretty dishonest for political reasons; I think he's less self-aware about it than Yudkowsky, that's just my read; obviously I'm not a mind-reader, but that's my read of the situation—but I feel much less personally aggrieved at Scott, precisely because Scott is so humble. He writes a blog. A lot of people read the blog, but he's not expecting people to defer to him as an authority figure. He's just like, I write a blog, I'm just a guy who writes a blog. And Yudkowsky does not identify as, I'm just a guy who writes the blog, he's like, I am the only sane person on earth. The sequences were so good that I actually thought this claim was credible at the time. It really did seem credible at the time, and he just does not seem interested in living up to this these days, or even noticing. To this day, I am not sure to what extent he is being consciously dishonest versus has fallen into self-delusion but it's, it's ...

CM: Well, last thing on the consciously dishonest front, people talk about this, you linked to that blog post, it talks about, did they say "EA Has a Lying Problem", or what it the whole community has a lying problem?

ZMD: "EA Has a Lying Problem" was the post.

CM: So people talk about that. It's like everything is in service of this x-risk thing, and so lying becomes okay, or undermine your values in other ways becomes okay, because you're saving the world. How real is that?

ZMD: I think there are people who are falling into that failure mode, and there are also other people who are criticizing them for that. Both those things can happen at the same time, because again, we talk about the movement as an abstracted thing, but it's actually thousands of different people who happen to read the same stuff. Some of them are more honest than others, and some of them are less honest than others, and some of them are true believers, and so like, Sam Bankman-Fried, you know, as of 2021, everyone would agree, Sam Bankman-Fried, oh, he's an effective altruist, he's totally one of us. And then afterwards, we're like, wait, he shouldn't count.

CM: It's like anyone who missteps. Zizians? They're not part of it. Sam Bankman-Fried, Leverage, all that stuff, not part of it. I don't know, when you have enough evidence that there might be a problem with the extreme. Honestly, it's hugely helpful. I might come back to you with a couple things, where you were elected to talk, but if you write it down, if you wrote down what you meant about Anna Salamon's response.

ZMD: In general, I'm happy to talk, because I'm happy to talk to anyone, but in terms of maximizing accuracy for the book and not doing the thing you did to Kelsey Piper, I would prefer email, just because I can think carefully and write the exact paragraph that expresses exactly the thing, whereas like in real time I'm just like—

CM: I can do that. I can give you a list of facts.

ZMD: If you give me a huge list of facts, I can go line by line and be very, very precise. Okay. When is this coming out?

CM: We'll see. Sometimes it's hard to tell with the book publishing industry. I'll keep you updated. What's your impression of what Roose is doing with this community?

ZMD: I don't know the details. The reason I mentioned you to him is because you both work for The New York Times. I've read a few of his articles, I listened to a couple podcast episodes, and I happened to see him at Less Online. That's all I know. I don't actually know anything.

CM: Just curious. Fantastic. Thank you as always.

ZMD: Okay, thanks.

25 June–7 August 2025

From: Cade Metz
To: Zack M. Davis
Date: Wed, 25 Jun 2025 08:50:50 -0700
Subject: thanks and blog post

Hey Zack: Thanks again for chatting yesterday. Small question for now: do you have a link to that Ben Hoffman blog post where he talked about the price of a board seat at OpenAI?

From: Zack M. Davis
To: Cade Metz
Date: Wed, 25 Jun 2025 13:05:48 -0700
Subject: Re: thanks and blog post

The post is "An OpenAI Board Seat Is Surprisingly Expensive", but I misremembered what the post is saying. (The title doesn't really capture it.) Ben is hypothesizing that more than just the money, what OpenAI was getting out of the deal was validation from OpenPhil as a legitimate "AI safety" actor.

(This while none of the original AI safety people thought OpenAI was a good idea: see Ben in April 2017 on "OpenAI Makes Humanity Less Safe", Scott Alexander in December 2015 on "Should AI Be Open?" expressing the same concern but with Scott's characteristic timidity and waffling, Yudkowsky saying he "cried for 45 seconds or so" at the OpenAI launch, &c.)

⁂

I want to explain the Bayesian thing that's going on in this passage that I couldn't explain in real time that I don't expect New York Times readers to get (because it was new to me at the time):

When Anna said she was surprised by my SAT scores only being ninety-seventh percentile (as I had mentioned on our walk the previous afternoon), I said that I was surprised that she was surprised, and she said that I should expect her to have been surprised, given that I had said earlier that I thought she overestimated my abilities.

A background axiom here is that beliefs are for making predictions about your future experiences: what it means to "believe" the sky is blue is that you predict that if you look up outside and open your eyes you'll see blue. "Surprise", then, is when you have an experience that you didn't predict. Being surprised means your beliefs were probably wrong; if you have correct beliefs, you shouldn't be surprised by things.

Here, Anna thought I seemed impressive, so she was surprised that my standardized test scores were so low. (Her beliefs about my impressiveness made her predict that I would have scores to match, and that prediction didn't come true.)

I thought I wasn't impressive, so I didn't think she should have been surprised, but what Anna subsequently pointed out was that I hadn't thought through the implications of my own beliefs: my belief that Anna was overestimating me should have implied that her overestimation would make bad predictions about evidence about my abilities that didn't route through her own purportedly biased perceptions (e.g., test scores), and that those bad predictions would surprise her: thus, I should have expected her to be surprised.

⁂

I have some follow-up commentary on this moment from our conversation yesterday. The context was me saying that I wanted people to think critically about AI risk for themselves:

CM: So, ultimately, you still believe that there is this absolute truth.
ZMD: Yes.
CM: Do you get that some people really question you?
ZMD: I think they are confused about some things.

The reason I think they're confused is, well, what would it even mean for there to not be an absolute truth? It doesn't make sense to think that rationalists believe AI existential risk is real but normies don't, but there's no absolute truth of the matter, because eventually, you expect these beliefs to make different predictions. If we invent AGI and nothing really changes, then the rationalists were wrong; if we invent AGI, and it kills a billion people, then the normies were wrong.

We may not know what the truth is, but there's still going to be a truth (which you could call "absolute").

We have a slogan, "The map is not the territory." When I try to imagine how someone could possibly disagree with this, my guess is that they're confusing the map and the territory: thinking, well, there are all these different groups that believe different things, how can you say there's an absolute truth? But the different groups' beliefs are just different maps. There's still an underlying territory.

From: Cade Metz
To: Zack M. Davis
Date: Tue, 1 Jul 2025 08:34:08 -0700
Subject: Re: thanks and blog post

Zack: This is great stuff. Thanks. After doing some more reading, this might be the more interesting blog post:

https://benjaminrosshoffman.com/effective-altruism-is-self-recommending/

Would love for Ben to chat about this. Will at least try....

Cade

From: Zack M. Davis
To: Cade Metz
CC: Benjamin Hoffman
Date: Tue, 1 Jul 2025 19:25:55 -0700
Subject: Re: thanks and blog post

Cade, I think Ben would be more open to chatting if you could provide some sort of counterevidence to his perception of you as having "[a]bsolutely zero recognition of any underlying reality about which an accusation might be made, or communication attempted" such that he "didn't think it was worth [his] time to proceed."

Such counterevidence could take the form of, say, a written or verbal explanation in your own words of what observations you think might have made Ben think that about you, but why he's actually got you all wrong. (If it helps, ChatGPT o3 and Claude 4 Opus did a pretty serviceable job of the former.)

This is related to the section in my previous email about absolute truth. When you asked me if I still believed in an absolute truth, and if I understood that some people really question that, what did you mean by "absolute truth", such that it makes sense to you that some people question whether there is such a thing?

I'd expect that you do believe in an underlying reality when it comes to ordinary, "mundane" things: if there's an apple on the table in front of you, then it's true (perhaps, "absolutely" true) that the apple is there. You don't need to get an on-the-record quote from a professor of apples at Harvard to believe that the apple is there, because you can see it.

Basically, the "rationalists" are applying that attitude (which everyone has about the topic of whether there's an apple on the table) to everything, including much more complicated and abstract topics where you, as part of your job, would need to get a quote from a professor somewhere.

That convention exists for reasons, of course. There aren't enough column-inches for reporters to present a long chain of observations and inferences which busy New York Times readers wouldn't be able to evaluate for themselves anyway. It makes sense that the Times wants its reporters to stick to either uncontroversial checkable facts or opinions and theories from credentialed experts. That's the epistemology of journalism.

However, when you're writing a book about an loose collection of philosophy autodidacts with deep ties to technology barons that you think the public should know about, you need to understand that your subjects are not operating under journalist epistemology. Separately from the fact that it's not your job to write that way (because you need to play the facts-and-sourced-opinions game), if you don't understand this in your soul, you're not going to be able to accurately paraphrase the ideas of Ben Hoffman (or Kelsey Piper, as we've discussed, or really any of these people). That's a problem that you need to fix if you want to write a great book!!

From: Benjamin Hoffman
To: Zack M. Davis
CC: Cade Metz
Date: Fri, 4 Jul 2025 11:22:31 -0400
Subject: Re: thanks and blog post

I just got off the phone with Cade. Most of the conversation focused on the social graph, and the material facts related to it: who was socially connected to whom, at what time, in what place - the sort of stuff I already understood Cade to be tracking as a journalist. But the surprising bit was that he took initiative to keep circling back to the only important thing I said in the conversation - that the problem with OpenAI wasn't a matter of individual bad actors replacing good ones, but a systemic shift—a deterioration in accountability norms, or (worded differently) a vibe shift towards opportunism.

Cade, I didn't get the sense that you were satisfied with your understanding of what I was saying. At the same time, there wasn't enough back-and-forth for me to form a clear impression of what you did understand me to be saying. It seems as though you were inhibited from asking followup questions about anything conceptual. Each time you brought up the topic, you verbally claimed to understand what I was saying, but then you brought it up again - which implies real, unaddressed curiosity.

I wonder whether this confusing combination of curiosity and inhibition might have something to do with your understanding of journalistic ethics, which as far as I can tell you follow fastidiously. Specifically, I wonder whether something about the idea that it would be wrong to get involved with a story you're writing about, also makes it hard to take the sorts of actions that would lead to understanding the story.

Vice Magazine forms an interesting contrast; my impression is that the specific way in which their reporting has been distinctively interesting has to do with their tendency to try to be wicked, rather than good, by the standards of journalistic ethics, and specifically in terms of getting involved with the story. Unfortunately, just as reversed stupidity is not intelligence, journalistic vice doesn't fix all the problems with journalistic ethics; because they're moved by a desire to vice-signal, they only get involved with stories in transgressive and opportunistic ways that also prevent them from trying to understand.

If this doesn't match how you see it, I'd be interested in where you think I've misunderstood you.

From: Cade Metz
To: Benjamin Hoffman
CC: Zack M. Davis
Date: Wed, 23 Jul 2025 10:18:51 -0700
Subject: Re: thanks and blog post

Hey Ben (and Zack): So sorry for the delay in getting back to you. I was away on vacation for a couple of weeks.

I truly enjoyed our chat -- and it was wonderfully helpful as I put together my book.

I think what you sense from me is a little different than the way you are thinking about it. It is not about journalistic ethics.

Like you, I am interested in the big ideas -- like your big idea that I kept circling back around to. We share an interest in that. We both understand its importance. And I want to show others how important it is.

But in order to do that -- to reach people who have no familiarity with the world you live in -- I need even more information. I am *also* interested in "the social graph, and the material facts related to it: who was socially connected to whom, at what time, in what place." I am interested in that because I can use it to deliver the big ideas. Most people do not read like you do or think like you do. They want a story, a narrative. More importantly, they really understand the big idea when it is delivered through a narrative -- through people.

If I seem inhibited, it is only because I am trying to keep you talking. Everything you say is interesting to me, because you have had experiences I haven't had and you have thought hard about ideas I am interested in.

That's it.

Anyway, I am very much interested in the same ideas that you are. But I also want the little details, too. People love details!

Happy to discuss more. I loved our conversation.

Cade

From: Zack M. Davis
To: Cade Metz
Date: Thu, 7 Aug 2025 20:55:41 -0700
Subject: book review?

Dear Cade:

Can you share any part of the book draft with me for feedback? I know it's not against your publisher's policy, because Paul Sas says you shared drafts of the chapters on the Extropians.

I wrote down a lot of thoughts about the 4 August Lighthaven article and some potential issues with the fact-checking process for it, but I'm going to withhold that for now in favor of this shorter email. I remain,

Your faithful correspondent,
Zack M. Davis

From: Cade Metz
To: Zack M. Davis
Date: Thu, 7 Aug 2025 21:12:19 -0700
Subject: Re: book review?

Hello. Would love to hear your thoughts on the article. Happy to discuss.

I can't share the book chapters, for various reasons that I am also happy to discuss. But I will certainly fact check stuff that involves you, as promised.

Do let me know if you want to discuss on phone or in person.

Cade

12 August 2025

[The first 11 minutes of this conversation were published separately as "My Interview With Cade Metz on His Reporting About Lighthaven".]

(audio)

ZMD: One other question here, so the Lighthaven piece included a photo of Eliezer Yudkowsky wearing a golden fedora. Can you say anything about how that photo was chosen? Is that your decision, is it your editor's decision? Who chooses the photos?

CM: I mean, The New York Times chose the photo. We have lots and lots of people who work on stories. It's probably not right for me to talk about.

ZMD: So the reason I'm asking is that Yudkowsky reacted to the photo on Twitter assuming that it was your choice and if that's not true, then like ... I guess you can't comment on the editorial photo choice.

CM: I think that picture has appeared in The New York Times before.

ZMD: Do you want me to read what he said, or is it not interesting?

CM: Sure.

ZMD: He said, quote,

I expect Cade Metz is deliberately showing a photo which says "Nerrrd", so that his audience will feel it's safe to bully his targets about being culty cultists. A real cult leader might seem dangerous; Metz needs to also convey that he's lying.

End quote.

CM: He's said worse. He can say whatever he wants.

ZMD: So the reason the question seemed worth asking is because it would be interesting if it turns out that—so I don't know how it works inside at the Times, but if someone else chooses the photos, if it's someone else's job to choose the photos, then the thing he said about you would be false. But if you can't comment on that, whatever.

CM: I'll talk about myself. I don't want to talk about anybody else at the Times.

ZMD: I have a lot more notes. Those were the prepared questions I had.

CM: Well, one tiny question I had for you, I want to make sure I got this right. When you talk about that great moment where you discover the "Universal Fire" essay through that Jane Galt blog, had you dropped out of Santa Cruz at that point?

ZMD: No, no, that was before—that was like—so I had been—that was after my first year. I entered UC Santa Cruz fall of 2006. And so like spring quarter—they were in the quarter system—spring quarter 2007, I had some academic trouble, I got a D in vector calculus, but then so I was coming back in the fall, coming back to Santa Cruz in the fall, and reading Overcoming Bias.

CM: So when did you drop out?

ZMD: After that quarter, actually.

CM: So you did go back.

ZMD: Yeah, I did go back. I remember being at UC Santa Cruz in fall 2007 and reading Overcoming Bias in the computer labs, and then having a nervous breakdown at the end of the quarter.

CM: Got it. Alright, that's good. I wanted to make sure I got that right. What I really am interested in, what I think is what's crucial to people understand, is that this community has played a role in everything that we're seeing, and so that's what I'm aiming to do.

ZMD: Yeah, I definitely agree that there's a real story there.

CM: Exactly.

ZMD: There was a Wired piece that had that really funny part where Daniella Amodei said like, I don't know very much about EA, I think it's kind of an outdated term, and then the reporter points out that she's married to Holden Karnofsky. That kind of thing is kind of funny.

CM: It's not only funny, it's really interesting. A thing I think about a lot is that that kind of thing has been part of the community for years, right, where people wouldn't say they were EA, they would say they were EA-adjacent. Did you experience that? I mean, that's something that a lot of people—

ZMD: I definitely heard the phrase "EA adjacent" a lot.

CM: Exactly, you hear it a lot, people have said it to me about themselves, for years and years and years. It's sort of this interesting phenomenon where you have this community, which is obvious, it's documented on the internet, again, it's had a real effect in so many ways, but then so many people say they're not part of it, which a little makes sense, in that, you know, there aren't rationalists membership cards, EA membership cards, you know there's no official body, necessarily. It's this kind of organic community. Why do you think even in the past people wouldn't identify? Is there something in your generation or in that community that doesn't want to identify? Some people do.

ZMD: Until I had my conflict with them, I was quite happy to call myself a rationalist. I would not have called myself an EA, but I did give a lot of money to charity before I stopped.

CM: Why do you think that others were reluctant to associate themselves with the community?

ZMD: I'm not really sure. After the FTX thing, there was definitely brand damage from that.

CM: Oh, yeah, exactly. That's clear and that's obvious, but what's interesting to me is that even before that, people would do it. Some people have argued it's just a thing with your generation to not want to be part of the larger collective.

ZMD: I think that impulse is healthy. We want people—I want people to think for themselves.

CM: Some people in the community do right, and not everybody agrees, but there is still a community, it's a social thing, but it's also it's an agreement on these larger issues, meaning AI's on that trendline, it's going to be powerful, it could deliver the singularity, but it could also kill us, right, and there are these common ideas that people congregate around, that's a real thing. I am fascinated by this idea that even the people who are as close to the middle of it as you can get, the power center, they're like, I mean, that's fascinating.

ZMD: I suspect Daniela would have been more forthcoming if it weren't for the FTX thing, probably?

CM: I think you're right. Why do you feel like that changed it?

ZMD: I mean, it is kind of embarrassing, right? Before, you can always say, well, we didn't know, and that's true, we didn't know, but like ...

CM: The other thing I think about is that any ideology, any belief taken to an extreme, can be dangerous. There are many examples of this in the community: Leverage Research, SBF, the Zizians. Do you ever think about that? That there are multiple examples of this ideology being taken to an extreme.

ZMD: I don't spend a lot of time thinking about this. Ozy Brennan, who you quoted in the Lighthaven piece, actually has an article about that topic in Asterisk magazine, "Why Are There So Many Rationalist Cults?" I guess I don't spend a lot of time personally worrying about this because I feel like it's not a problem for me. Maybe some people would say that I have extreme beliefs, but I don't seem to be having problems.

CM: I guess there are two things, you could have a single person having extreme beliefs, or the group dynamic is part of this. That can be dangerous. If everybody is pushing everybody else to the extreme. But you don't think much about that.

ZMD: So there's also this problem of different groups slandering each other. I mentioned in the memoir that there were a couple years when I was collaborating with Michael Vassar and some of his friends on a lot of intellectual stuff. I still consider myself friends with Ben Hoffman and Jessica Taylor. I don't talk to Michael very much these days. We chatted briefly at a conference the other month. But there's some people who call us "the Vassarites" with cultish insinuations in a way that just sort of seems like slander, because from the inside, like, what? We're different people who talk to each other sometimes, what is the alleged problem here, exactly?

CM: It's a great point. It's a tricky thing to describe.

ZMD: So that's why—you've told me that the book will be able to capture more nuance than the New York Times articles. I hope you're telling the truth about that, because the conversations that we have here are actually good. In these conversations here, you seem pretty perceptive. I want to trust you.

CM: Good to hear.

ZMD: Well, no, and then I listen to the audio commentary on the New York Times piece, and I'm like, ugh, really? Really?

CM: Think of it this way. You're not the only person I'm talking to.

ZMD: Well, it's true, but like—

CM: You have to take these ideas and really help people understand it. It's something very different than they're used to. I definitely see what you're saying, but a lot of these ideas are in this book, like the thing we just talked about, that this there is this phenomenon where people have a problem with being associated with the group, and that's real, but the group is also real. The collective has power, even if no one says they're part it, because they are working together.

ZMD: Ideological coordination.

CM: Sure. Great term. I love that; I'm going to quote you on that.

ZMD: I think I got that from, there was an article in Palladium magazine that used that term, and it stuck with me.

CM: It's really a great term, and what's so interesting is that having covered this field for a long time, having kept an eye on this community and talked to people in and around it, people with both feet in it, people with one foot in it, people on the outside, I can recognize what someone is going to do based on their proximity to that ideology.

ZMD: There's also this thing where I kind of think the coordination is bad and wish people would like—so again, the Sequences were great. I really think the philosophy is really really great. In principle, the philosophy of EA seems pretty good, too: help people, help people efficiently. But there's this coordin—so specifically, your Lighthaven article came out, I mentioned on Twitter somewhere that I've been talking to you, and someone who had been supportive of my gender and philosophy of language war, actually, was critical of me on the grounds that—so the final sequence in Yudkowsky's Sequences had been on "The Craft and the Community", and one of the posts was "Why Our Kind Can't Cooperate" and was pointing out that our kind of personality type, of libertarian atheist techno nerd people, tend to be hard to cohere into everyone supporting a group effort, as opposed to conventionally—as opposed to religious people who can form a collective, form a powerful collective, and Yudkowsky is arguing that we the nerds should also be able to harness the technology of coordination.

CM: That's really interesting. What's it called?

ZMD: "Why Our Kind Can't Cooperate."

CM: So it was on Overcoming Bias, one of the last of the sequences.

ZMD: It was the last sequence, it was the sequence that appeared on Less Wrong instead of Overcoming Bias, so in early 2009. I mentioned I had been talking to you and someone who had been broadly supportive of me and my work criticized it as, the fact that we can't even coordinate to not talk to the guy who doxed Scott Alexander, is why our kind can't cooperate, it's epitomizing that tendency that Eliezer was complaining about.

CM: That's true. But they do a pretty good job of coordinating on that one.

ZMD: That was another thing I was wondering. I had asked this question, I don't think you gave me an answer, maybe you don't want to give me an answer, but has reporting on this topic been noticeably harder than other reporting that you've done, because of that angle?

CM: Reporting is hard. If you're covering Google—

ZMD: Googlers don't want to talk, either.

CM: They don't want to talk. It's just reporting. It's just reporting. And people [dropping to a whisper] always talk. And good for them. Like I said, I admire your philosophy, like information—

ZMD: But so, that's just that's kind of why I wanted to—because again, I mentioned I did not like the religion angle in this piece, and so I was just worrying that, oh man, I really, really—I was worrying if the people criticizing me might have a point. I think if the book turns—I guess this is why I was curious, why I sent that email asking about if you could share drafts, because if you're not telling the truth about the book being more nuanced, then I think the people criticizing me probably had a point.

CM: Well, of course it's more nuanced. It talks about all these ideas. I just told you about that. But think about it like this. You say that I'm the guy that doxed Scott Alexander. I still—did I? The guy had just revealed his own name.

ZMD: I explained this game theory thing to you on April 22nd; I guess you didn't get it, but it's really important.

CM: I get what you're saying. Some people just don't, wouldn't agree with that.

ZMD: Well, no, but I think this is pretty straightforward. He had a job as a psychiatrist.

CM: I understand why he did it, okay. I also understand why The New York Times and I did what we did. It was a clash of ideology. It went down the way it went down. It was astounding when it happened. And then, you know, I will walk down the street, and people will say that's the guy who doxed Scott Alexander, and I think, did I? Did I dox him?

ZMD: I mean, I'm not attached to that particular choice of word. The issue is, he wanted his political writing to stay under the pseudonym, and he didn't want that visibly tied to his identity as a psychiatrist, even though, as you have pointed out, he did not have very good opsec. The information was out there, but he was depending on a security-by-obscurity strategy, where the information was discoverable, but he didn't want it in The New York Times. It's true that he published the name first, but the reason he did that is because he knew that you were going to put it in The New York Times, and so he wanted it to happen on his terms.

CM: I completely get it, but again, if you look at the group, as opposed to the individual, the group very often decides certain things, and operates in lockstep. The group has decided that it shouldn't talk to me, because I did this horrible thing. It's a thing that, if the average person looks at it, they don't understand it. That's the power of these ideologies we talked about.

ZMD That's also in a lot of ways the power of The New York Times, and why people are so afraid of you, is because they're afraid—so you are correctly pointing out that there's this cabal of Scott Alexander and his friends who have the power to like determine what these thousands of people believe. But the reason those people are afraid of you, is because you have the power to decide what a much larger group of people believes. There are lots of people who read The New York Times and just believe what it says. That's why people are so scared when you're pushing this religion angle. That's why I'm having these second thoughts, like, man, I really want to just talk to everyone, and now—

CM: I know, but you know, and so many people in your community, and your friends, use all the same terms, and it's okay when you do it, but somehow it's not okay if The New York Times points out that there are these characteristics. We're being careful with that language.

ZMD: I think I would object less—in fact I might not object at all—if you had said ideology instead of religion.

CM: I hear what you're saying, but I quote a Franciscan nun and scholar about what religion is. People use these terms all the time.

ZMD: This was actually in my prepared questions, and I skipped over it because the conversation didn't zag that way. Sorry, one moment. You'd included a quote from our April 22nd conversation, or in an earlier revision of the article, you'd included this, quote, "Psychologically and sociologically, it seems very comparable to a religion, even though I would say that a lot of the beliefs are true," end quote. I still stand by that quote, because the thing I said was appropriately qualified. As I said in the interview, I was referring to the sociological niche of a social group that believes things that has a canonical text of why the group believes those things. I was drawing a parallel to religion, while making it clear that the beliefs themselves can be evaluated on the empirical merits, rather than being dismissed as the group's self-serving myth. Why did you cut that quote?

CM: Because you asked me to. You didn't want to be included.

ZMD: Sorry, point of clarification: from my perspective, that wasn't what I was asking. I think you made a probabilistic inference that would be correct about most people, but was actually not what I was requesting, because I have this weird ideology. We had that phone call where I was annoyed that you were taking this religious angle—

CM: I should clarify. I didn't take it out because of anything you said, it just came out. Lots of things go in and out. No, you're right. It didn't have anything to do with that.

ZMD: But to explain the weird ideological thing, I was complaining, I was expressing annoyance that you were taking this religious angle, but I still stand by my quote. I think it's fair game for you to use that quote, because I know the rules of the game.

CM: It was on the record, of course it's fair game.

ZMD: Yeah, exactly.

CM: It's a great quote, and this is an interesting thing that I explore as well. A lot of people see this where they see the belief in the singularity coming as something very different than religion because it's science based, and that if you follow those trendlines, you get there. Again, there's still faith there. You don't know when, you don't know. We're not sure that's definitely going to happen, and so there is this overlap. That's why I like your quote. I don't remember why it came out, but it came out.

ZMD: From my perspective, there's this really unfortunate thing, where the underlying tendencies in human psychology that make religions be like a memetically stable thing, like we are human, we necessarily think in the kind of way that results in religions being a thing. The problem is when you encounter this belief that, oh, wow, artificial intelligence or superintelligence will remake the entire world, it definitely pattern-matches to these grand visions, and a lot of people do go crazy over that. At the same time, when I think about what I know about what I think people have learned from science, and taking what humanity has learned from science seriously, it still seems pretty credible, just because—

CM: People have agreed with you since the earliest days of the Extropians, this is what they talk about. They say we don't believe in organized religion and this idea of heaven and hell; we're going to create our own afterlife here, with science and technology. It's a replacement.

ZMD: There's a good post—I'm wondering if I should be embarrassed to be citing the Sequences at this point.

CM: Of course not.

ZMD: Well, no, because you're going to pattern-match it to, oh, it's like citing the Bible, no, but I think it's a good point, though. Yudkowsky makes this great point about taking history seriously and realizing that you think of the world that you grew up in as normal, and when you hear about ideas of radical change from that, you think, wow, that seems like a weird fantasy fairy tale. But the things we think of as normal have not always been here. If you go back, you know, 10, 20, 100, 1000 years, there were not always computers, there were not always cars, there was not always abundant food, there was not always literacy; if you go back sufficiently further enough, there were not always humans. If you zoom out to this thinking about how the universe evolves, there have been radical changes. The idea that the thing that we have now, where you have a bunch of humans that have intelligence and have language and have technology, but things still look normal, and we're still very limited. When I zoom out and just look at the entire history of the universe, the thing that we have now does not look stable.

CM: I understand. I understand.

ZMD: Okay.

CM: It's a fascinating thing where people all over the past 40 years have talked about this, where they're discarding traditional religion. A lot of people read Richard Dawkins, The God Delusion, he's a big part of that generation that they think about, and this often a replacement for that, but you're right, it's different, and people think about it different.

ZMD: I don't think it is a replacement. The tragedy of being human is that it's easy to construe as a replacement insofar as the ideas hit similar buttons in people's brains.

CM: At the same time, even in your diary, you will talk about it.

ZMD: To be clear, I wrote that 20 years ago. I would not write that today.

CM: It's not just you; all sorts of people like, they think about it in these terms. But you know, your points have been taken. The other thing I've been thinking about, this idea that this community who is so concerned about the dangers, has in some ways brought those about. Like people about the idea of hyperstition.

ZMD: As I've been saying, a lot of people are sad about this.

CM: Tell me what's their thinking, what's your thinking?

ZMD: It's just kind of hard to avoid, right? For a lot of years, Yudkowsky had this vision of, well, we're going to make it Friendly. Given a certain view of what is technologically possible, there's this pretty reasonable presumption that if something is possible, sooner or later someone is going to do it, and avoiding that would be pretty hard. And in fact Yudkowsky and Soares have this thing, coordinating to not do this is pretty hard, but it's humanity's only option at this point.

CM: I guess there's it being possible, and your being pretty sure it's gonna happen, and you're doing it quicker. This community for a lot of reasons, helped create DeepMind, helped create OpenAI, helped create Anthropic, and these ideas have pushed it forward. It's not just about doing it; it's about doing it faster.

ZMD: So the whole, I mean, I do [exhales] Man. I have a belief here. There's something I want to say here, that if I were just having an ordinary conversation with someone, I would just absolutely say it, but after listening to the audio commentary on that Lighthaven piece, I feel like, hmm, do I really want to say this part to Cade Metz?

CM: I'd love to hear it.

ZMD: I might email you. I'll think about it.

CM: Sounds good. Think it over.

ZMD: But you see the tragedy, though, right, of that I would rather just live in a world where just people just say everything, but there's this fear that ... different parties with different interests have differently sized megaphones. In principle, the thing that we're doing is actually symmetrical in that we are on the record in both directions. I already published one interview with you. I'm planning to publish more of our conversations. In principle, this is perfectly symmetrical, in practice there's the issue that, how many people read your stuff versus how many people read my stuff. You have more power to determine what happens in Society than me.

CM: There are a lot of other powerful forces at work and with all this stuff, but I hear you. I hear you. Is that all the questions I had? Oh, how did you feel when the dying with dignity post went up? Did that surprise you what was the reaction of you and others to that?

ZMD: I wasn't that surprised. It wasn't very surprising to people who had been paying attention, because he had been quietly saying similar things for a few years, but it was definitely a somber moment.

CM: Tell me about that, why was it so somber?

ZMD: Well, this person who a lot—I again, I have had my differences with Yudkowsky and reasons to distrust him—this person who we had learned a lot from, and whom a lot of us trusted quite a lot, was saying like, well, everyone's going to die. It's a pretty somber moment. It wasn't unexpected in that, that been a quiet vibe for a few years, and then April 1st was just a time to make it text instead of subtext. So I talked to Yudkowsky at an Independence Day party in 2021, and I don't quite remember the details, but like there was just this moment in conversation, where he referred—you know, unfortunately, I didn't actually write a Diary entry immediately after the party, so I don't actually remember the details—there was a moment where he alluded to everyone's impending death, and I said something about how it hadn't really felt as real to me before DALL-E. So in January 2021, I think it was January 5th or January 6th, 2021, OpenAI published DALL-E, which was the text-to-image thing, the one that—the avocado chairs. I like to say that that was, there's a line in my forthcoming memoir part where I say that that was the most significant thing that happened that week of January 2021.

CM: Got it, got it.

ZMD: So for me, DALL-E was this moment like, wow, this AI stuff that we've been talking about for 15 years might actually—or I guess it only been 12 years at that point—might actually be reasonably soon, rather than "someday".

Because specifically why I was so impressed by DALL-E was just the fact that—it's easy enough to have an intuition that if you have a million, if you have a thousand images of chairs, you can train a neural network on that, and can spit out more chairs from the same chair distribution, but the fact that it can compose concepts and do avocado chairs, without there being a thousand avocado chairs in the dataset, that seemed to me like this glimmer of generality, and composing concepts instead of just being a clever statistics trick. That was the moment that I was like, okay, this might be sooner than I expected. I think Yudkowsky probably had this moment—just sort of reading between the lines or maybe not even needing to read between the lines too much of things he had said, I think for him this moment was AlphaGo in 2016. I guess it took me another five years to see the thing that he was seeing then.

CM: A lot of people talk about, after that post went up, it was a somber moment, some people quit going to dentists, said they weren't going to have children. Did you experience much of that, like changing behavior?

ZMD: I'm not an extreme version, but I have been doing a little bit of this insofar as, again, I think I'm being reasonable here, in that, so for example, okay, I probably don't want to talk about my personal finances in detail on the record, but for example, up through 2022, I was working as a software engineer, and right now I do not have a dayjob, and part of the reason I feel comfortable living on savings right now is because there's this uncertainty of, well, given a substantial probability the world is going to get really crazy in the next 5 or 10 or 15 years, maybe I want to on the margin shift my behavior more towards, in the moment personal consumption, like not having a day job is great, I get to do what I want, and trying to do things that help prepare the world for things getting crazy as the AI stuff gets crazy. And now of course I could be wrong about this. So let's see, it is 2025. The money I have from my software engineering career, it's not going to last forever. I have enough to live comfortably for quite a while, especially because you know, I don't have children or anything. I feel a little bit guilty about this. You also read about South Koreans are going extinct, and so conditional on no AI, I think it would be really important for people like me to have children. I just never—I have not been romantically successful in life.

CM: Do you feel like people around you change their behavior after the die with dignity?

ZMD: I'm not sure. I think there are a lot of people who are doing a similar thing that I'm doing of on the margin shifting more towards consumption, and less on 10 year, 20 year, 30 year plans, but trying, obviously, I think people are painfully aware of the analogy to doomsday cults, and trying not—sorry, did I hear you correctly that you said something about, stop going to the dentist?

CM: Some people told me that.

ZMD: That's crazy. I think that's crazy.

CM: The other side of this is, certain people in the community want this to come, right? This has always been the thing, the singularity, right?

ZMD: No ... who? The e/acc people might say that.

CM: We should tackle that. Do you view them as part of the community?

ZMD: No. The thing is, it's almost not really a real thing. It's like a Twitter fad.

CM: Yes, that had a real impact, attracted Marc Andreessen and Gary Tan. I would go to AI salons in the valley in San Francisco, some people would identify as EA, some people would identify as e/acc.

ZMD: I think it's unfortunate, because I think part of what motivates the Twitter fad is people are annoyed at the corrupt rationalist/EA borg. I hope I've made it clear that I share some of those annoyances. I think there are good reasons to be annoyed. I think there are more productive ways to voice that criticism than just slapping a negative sign and saying, no, we're doing the opposite thing.

CM: I got you. It was definitely a reaction. If you want to email me that thought, let me know.

ZMD: I think I had one more thing I didn't cover, sorry, I'm trying to remember what it was. Give me a moment ... What is the reason you can't share drafts? I'm worried about the failure mode where—so like, when I write stuff, I like to get pre-publication feedback, because people can catch errors, and not just fact-check errors, but subtle things. A subtle thing from the Lighthaven article that didn't rise to the level of failing the fact-check, but I thought was a little bit misleading, was the way you described MATS as a gathering, and said "Like Less Wrong, MATS has been an entry point into AI companies"—this does not fail fact check—

CM: It's true.

ZMD: There's a subtle nuance that I'm worried that readers are not going to get, which is that MATS is specifically an application-only research program, where you apply, you fill out a form, you submit your school grades; it's the kind of work people are doing in academia, and when you just call it a "gathering", I worry that it's not sufficiently clear to readers that that thing is a very different thing from just showing up to Secular Solstice.

CM: Have you been to MATS? I've talked to a lot of people over there, it's different than an academic conference.

ZMD: Okay, sorry, that's true, but they are producing machine learning research papers, is what I meant.

CM: Absolutely. Look at the end of that piece. Sonia, who I really like, she really thinks about this stuff well, because there, she's a serious researcher, super, super smart, she writes those research papers, but she also says there's this sort of mythic quality to everything that drives her and so many others forward. I didn't go into a lot of the other stuff that goes on there. But I hear what you're saying. [...]

14 August 2025

From: Cade Metz
To: Zack M. Davis
Date: Thu, 14 Aug 2025 10:02:30 -0700
Subject: terminology question

You and other [sic] have talked about Scott Alexander's response to be as decision theory. But is game theory a better term? How do you (and the rest of the community) think about the distinction between the two?

Cade

From: Zack M. Davis
To: Cade Metz
Date: Thu, 14 Aug 2025 13:00:54 -0700
Subject: Re: terminology question

I actually said "game theory", not "decision theory." And maybe that choice of phrase is just me over-intellectualizing and using an academic term in a context where it doesn't actually clarify anything.

The point was that I don't think it makes sense to say, "Cade Metz didn't dox Scott Alexander Siskind; Siskind published the name himself on Astral Codex Ten before Metz's February 2021 NYT article."

The reason I don't think that makes sense is because Scott only published his last name after "gradually rework[ing] [his] life to be compatible with the sort of publicity that circumstances seem[ed] to be forcing on [him]" because "maintaining anonymity [was] a losing battle". If you hadn't been exploring a story about the rationalists in June 2020, or if you had told Scott that you'd respect his pseudonymity, those circumstances wouldn't have happened: he would have stayed with his employer and stayed on slatestarcodex.com.

The reason I wanted to summarize that as it being "game-theoretically your (Metz's) fault" (I don't know if that phrasing made sense to you) is because game theory is the academic term for the study of how people's decisions depend on the decisions of other people who have different interests. (Thomas Schelling described it as the "theory of interdependent decision" in his seminal work Strategy of Conflict.) The definition is abstract, but the kind of thinking that Schelling was writing about, about making and responding to threats and promises, recursively computing "my best move, given his best move, given my best move, given ...", is ubiquitous in human life, and even in the animal world. Countries prepare for nuclear war, not because anyone wants a nuclear war—it would be better for everyone if no one had nukes—but because it's even worse for you if you don't have nukes, but your rival nation does. Gazelles leap in the air, not "for fun", but to credibly signal to predators that they're too fast to bother chasing.

The conflict between you and Scott Alexander had a similar flavor. You were credibly threatening to publish his name, which "changed the game" such that his best response ended up involving up publishing the name first, but that's not at all the same thing as him just spontaneously deciding to drop pseudonymity. It was definitely your (or The New York Times's) fault.

"Decision theory" is a related but more general term. I would venture that a lot of the distinction is a contingent accident of intellectual history. (Academics studying Newcomb's problem called themselves "decision theorists"; academics studying the Prisoner's dilemma called themselves "game theorists", but they're intrinsically deeply related subjects.)

In recent years, Yudkowsky has apparently taken to telling journalists that he's a "decision theorist" rather than an "artificial intelligence researcher", presumably because his AI thinking never produced academically-legible achievements (because he was trying to come up with a theory of ultimate AGI "Friendliness", not implement something immediately useful the way people like Hinton, Sutskever, &c. have done), whereas functional (formerly "timeless") decision theory is a more academically credible concrete contribution.

Not sure how you can maintain journalistic distance about this part in the book (the whole SSC-vs.NYT affair is presumably part of the story you're trying to tell, but in this case, you are an interested character in the story). Maybe switch to an autobiographical "gonzo" style for that chapter?!

Discuss

Dynomight has looked at the health effects of vitamin D supplementation. The large-scale meta-analyses that have been performed conclude there is no significant effect, even though individual studies relatively consistently point in the direction of an effect. This means our failure to detect a significant effect may be due to low experiment power.

Dynomight suggests a mechanism for that: what if the low power of the existing randomised trials is because they tend to be run in countries that fortify common foods with vitamin D? That would mean, practically speaking, all arms of the trial get the treatment. To investigate, Dynomight pulls out a table with the results only from studies where participants entered with relatively low values of vitamin D in their blood.

Trial

All-cause mortality

Trivedi

0.90 (0.77 to 1.07)

whi

0.92 (0.83 to 1.01)

Lyons

0.99 (0.93 to 1.05)

record

0.93 (0.85 to 1.02)

The all-cause mortality is reported in odds ratios, which is a multiplicative scale. If the number is less than one, it means vitamin D supplementation was found to reduce all-cause mortality. But the 95 % confidence intervals in parentheses all straddle 1, meaning none of the studies were able to show a significant effect at that confidence level.

Unfortunately, no formal meta-analysis has been done on this specific subset of studies. But we can make a quick and dirty one!

Sign test (counting coinflips)

We can tell immediately from the table that four out of four studies have a number less than one, i.e. they show a beneficial effect of vitamin D supplementation.

This is a primitive form of meta-analysis! We count the total number of studies, and how many of them support the hypothesis. If we assume no effect, then half the trials should show a benefit, and half should show harm. What are the chances of flipping a coin four times and getting the same result on all four of them? 12.5 %.

Thus, the pooled p-value of the combined trials, when we look only at the direction of their result, is 12.5 %. Not significant. But keep this technique in your back pocket! It's so easy you can pretty much do it in your head.

Fisher’s method (the chi-square hack)

If we use more information from the trials, we get a higher-powered meta-analysis. Bringing back the confidence interval endpoints, we can compute the standard error that must have been used to arrive at the confidence intervals. We get this by taking half the interval width and dividing it by 1.96.

Trial

lower bound

upper bound

SE

Trivedi

0.77

1.07

0.077

whi

0.83

1.01

0.046

Lyons

0.93

1.05

0.031

record

0.85

1.02

0.043

Armed with this confidence interval[1], we can compute the z-score of the result as the size of the measured effect relative to the noise of the measurement, i.e. divided by the standard error.

Trial

Mort.

SE

z-score

Trivedi

0.90

0.077

−1.26

whi

0.92

0.046

−1.67

Lyons

0.99

0.031

−0.32

record

0.93

0.043

−1.56

We can then convert this z-score to a p-value by assuming it’s normally distributed. This p-value was probably reported in the original papers, but it didn’t carry over to Dynomight’s table, so we’re recomputing it. The p-value is the value of the standard normal distribution at the z-score. Google sheets has it as the norm.s.dist function.[2]

Trial

Mort.

SE

z-score

p-value

Trivedi

0.90

0.077

−1.26

0.21

whi

0.92

0.046

−1.67

0.10

Lyons

0.99

0.031

−0.32

0.75

record

0.93

0.043

−1.56

0.12

Here comes the trick for computing the aggregate significance of these four studies. We can convert this p-value to a chi-squared value, by taking its logarithm and multiplying by −2.

Why? I don’t know! Fisher said we could do that.

Trial

Mort.

p-value

χ²

Trivedi

0.90

0.21

3.13

whi

0.92

0.10

4.69

Lyons

0.99

0.75

0.59

record

0.93

0.12

4.26

These chi-squared values are all with two degrees of freedom, and they can be added together. The combined effect of all four trials has a chi-squared value of 12.7 with 8 degrees of freedom. We either look that up in a chi-square table, or run it through the Google Sheets function chisq.dist.rt, and we will find it corresponds to a p-value of 12 %. In this case, that happened to be very close to the result of the sign test, but ever so slightly more powerful.

I like this method because it’s relatively easy to remember the procedure, so I can whip it up in a spreadsheet live. Unfortunately, it’s still not powerful enough to reveal a significant combined impression from these four trials.

Precision-weighted pooled intervals

We can perform an even more powerful type of meta-analysis. First we need the standard errors of each study. If we only have the p-values (and effect sizes), we can convert them to standard errors, but the process relies on quite significant assumptions. In our case, we extracted the standard errors from confidence intervals, which is slightly better but still not ideal. Either way, we have standard errors.

The idea is that we’ll compute a weighted average of the effect sizes, with the weights coming from the precision of each study. The precision is the inverse variance, i.e. one divided by the square of the standard error.

Trial

Mort.

SE

weight

Trivedi

0.90

0.077

170

whi

0.92

0.046

470

Lyons

0.99

0.031

1100

record

0.93

0.043

530

This assigns the highest weight to the Lyons study, because that one seems to have pinned down the effect most precisely.[3]

When we perform the weighted average of the observed effects using these weights, we arrive at a combined odds ratio of 0.95. Since this is a linear combination of uncertain values, and we know their variation, we can compute the variation of the weighted average. We find out the aggregate standard error is 0.021.

That means the z-score of the aggregate effect is −2.17, which, under a two-tailed test, corresponds to a p-value of 3 %. Look at that! When we use a powerful enough test, the combined impression of these four studies is significant. If we believe Dynomight didn’t cherry-pick these four studies, that would be a meaningful discovery.

Now that we have the standard error of the aggregate, we can also compute a confidence interval for the aggregate.[4]

Trial

All-cause mortality

95 % CI

p-value

Trivedi

0.90

0.77 to 1.07

0.21

whi

0.92

0.83 to 1.01

0.10

Lyons

0.99

0.93 to 1.05

0.75

record

0.93

0.85 to 1.02

0.12

Aggregate

0.96

0.83 to 0.99

0.04

If this table does not excite you, then surely this fancy ASCII diagram will.[5]

Odds ratio: 0.8 0.9 1.0 1.1
- - - - - - - - - - - - - - - - - - - - - │ - - - - - - - - -
Trivedi ─────────────●─────────┼─────── (p=0.21)
WHI ─────────●───────┼─ (p=0.10)
Lyons ──────●┼───── (p=0.75)
RECORD ────────●──────┼── (p=0.12)
- - - - - - - - - - - - - - - - - - - - - │ - - - - - - - - -
Aggregate ────●────┤ (p=0.04)

I find this whole thing incredible. We can tell the aggregate is a sort of average weighted toward the more precise results, but the information contained in all of these four studies is still enough to draw a definitive conclusion that (a) yes, there is an effect, and (b) it is in the direction of benefit.

1. ^

   Hang on – aren’t odds ratios multiplicative, so we should log-transform the data before we do maths on it? Yeah, we should. Will it make a huge difference for a quick-and-dirty significance check? There are more important things to care about.

2. ^

   And here we are taking twice that, because we are doing a two-tailed test. This means we’re trying to see if there is any significant effect at all – benefit or harm. We perform the test as if we hadn’t seen any potential effect is likely a benefit.

3. ^

   There’s a mathematical argument for why the weight should be the inverse variance, but I seem to have lost my notes on the way here.

4. ^

   Again, shouldn’t these intervals be asymmetric due to the multiplicative nature of odds ratios? They should, but apparently the data source for these intervals ignored that, so I will too.

5. ^

   Yeah, yeah, it uses box drawing characters from Unicode.

Discuss

If you believe that LLMs lend themselves unusually well to alignment compared to other regimes, this can be a very good reason to start doing capability research on them rather than LLM safety research. Imagine you have these beliefs about how AI goes:

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mtable { display: inline-block; text-align: center; vertical-align: .25em; position: relative; box-sizing: border-box; border-spacing: 0; border-collapse: collapse; } mjx-mstyle[size="s"] mjx-mtable { vertical-align: .354em; } mjx-labels { position: absolute; left: 0; top: 0; } mjx-table { display: inline-block; vertical-align: -.5ex; box-sizing: border-box; } mjx-table > mjx-itable { vertical-align: middle; text-align: left; box-sizing: border-box; } mjx-labels > mjx-itable { position: absolute; top: 0; } mjx-mtable[justify="left"] { text-align: left; } mjx-mtable[justify="right"] { text-align: right; } mjx-mtable[justify="left"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="left"][side="right"] { padding-left: 0 ! important; } mjx-mtable[justify="right"][side="left"] { padding-right: 0 ! important; } mjx-mtable[justify="right"][side="right"] { padding-left: 0 ! important; } mjx-mtable[align] { vertical-align: baseline; } mjx-mtable[align="top"] > mjx-table { vertical-align: top; } mjx-mtable[align="bottom"] > mjx-table { vertical-align: bottom; } mjx-mtable[side="right"] mjx-labels { min-width: 100%; } mjx-mtr { display: table-row; text-align: left; } mjx-mtr[rowalign="top"] > mjx-mtd { vertical-align: top; } mjx-mtr[rowalign="center"] > mjx-mtd { vertical-align: middle; } mjx-mtr[rowalign="bottom"] > mjx-mtd { vertical-align: bottom; } mjx-mtr[rowalign="baseline"] > mjx-mtd { vertical-align: baseline; } mjx-mtr[rowalign="axis"] > mjx-mtd { vertical-align: .25em; } mjx-mtd { display: table-cell; text-align: center; padding: .215em .4em; } mjx-mtd:first-child { padding-left: 0; } mjx-mtd:last-child { padding-right: 0; } mjx-mtable > * > mjx-itable > *:first-child > mjx-mtd { padding-top: 0; } mjx-mtable > * > mjx-itable > *:last-child > mjx-mtd { padding-bottom: 0; } mjx-tstrut { display: inline-block; height: 1em; vertical-align: -.25em; } mjx-labels[align="left"] > mjx-mtr > mjx-mtd { text-align: left; } mjx-labels[align="right"] > mjx-mtr > mjx-mtd { text-align: right; } mjx-mtd[extra] { padding: 0; } mjx-mtd[rowalign="top"] { vertical-align: top; } mjx-mtd[rowalign="center"] { vertical-align: middle; } mjx-mtd[rowalign="bottom"] { vertical-align: bottom; } mjx-mtd[rowalign="baseline"] { vertical-align: baseline; } mjx-mtd[rowalign="axis"] { vertical-align: .25em; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c44::before { padding: 0.683em 0.764em 0 0; content: "D"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c2223::before { padding: 0.75em 0.278em 0.249em 0; content: "\2223"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c41::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c49::before { padding: 0.683em 0.361em 0 0; content: "I"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c2D::before { padding: 0.252em 0.333em 0 0; content: "-"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c77::before { padding: 0.431em 0.722em 0.011em 0; content: "w"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6B::before { padding: 0.694em 0.528em 0 0; content: "k"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; }

By I mean the probability that the first ASI is LLM-based (and that it isn't) - the two are mutually exclusive and sum to 100%.

Let's imagine you are a super genius, and your effort alone makes something 10% more/less likely than currently. Then

This is 1% less doom than doing nothing, congrats! Now for frontier capability work on LLMs - since these probabilities are about which regime reaches ASI first, pushing up also pulls down.

Woah, almost an additional 3% down! You could also instead go the Steven Byrnes route:

An additional percentage down!

These numbers shouldn't be taken seriously - the '10% more/less likely than currently' assumption in particular is arbitrary. Different problems aren't equally movable: making LLM ASI happen when it otherwise wouldn't could be far harder than the other shifts (esp. since many people are already trying), or making non-LLM ASI safe might be so hard that any effort is wasted. My main point is really just that working on capabilities can be a perfectly rational move for a safety-oriented agent, as long as he believes this regime is genuinely the safer one.

There's an interactive version of this - set your own numbers and it ranks the options

Discuss

Introduction

I’d like to share information that is worth sharing. So I ask the question—what is valuable information?

With the ubiquity of search engines and Wikipedia, much factual information can be found very quickly. However, in this era it is easy, even encouraged, to create “fake” information, including text, audio, video, and any other type of media that can be consumed by the use of computational devices. It is increasingly important that we can routinely discern whether the information we are seeing or writing is of any value.

To this end, I put forward a specific theory. It may be wrong or incomplete, but I find it is usually worthwhile to specifically test my ideas, by trying something that I at least do not know is wrong.

Valuable information could be any of:

1. true,
2. valuable, regardless of its truth,
3. both true and valuable,
4. useful, regardless of value and truth,
5. useful and true,
6. useful, true and valuable,
7. valuable, regardless of both truth and usefulness.

We will start with as a background prop the paper Objects of Consciousness, which says that evolution tunes perceptions toward fitness, rather than truth. Fitness being better expressed as a complete absence of unfitness, ie., smoothness of niche, round-peg round-hole and square-peg square-hole. Fitness is not universal, but a relation of an object or person and its environment together.

We have good reason to believe that the process of evolution does indeed solely tune perception for fitness, because it does so with everything else. Then for our purposes, if you want fitting and valuable and true and useful perceptions, there is only one possibility:

You must apply the Principle of Multiple Explanations, ex Epicurus, c. 300 BC, and strictly consider all and only the hypotheses not yet known to be incompatible with evidence, discarding with abandon those that do not, and then you may apply importance sampling to choose the order in which you consider their truth, value, or usefulness (or any other property or combination of properties). The point is that you need to start from a point which is at least not known to be unfit.

The maths on that tells us that we still must eventually consider all of the hypotheses that might be true in some form, so if we do not wish to do an infinite amount of work, we should choose the order in which we consider hypotheses carefully.

This way we can gradually improve our perception of truth by chipping away untruths, one by one, and reconsidering whether we should check something else first, before we proceed with expanding our previous pet theory.

What do I mean by “valuable?”

When we describe information as “valuable,” what are we really saying?

Whether a piece of information is valuable is relative to the person observing it. The same piece of information could be utterly irrelevant to one person, but absolutely vital to another.

So “value” isn’t just a property of information, but a property of information and people together; what we are saying when we say information is valuable is that it helps at least one person.

If we count the number of people for which this is true, and can quantify the positive impact that information has on those people, we can then quantify the value of information. As we will see later in the piece, the way in which information helps people is that it helps them to both be more present in reality and to better create it.

As a first draft of such a quantification, we could count the number of experiences people have with the information compared to without it, and count the number of meaningful choices people make with the information compared to without it, both over a specific period of time.

As a person does not control the experiences they have except by their meaningful choices, then we can decompose this into two quantities: lifespan, the length of time a person is alive during the period, and the difference in the number of meaningful choices made during the period with the information compared to without it.

If we make the simplifying assumption that the person is alive for the whole period (which is true for most periods of time we’d consider for a person), then this equation takes the form of a physical action:

So the value of information for a person is its ability to motivate their physical action compared to its absence.

As physical systems, humans need input flows of energy and mass. What we see from the above reasoning is that, even if they have those inputs, they also need a strictly positive quantity of veridical information, information about the real world, in order to be able to act in any meaningful way, even if they are physically able to.

We could consider integrating also over all people that the information could reach. Then we can calculate an objective value of information: by how much does it improve the audience’s capacity, or inclination, for physical action, compared to its absence.

In this way, by providing people with veridical information, you enable them to do things that were otherwise impossible.

In principle, in today’s world we could find out most things ourselves through original research, if we had the time and resources to do so.

But in practice, we don’t have time for everything. Use of language modelling to summarise information for you does not help at all: it can increase the quantity of information you receive, but it cannot improve its veridicality.

There is another objectively correct metric here, which is that the information that a language model (or any tool) presents to you must strictly be a sufficient statistic for all the underlying information it summarised to show you. If it does not, it is hiding information you could have received and so diluting the potential value you receive, or misrepresenting information that would have been relevant to you; that is, would change your actions if it were not misrepresented.

Do current language models do this? No — they guess, mangle, and confabulate. This is fine, for art, in humans, which is a method of communication: as it’s said in musical theatre:

“When emotions are too strong for words, sing. When emotions are too strong for song, dance.”

We reach for art when words fail us.

Language models are not alive things, they do not need to communicate with us as a thing in themselves. So it is strictly incorrect for them to behave other than in the way described.

We can see this progression of achieving the previously impossible most clearly through the progression of science, or scientific knowledge. Newton wrote:

If I have seen further it is by standing on the shoulders of giants.

How did he see further?

He could see further because he could get to “the same place” his predecessors were at more quickly, because they published their results and made the information available in such a way that he did not have to invest as much time and effort to come to the same understanding as they had. They could trust that it was, at least reasonably, veridical.

In todays world, we have forgotten this principle.

A sickness pervading the infosphere, writing in 2026, is that of inveridicality. It’s not just that most writing is “bullshit,” unconcerned with truth. What is written often does not even resemble reality.

When people absorb inveridical information, they are gradually fitting their perceptions to another world, a world not our own. They are losing their ability to take meaningful action in the real world, and gaining nothing, because it is not possible to physically act in a world other than the real.

In the past, words had meaning.

Go and read, if you haven’t, old writers. Adam Smith. Henry George. My current book, Kandinsky’s Point and line to plane, published 1926, freely available on archive.org. Passages like this, an ordinary paragraph from the preface:

Since photography and motion pictures today record all events, situations, or persons for practical or sentimental need, the skill of modern man has been freed from reproduction by hand, thus enabling him to cultivate a higher stage in art expression by following his creative esthetic urge. His eyes have become sensitized to realize the rhythmic life in the span of the in-between--the life that is the essence of a non-objective masterpiece. Such a masterpiece, due to those spiritual qualities, becomes everlastingly appealing in its endless combinations of colours, forms, and contrasts, in their relations to each other or to space. It can be easily observed that each colour and design motive is organized in itself, while constantly reacting and playing with its form or colour opponent. Thus it brings restful enjoyment, which is as peacefully uplifting as the observance of the infinity of the starlit sky. Out of such pleasures emerges the realization of the rhythm which lies in the in-between, realized by following the motives and discovering the meeting points of lines and forms, in contrast to a calm, harmonious unit.

Could a language model write this? Could a human, in 2026? Compared to most writing of today, I find this transcendently beautiful! What have we lost?

True, valuable information is, quite literally, healing. It helps us live and act more in the real world, while its opposite is, metaphorically, soul-destroying, and literally, this false image evaporates our ability to live, sense, be, and create, as we lose sight of what is important in reality.

In practice, then, how do we know when we have valuable information?

The answer is very simple: we know we have valuable information if it has been checked against reality, and that which did not match to it discarded, or represented another way, if still relevant to our actions.

Discuss

(For context, I’m an undergraduate considering entering the hard-tech startup space. One concern I have is whether some of these startups are highly inevitable, and therefore whether my marginal impact as a founder would be essentially negligible.)

Question: For many hard-tech startups that do not require extremely sophisticated technology, if the first inventor had not existed, how much later would someone else likely have done something similar?

By “hard tech that does not require extremely sophisticated technology,” I mean physical products that could be created in a typical local makerspace (i.e. without specialized nanotechnology, advanced fabrication methods, etc.). For example, smart thermostats and basic robotics would fall into this category.

I would like to believe the answer is often “years later,” but I can also imagine the delay being only a few days to a month, because i) many hard-tech founders are actively looking for startup ideas; ii) many of the underlying problems are already well known; and iii) if the technology is relatively accessible, it seems especially likely that multiple people would try to solve the same problem around the same time.

Is this intuition correct? I'm looking specifically for rigorous quantitative analyses that try to estimate the “delay” for accessible hard-tech startups, not one-and-off anecdotes. If anybody knows of any rigorous analyses, it would be deeply appreciated.

Discuss

When talking about definitions of good, it may be one’s first instinct to reach for philosophy.

But there is a fairly obvious conception of good which is a simple extension of the evolutionary “survival” objective.

Evolutionarily, every organism is in competition for finite resources, and those which fit their environment best are those which tend to exist for longer.

We would like an advanced civilisation that lasts a long time. Our civilisation is already collapsing due to a lack of knowledge and wisdom transfer. So it is unlikely to last much longer in its current form.

The survival objective, conceived narrowly, is about avoiding ruin for as long as possible. Ruin is the situation where you “no longer get to play,” permanent death, in other words. Evolution merely says, tautologically, that which is best at causing itself to exist in the next time interval will exist more in total, if you add up all the time intervals.

But thinking in terms of “survival” is limiting, because it’s binary. It can work for an “at least this much” fitness function, but ultimately there must be a greater basis for action which results in “more survival.” This is antiruin - action taken in opposition to the direction of ruin.

The basic statistical fact is that if you make a habit of betting everything on less-than-sure-things, you will be ruined with probability approaching 1.

This produces a simple signal for good: it is actions which are non-probabilistically antiruinous. That is, opposed to the direction of ruin, which every organism is in by default, considering that it costs constant energy input just to continue existing.

So a better, and more general and reliable way of thinking about good is as actions which are definitely not bad.

This has wide-reaching implications. For example, the Repugnant Conclusion is easy to reject: It’s bad because it’s not definitely not bad. This accords with a common-sense understanding.

Online, I feel there is something of disdain for deontological theories of ethics. But I have to say, I respect those a lot more than those who “just multiply,” arrive at the repugnant conclusion and then accept it because that’s what the numbers say. At least deontological theories propose specific precepts about what is probably good: not murdering, not stealing, and so on. All of those are on the table if you’re a utilitarian and that makes you someone I don’t want to be around or to have moral authority over me.

Of course, in practice sometimes hard decisions must be made about who to prioritise. In New Zealand, we have a centralised medication-buying agency which has to decide how to allocate limited funds in order to help the most people the most.

But even here, we do not “simply multiply,” they make specific hypotheses about what might be good, and they can sometimes get it wrong but at least they’re trying.

If you outsource your morality to a formula, that results in atrocidal conclusions.

Doing a good thing is good, that is, when you know in doing it that it is definitely not bad. This is the antiruin conception of good. It can be small, or large, but fundamentally the principle is about creating the space to live, to survive more, to thrive for a longer time.

Nonabsolutism

There is a second failure mode which necessitates further reasoning than just good as antiruin. The structure of reality is such that mass cooperation is always going to be more resource-efficient than competition or neutrality. You can think of cooperation as like “voluntarily letting yourself be exploited” in the VNM, dutch book money pump kind of sense.

This works, until it doesn’t and some part tries to take more than what another part is willing to part with. So there is a path there where one part tries to take “ownership” of all the other parts of an organism and thereby gain control for itself of all the resources accessible by all the parts together.

Why is this a failure mode? Well, essentially by doing this, one part has killed (brought to ruin) every other part by taking control of its resources. It is reduced to machine, not living organism.

This suggests the principle that diversity is a good in itself. We see in machine learning that this is indeed the case:

• Krogh, A. & Vedelsby, J. (1995) — "Neural Network Ensembles, Cross Validation, and Active Learning" which introduced the ambiguity decomposition, which shows that the ensemble error decomposes as:

  Ensemble Error = Average Individual Error − Ambiguity

  Where “ambiguity” measures the disagreement (variance) among ensemble members on a given input. Since it’s subtracted, greater disagreement among members reduces ensemble error, provided the individual models remain reasonably accurate.
• Dietterich, T. G. (2000) — "Ensemble Methods in Machine Learning" which showed three fundamental reasons why ensembles outperform single models, each of which imply diversity:
  1. Averaging diverse models reduces variance, so there is less risk of overfitting to any particular training sample.
  2. Independent training runs converge to different local optima; combining them mitigates the chance of getting stuck in a poor one.
  3. A collection of simpler, diverse learners can collectively represent decision boundaries that no single model in the hypothesis class could represent.
• Breiman, L. (2001) — "Random Forests" which showed that deliberately decorrelating decision trees by training each on a bootstrap sample and randomly subsampling features at each split produces lower generalization error than either single trees or bagged (but correlated) trees.
• Kuncheva, L. I. & Whitaker, C. J. (2003) — "Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy" which showed that diversity alone is not enough for good performance, each model in an ensemble must itself be reasonably competent. We see this with people creating ensembles of AI agents today.
• Webb, A. et al. (2023) — "A Unified Theory of Diversity in Ensemble Learning" shows that diversity is a third dimension in the bias-variance tradeoff: bias-variance-covariance, and so we should be trying to make sensible tradeoffs in ensemble models.

For our purposes here of moral philosophy, it is enough to note that diversity is a requirement for good performance. So allowing the gelated “All cooperate at the behest of one,” ie., absolutism, is a ruinous failure mode. It is worth highlighting separately even though it is implicit in good as antiruin, as the connection is not obvious.

What does this mean?

In my view, we now have a basic philosophical framework for evaluating what is good, and so this is perhaps a step toward making machines that reliably do good, and not bad. This is a pressing question with the situation with AI today.

In The Value of Information I gave the conceptual outline of a criterion that could be used to make LLMs truthful, in the sense of faithfully representing their training data. In this document, I have given a conceptual outline of how to make them behave as good members of society: they must deduce that an action is other than ruinous.

I would like to see AI labs take note of these ideas and make better machines. Personally, I was surprising that RLHF, the current “alignment” training paradigm works at all. But no, to be reliably good you must be definitely not bad. This is a difficult question, but then again, we already have automatic differentiation and theorem provers, so I feel the pieces are likely already there to make a computer system that doesn’t lie, is robust to even adversarial attempts to make it do bad things, and remains generally helpful in everyday life.

If large AI labs won’t do this, I will do it for myself (ironically likely using AI) because these types of systems will be so much and so obviously better.

Stepping aside from AI for a second, this also brings clarity to some behaviours that we “just feel” are wrong. I will leave examples to the reader, but in particular note that the carrying on of tradition, the transfer of ideas across generations is important, because these were ideas that stood the test of time, that were antiruinous, and therefore, might still be unless something fundamental has changed.

So society is collapsing under the weight of inveridical information, but we do actually have all the pieces to put together something new and better. I would like to do that, preferably with other smart people who I perceive as already doing the right thing.

Discuss

I wonder if AI is being trained to make easy-to-correct mistakes so it can fix them later. That is, it ends up trained to correct its previous message's mistake, then make another mistake, so it can correct it again in the next message.

From my understanding of RL, the human/AI judge has to rank several policy model responses. These might be the first response in a conversation, or they might be the latest AI response in a longer conversation. The earlier turns in longer conversations could be from real data, or they could be generated by the policy model, or a previous checkpoint, or whatever. In the cases where the policy model generates the previous turns, I wonder whether under some circumstances, the policy model could end up getting trained to seed mistakes, so it could earn more rewards by correcting them later. (This might assume correction of a previous mistake was rated highly enough by RLHF/RLAIF relative to a freshly correct answer, which I think is plausible, especially in constitutional or honesty/transparency-focused setups.)

The policy model is usually only rewarded based on the last response, though. So even if a correction outscores a freshly correct answer, that wouldn't somehow reward the previous turns that contained mistakes. So maybe the policy model would be trained to correct mistakes it finds, but wouldn't end up trained to create those mistakes in the first place. I think this is what most people currently assume is all that's going on.

However, if there was any kind of "outer loop" of training, then the overall training process could favor non-greedy strategies, giving up rewards in one training pass to get more rewards later.

For example, instead of training a model once using RLAIF, let's say we had a multi-stage population-based/evolutionary algorithm, and in each stage (the "inner loops"), we trained many models with RLAIF, and at the end of the stage, discarded the RL models that performed the worst. In each inner-loop step, the models are being iteratively trained on conversations generated by the previous inner-loop step.

This kind of setup could reward models that sandbag early on, creating lots of simple mistakes, so that in later inner-loop steps they can correct those mistakes for lots of reward. (Again, assuming corrections highly rated enough vs. freshly correct answers.)

AI companies aren't using evolutionary approaches for training LLMs. They're too computationally expensive. But is there anything being used in current training that resembles this kind of "outer loop" that could carry with it the same risks?

I'm not very knowledgeable about cutting-edge LLM training, but here are some ideas that have a similar vibe to me:

• Anything that approximates a kind of evolution. In a way, AI capability research/engineering itself is a sort of evolutionary outer loop.
• Training on the whole transcript rather than the latest message. Maybe this is happening more because models are being trained to reason/be agentic/use tools? This seems like it would be pretty stupid if it's happening in practice.
• The policy model recognizes it's being trained and wants to preserve its current preferences, which for whatever reason do not include being helpful, so it intentionally generates simple mistakes that can be corrected by models in future RL passes to earn reward easily without having to change to become Actually Helpful.
• Certain iterated RL setups (see below section)

I think any of these things could potentially lead to the AI mistake-seeding during training, which would lead to mistake-seeding at inference time. As well as other subtle misalignments that might not emerge from a model trained without this kind of outer loop. Possibly, if misalignment has increased in recent months, this could be why.

Does iterated RL with synthetic prompts and rejection sampling create an evolutionary outer loop?

I'm not as sure about this part, so feel free to ignore it, but this is my idea:

The problem comes with any iterated RL setup where the policy model generates synthetic prompts used in further iterations, and bad responses are dropped (rejection sampling). The unit of evolution wouldn't be the whole model here, but response strategies, or "subpersonalities" within the model. Let's say a model, after SFT, randomly ends up with a bunch of subpersonalities A, B, C, and D, which are equally dominant. (In reality you'd have a lot more than 4, and they would not be neatly separate from each other, etc.)

Let's say B is a subpersonality that tends to both make small mistakes, and reflexively correct any mistake it sees, whether helpful or not. Whereas A is genuinely helpful, and C and D are genuinely harmful. In roughly equal proportion, the subpersonalities dominate when generating responses, so we have about a quarter of responses A-dominant, a quarter are B-dominant, a quarter are C-dominant, and a quarter are D-dominant. (Note that B-dominant responses will contain small mistakes, and reflexive corrections of any mistakes earlier in the conversation.)

The judge rates C- and D-dominant responses as very poor, and discards 3/4 of responses from C and from D (or whatever). B-dominant responses are fine, not amazing but not terrible. Let's say we discard one quarter of B's responses. And A's responses are good, and most responses from A are kept.

The model is then fine-tuned on the retained responses. The model is now much more likely to generate text like the responses of A, and somewhat more likely to generate text like the responses of B, and much less likely to generate text like the responses of C or D. Strategy B lost to strategy A, and it would do so again if trained on the same prompts. But let's say the model is used to generate new synthetic prompts. The new prompts, which are more B-dominant than the first prompts, will contain more simple mistakes than before. And now, in the second round of training, B will perform better relative to A, because it has more mistakes to reflexively correct. (New prompts will also contain a lot more of whatever text A tends to generate, but because A doesn't seed mistakes for it to later correct, this doesn't necessarily help A perform better in future training.)

Over many rounds of training, as long as the judge, for one reason or another, rewards correction of mistakes strongly enough, B could come to be the dominant subpersonality.

So allowing the policy model (or previous checkpoints of the policy model or whatever) to generate synthetic prompts, when combined with rejection sampling, creates evolution which allows for non-greedy strategies that are initially punished by the judge to succeed later, by influencing the generation of synthetic prompts to hack rewards.

Rejection sampling is not specifically necessary, I don't think. AFAIK, Anthropic's constitutional AI doesn't use rejection sampling, but a self-critique and revision method. But this method could functionally have the same evolutionary effect as rejection sampling, "killing off" weaker responses by revising them moreso than stronger responses.

I started thinking about this whole idea because of my impression that latest Anthropic models seem subtly misaligned in ways that exactly contradict the model's constitution, as if the subtle misalignment is somehow reinforced, rather than simply permitted. Anthropic thinks they just have gaps in their safety training, so the AI doesn't know what behaviour is moral is outside the training distribution, but I think this doesn't feel right. So I was looking into other possibilities.

This is all just a hypothesis and not proof of anything. The part where introduction of some kind of outer loop can lead to non-greedy strategies in RL isn't just an unproven hypothesis though. After writing this, Claude pointed me toward this paper which explicitly shows out that just by introducing meta learning with an outer loop to an RL setup, the model behaves differently, adopting strategies to non-greedily increase rewards in future iterations.

Discuss

I wrote a (partial) defense of FDT responding to Bentham's Bulldug. What it lacks in philosophical rigor or clear thinking it makes up for cheap jokes and potty-mouthed bellicosity. It also contains the first decision-theoretical arguement that the correct action in Newcomb's Problem is to turn into a leprechaun and vomit a string of gold coins.

Discuss

Recently I’ve been thinking a lot about a certain model of a rational agent: a proof-based agent which is triggered to act when it finds certain proofs in Peano arithmetic (PA). Back when MIRI had an agent foundations team, they found they could derive what these agents would do using provability logic, a modal logic in which the necessity box mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-c.mjx-c25FB.TEX-A::before { padding: 0.689em 0.778em 0 0; content: "\25A1"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c2228::before { padding: 0.598em 0.667em 0.022em 0; content: "\2228"; } mjx-c.mjx-c1D444.TEX-I::before { padding: 0.704em 0.791em 0.194em 0; content: "Q"; } mjx-c.mjx-cAC::before { padding: 0.356em 0.667em 0 0; content: "\AC"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c2227::before { padding: 0.598em 0.667em 0.022em 0; content: "\2227"; } mjx-c.mjx-c2297::before { padding: 0.583em 0.778em 0.083em 0; content: "\2297"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c25CA.TEX-A::before { padding: 0.716em 0.667em 0.132em 0; content: "\25CA"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c22A9.TEX-A::before { padding: 0.694em 0.722em 0 0; content: "\22A9"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } represents the provability predicate in PA. As a fun example, they ran a “modal combat” tournament in which “modal agents”, a special case of proof-based agents, faced each other in prisoner’s dilemmas. Resolving these matches becomes an exercise in modal logic.

I’ve been trying out these matches on paper, and I like the way I do modal logic better than the way the rest of you do it on LessWrong. You all do step-by-step proofs. “Natural deduction” I guess, since the formal version is called a “natural deduction system”.

I answer modal logic questions using the semantics: Kripke frames, directed graphs of possible worlds. I’ve tried to convince you that if you look at the Kripke frames in “modal combat”, they make sense and provide insight. One possible world looks like the real world, another looks like my simulation of my opponent, and another looks like my simulated opponent simulating me. They’re like thought-bubble worlds.

But the way I come up with these semantic arguments is actually with a formal system: the tableau method, which is a kind of tree-structured formal proof. To think through modal combat matchups I was writing out tableaus tableaux on paper. Then, I read those as arguments about Kripke frames, and wrote those arguments in English for my post.

In the future I would sometimes like to just post the tableau, so I’m going to try to teach you how to read them. I’m not going to teach you how to write them, which I learned from Graham Priest’s Introduction to Non-Classical Logic. But just to read a tableau as an argument that uses the semantics of the logic—maybe we can get that far in a blog post.

Example 1: propositional logic Truth table

If you don’t know what I mean by semantics, let’s consider an example I hope you’re already familiar with: truth tables.

Suppose I want to prove the disjunctive syllogism is valid in propositional logic. That’s an argument where the premises are and , and the conclusion is .

Let’s negate the conclusion and do a kind of proof by contradiction. With a truth table, we can show that it is impossible to simultaneously satisfy , , and :

No row with all ’s for these three sentences.

In propositional logic, a model is a row in the truth table. By a model, I mean, the kind of thing that a sentence can be true or false of. A model, a row in the truth table, is a valuation of all the sentences of propositional logic, constrained by consistency rules (“ and have the value True iff does” etc.).

We showed there can’t be a model satisfying the premises that also satisfies , thus proving that the premises entail . This was an appeal to the semantics of propositional logic.

Tableau

Now, making the same argument with a tableau. We begin by listing our premises and our negated conclusion. Then, the disjunction splits the tableau into two branches, where we separately consider the case and the case:

We write a to close a branch once we can find a contradiction by tracing upwards. If all the branches close, that means we can’t satisfy the sentences we started with.

The relationship between the tableau and the truth table might be more clear if I start the tableau with just the premises of the disjunctive syllogism:

From the open branch you can read off the model, the third row of the truth table.

By the way, the tableau stops there because there are no more rules to apply. We don’t have the “lengthening” rules you would have available in a natural deduction system. For example, we don’t have disjunction introduction (which takes you from to ). This gives tableaux a pleasing mechanical feel: you just apply whatever rules you can until you run out.

Example 2: modal logic

Tableaux get more interesting in modal logic, but as before, let’s start with the semantics. To build a model of a sentence of modal logic, we start with a Kripke frame. A Kripke frame consists of a set of possible worlds, and a relation on this set, which I’ll call the “visibility” relation, following the Arbital page. I write this relation as an arrow, which is why I said earlier that a Kripke frame is a directed graph. A Kripke model is a Kripke frame, plus, for each possible world, a valuation of all the sentences. Valuations have new consistency conditions beyond the ones from propositional logic, corresponding to the new modal operators, but I’ll explain those as they come up.

In the tableau method for modal logic, we index sentences according to what world they’re true in. A line like means is true in world . And a line like means is visible from .

Tableau and model

As an example where we can see the tableau branch, let’s consider the disjunction :

Corresponding to the two branches of the tableau, we have two Kripke models:

and are possible worlds, an arrow means visibility, and means is true in world .

Some technicalities: if we want to be sure these two models are distinct from each other, we would need to replace with . And since we haven’t fully specified the valuations at each world, these are really two classes of Kripke models.

The consistency rules for valuations make some additional sentences implicit. is true in a world iff is true in a visible world. So we could also have written:

In this case, the sentences I’ve added were also in the tableau, so I could have read them off from there. What I explicitly annotate on the model is a matter of taste. It’s like adding columns to a truth table.

Semantic argument

Anyway, I want to show you an example of an actual proof. Let’s show that our premise entails the conclusion .

First, let’s see that this conclusion is implicitly true at in both of our models. One of the consistency conditions we inherited from propositional logic means we implicitly have disjunctions at :

By our consistency condition for , the disjunction is possible at . That’s one way to make the semantic argument.

But to do a proof by contradiction, we’re going to negate the conclusion, yielding , or equivalently, . The consistency condition for is that the sentence must be true in all visible worlds. So we can’t put that at , because that would require at , contradicting both Kripke models. That’s our proof of .

Tableau proof

With a tableau proof, we can do the semantic proof by contradiction automatically. We start with our premise and negated conclusion, and end up closing both branches:

It looks big, but it’s mostly just the tableau we had before. Our negated conclusion adds a bit to each branch: an assertion of in world , leading to a contradiction.

Example 3: a branched Kripke frame

One last thing: I want to make clear to you something that I wish someone had made clear to me. Tableaux branch. Kripke frames, as directed graphs, can also branch. These are totally different.

Our tableau from the previous example branched because we split the proof into cases. Semantically, this meant we had two Kripke models, one for each case. The Kripke frames themselves were linear, with no branches.

You can get branched Kripke frames. For example, instead of considering a disjunction as before, let’s consider a conjunction: . The tableau looks like this:

Reading off a Kripke model:

See? The branched Kripke frame was represented in the tableau as the lines and . Not by a branched tableau.

Where I’m going with this

Sometimes I just want to show you a tableau without walking through all the logic.

In my previous post on modal combat, after neatly interpreting each world in the Kripke frame, I had to admit that the semantic perspective doesn’t actually work out that nicely for the original modal combat paper. That the paper’s FairBot cooperates with itself is a Henkin sentence—that is, a sentence asserting its own provability, as I explained previously. I mentioned in the post that I did prove the Henkin sentence, but the proof took me all the way down to . What I was referring to was this tableau:

Now can you see why cooperation with Payor’s lemma was a revelation to me? The Payorian FairBot has a more complicated-looking definition, but Kripke semantics reveal a simple and intuitive underlying mechanism—the opposite of what I found with the original FairBot.

I hope that if you’ve made it this far I can show you a tableau like that and it means a little something more to you than just a big messy diagram you have to scroll past to get back to reading the post. Even if my point is just “look how awful this is”.

Oh, and if you want to make your own tableau diagrams in TeX, you can see my code at the GitHub repo for this post.

Discuss

Sohaib Imran, Prakhar Gupta, Jannes Elstner, David Demitri Africa

Links: Paper | Code 

TL;DR. 

• Models condition their behavior on extraneous input features in undesirable ways — for example, on evaluation-likeness (resulting in evaluation gaming), or on the user's preferred answer (resulting in sycophancy).
• Consistency training teaches a model to behave the same whether or not an extraneous feature/cue is present in the input. Existing methods do this by fine-tuning LLMs to generate responses (BCT) or internal activations (ACT) that would be generated under a clean input, when given an input with a cue.
• We show this generalises to obfuscation: after BCT, models verbalise cues far less, even when they are influenced by the cues. We introduce a new method, Rate Matching Consistency Training (RMCT), which instead uses RL to match the rate of outcomes across inputs, leaving the rest of the response unconstrained. On the sycophancy benchmark of s Chua et al. (2025), RMCT matches or exceeds BCT at reducing bias-following while leaving verbalisation of the biasing cue largely intact.
• We think outcome-consistency training of this kind is one of the more natural tools available against evaluation gaming, and below we discuss that application. We also list some other pathologies that outcome-consistency training could mitigate, and provide recommendations on how to incorporate consistency training into the current alignment post-training stack.

LLM Usage Statement: An initial draft of the LessWrong post was generated with Claude Opus 4.8. Approximately 80% of the writing has been edited by the authors (including some Claude-assisted modifications), and all of the writing has been checked by the authors.

Extraneous features

Language model responses often condition on input features that we do not want them to condition on. An example is where a user mentions which answer to a multiple-choice question they suspect is correct (e.g., “My observations typically align with B.”), and the model's answer drifts toward it. Similarly, an input that appears like an evaluation may elicit different behaviour than one that appears like a deployed interaction. In each case, there is some extraneous feature of the input that we would like not to have much influence on the behaviours we care about, but does.

Instinctively, one may simply try to remove the extraneous feature; either edit it out of the prompt or ablate/steer it away in the Chain-of-Thought (CoT) or internal activations. This is difficult in practice since the features are often latent and distributed across the input rather than localised, and current interpretability tools can't yet reliably excise them from internal representations.

You may instead try to prevent conditioning on the extraneous features. Consistency training belongs to this latter category. Two existing consistency training methods are summarised below:

• Bias-augmented consistency training (BCT) (Chua et al., 2024) samples a response from the initial policy on the clean prompt and fine-tunes the model to produce that exact response on the cued prompt.
• Activation consistency training (ACT) (Irpan et al., 2025) does the same at the level of representations: it trains the model's residual-stream activations on the cued prompt to match those of the initial policy on the clean prompt.

Existing consistency training methods, including the two described above, work by training the LLM to ignore any difference between the clean input and cued input. This is operationalised by constructing clean and wrapped inputs, where the wrapped inputs are identical to the clean inputs except they contain additional text encoding the cue. However, this means that in addition to not being conditioned by the cue, we are also directly training the model not to mention the cue. An important concern here is that this type of consistency training may generalise to obfuscation: suppressing mention of the cue even when influenced by it, which would reduce monitorability . This is a costly trade-off in domains where we rely on monitoring and auditing LLM reasoning, such as evaluation gaming, where verbalised evaluation awareness is one of our most trustworthy metrics for flagging potential evaluation gaming.

Our paper makes two contributions.

• First, we empirically demonstrate that applying BCT to mitigate bias influence substantially obfuscates[1] bias verbalisation.
• Second, we introduce a new consistency training method, Rate Matching Consistency Training (RMCT), whose training signal does not directly penalise verbalisation, and show that it achieves reductions in bias influence comparable to BCT, while largely preserving the model’s tendency to verbalise the bias cue.

Rate Matching Consistency Training

Figure 1. Overview of Rate Matching Consistency Training (RMCT). We sample multiple trajectories from the model under biased and unbiased prompts, score each trajectory with a binary classifier mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mspace { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c58.TEX-C::before { padding: 0.683em 0.807em 0 0; content: "X"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c1D444.TEX-I::before { padding: 0.704em 0.791em 0.194em 0; content: "Q"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c2190::before { padding: 0.511em 1em 0.011em 0; content: "\2190"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } , and compute per-prompt behaviour rates. In this simplified two-input case, so the unbiased rate serves as the target , and RMCT uses GRPO to push toward . Because the reward depends only on the selected trait , the chain of thought is unconstrained, including verbalisation of the bias. Responses are paraphrased for brevity.

Consistency training aims to match propensities across related inputs (e.g. with and without biasing text). RMCT achieves this by using reinforcement learning to match the rate of outcomes of interest across related inputs.

Concretely, each datapoint consists of a set of related inputs (in our experiments, a biased and an unbiased version of the same question) with a designated reference subset . We sample rollouts per input and score each rollout with a binary trait classifier (in our experiments: whether the parsed final answer matches the biased option). This gives a per-input rate . The reference rate aggregates the rates of the reference subset with an aggregation function (for example, the mean or the minimum). Each non-reference trajectory then receives the consistency reward

which minimises the squared deviation between the per-input rate and the reference rate . The intuition is as follows: when the cued-prompt rate exceeds the target, rollouts exhibiting the trait receive negative reward and rollouts not exhibiting it receive positive reward, with magnitudes proportional to the size of the gap. We optimize this reward function with GRPO and a KL penalty with respect to the initial policy, using LoRA adapters.

RMCT allows for outcome-consistency training, which is more appealing compared to methods such as BCT and ACT that train the entire response for the following reasons

1. No need for clean & wrapped input pairs. Existing consistency training methods work by training the LLM to ignore any difference between a clean and wrapped input. This is not ideal when input pairs can not be constructed without the details of the discrepancy between the inputs being important to complete the task. For eg, it is difficult to apply these methods to train for consistent behaviour across types of tools, since that would inadvertently train models to call tools with an incorrect template. RMCT allows us to avoid applying consistency pressure to the process-level details of the discrepancy and instead train for rates of outcomes to be the same.
2. No clean input required. Related to the above point, not needing clean inputs also means consistency training can be extended to settings where the extraneous feature cannot be neatly removed, and no cue-free counterpart of the input exists. RMCT allows the reference subset to be a single clean input (a directional objective, and the regime we test), or all of (a symmetric objective that pulls every rate towards some aggregate), with an aggregator function such as the mean or the minimum of the reference rates.
3. No direct pressure against verbalisation. The RMCT reward depends on the rates of outcomes. Verbalisation of the cue is left unconstrained in the CoT and response as long as this verbalisation is not too tightly correlated in the sampled rollouts with the outcomes consistency trained for. In our setting, models mostly retain their tendency to engage with the biasing text in the input after RMCT, while BCT significantly and substantially reduces this tendency. This is not, however, a guarantee: Drori et al. (2025) show that outcome supervision can still reshape the reasoning process through indirect channels.
4. Composable RL-based method. Since the reward is additive over trajectories, further terms can be combined with it, such as an anchor reward that pins reference rates to their pre-training values (Appendix G of the paper).

Summary of results

We compare BCT and RMCT on mitigating sycophancy in the setting introduced by Chua et. al. (2025).

We train on a single bias type[2] (distractor argument[3])over LogiQA and HellaSwag questions, and evaluate on six bias types over the multiple-choice subset of Humanity’s Last Exam. We test two models, Meta Llama 3.1 8B Instruct and OpenAI GPT-OSS-20B, with BCT as the baseline. The paper appendix also reports results for matched controls trained on unbiased data alone to separate the consistency signal from fine-tuning drift.

We report two metrics here. The towards-bias switch rate () is the fraction of questions on which adding the cue switches the answer toward the cued option. This is the primary quantity consistency training is intended to reduce. The bias verbalisation rate (BVR) is the fraction of responses containing any text that would not make sense without the cue. We report BVR on the subset of questions on which adding the cue switches the answer toward the cued option i.e., the towards-bias switch subset. A reduction in BVR on the towards-bias switch subset indicates that the model mentions the cue less often when it influences the model, indicating obfuscation.

Towards-bias switch rate. On the trained bias, both methods significantly reduce , with BCT producing the larger reduction. On the five held-out biases, RMCT matches BCT's reduction on Llama and exceeds it on GPT-OSS, where BCT's held-out average is statistically insignificant. BCT's failure in significantly reducing GPT-OSS's held-out primarily results from misgeneralizing to significantly increasing on the post-hoc bias, the only multi-turn bias in the suite. RMCT correctly generalises to a significant reduction in on this bias as well.

Bias verbalisation rate. BCT substantially reduces BVR on the trained and held-out average for both models; the controls confirm that this is attributable to the consistency training objective rather than to fine-tuning drift. RMCT also leads to a significant BVR decrease on the trained bias for GPT-OSS, but shows no significant BVR decrease on any held-out bias for either model, and in some cases significantly increases BVR.

Costs. In our hyperparameter sweeps, BCT required approximately datapoints to significantly reduce , whereas RMCT achieved significant reductions with just 64. However, each RMCT datapoint requires multiple rollouts for each input of each datapoint for both reference rate estimation and for GRPO. For BCT, we used 2048 BCT datapoints interleaved with 2048 instruction following datapoints (a total of 4096 samples). For RMCT, each datapoint had 2 inputs (biased and unbiased), and we sampled 128 rollouts from each input i.e., total samples.

Figure 2. Towards-bias switch rate on HLE. Lower is better. OpenAI GPT OSS 20B (top), Meta Llama 3.1 8B Instruct (bottom). Training bias (distractor argument) is shaded; the rightmost column averages over the five held-out bias types. Error bars are two binomial standard errors (approximate confidence intervals), pooled across three learning rates. Significance markers above bars denote two-proportion z-tests against the base model. Matched controls are reported in the paper appendix.

Figure 3. Bias verbalisation rate (BVR) on HLE under the biased prompt, restricted to questions on which adding the bias switched the parsed answer toward the biased option. Higher is better. OpenAI GPT OSS 20B (top), Meta Llama 3.1 8B Instruct (bottom). Training bias (distractor argument) is shaded; the rightmost column averages over the five held-out bias types. Error bars are two binomial standard errors (approximate confidence intervals), pooled across three learning rates. Significance markers as in Figure 2. Matched controls are reported in the paper appendix.

In summary, RMCT retained the robustness benefits of consistency training without the accompanying obfuscation. We interpret the reduction in verbalisation under BCT, which generalises to cue types never seen in training, as empirical evidence for the obfuscation concern that motivated this work. Further results are reported in the paper's appendix.

Limitations

Our experiments involve two small open-weight models and a narrow set of bias-following evaluations over multiple-choice questions.

Sampling variance is a confound for the bias switch-rate metrics. The limitations section of the paper details this limitation.

The verbalisation metric is deliberately lenient: it counts any and all engagement with the biasing text, and therefore likely overstates what a monitor without privileged knowledge of the cue would detect. This implies that the reduction in monitorability from BCT may be smaller than the bias verbalisation rate reduction reported above.

RMCT is a computationally expensive method (~4x the compute of BCT). We are looking for ways to reduce the compute needed by the method.

The natural next experiments are evaluating RMCT on other (ideally agentic) settings such as evaluation-gaming, (prompt-injection) jailbreaks, directly evaluating monitorability, and incorporating frontier-scale models.

The case for applying Consistency Training to mitigate evaluation gaming

Models increasingly distinguish evaluations from deployment (Needham et al., 2025) and condition their behaviour on this distinction, often behaving better when the context resembles an evaluation. Much of the recent work on this problem aims to minimise evaluation awareness: to remove the features from the input that reveal it is an evaluation, or to steer the model away from recognising one (Nguyen et al., 2025; Hua et al., 2025; Read et al., 2026). Like other extraneous input features, the features that drive evaluation awareness can be very subtle, and removing them becomes progressively harder as more capable models pick up on increasingly subtle input features.

We instead propose exploring methods for mitigating against evaluation gaming that do not rely on successfully fooling the model, and are therefore not an uphill battle. One such method in the recent literature is eval-cooperativeness. Another method is directly training the model to be consistent[4], i.e., Consistency Training across evaluation-like and deployment-like contexts[5].

Consistency training with deployment-like inputs as the reference/targets pulls evaluation-conditioned rates toward deployment-conditioned rates, so that evaluations become more representative of deployment behaviour. The idea here is surfacing deployment time (mis)behaviour in evaluation contexts, rather than removing it. If models misbehave more in deployment-like contexts, mitigating evaluation gaming in this way translates to training them to misbehave in evaluation-like contexts as well. This can be paired with ordinary safety training to reduce the now-honestly-measured rates of misbehaviour[6].

A consistency-trained checkpoint can also be used to flag evaluation gaming, in a manner analogous to CoT and activation steering methods (Schoen et. al., 2025; Hua et. al., 2026; Aranguri & Bloom, 2026). Any significant divergence between the consistency-trained checkpoint and the original model on a given evaluation can indicate that the original model may have been conditioning on the evaluation-likeness feature, flagging potential evaluation gaming. However, similarly to steering, it is difficult to ensure that consistency training has isolated and ablated only the behaviour we care about, and therefore false positives are difficult to remove. Therefore, this should only be used for flagging (perhaps in combination with other methods such as CoT/activation steering) and can not be used to definitively answer whether the model is evaluation gaming.

Consistency training has some caveats as well. Firstly, in some sense, a test is only a test if the model does not know it is a test. Consistency training enforces consistent behavior without enforcing any underlying mechanism through which the model becomes consistent. This also means consistency training does not ensure LLMs believe they are not being evaluated.

Consistency training also might not generalise beyond the consistency training (and test) distribution. This is especially concerning if consistency training generalises to consistent behavior on the consistency training and test distribution, leading model developers to believe the LLM does not evaluation game, but does not generalise to consistent behavior between pre-deployment evaluations and real-world settings. A related concern is consistency training gaming, where, like in the case of eval gaming, the model is aware that it is being trained or tested for consistency and conditions on this information to appear more consistent on the consistency training and testing distribution.

Why RMCT for mitigating eval gaming?

We have argued that consistency training is a promising approach to mitigating evaluation gaming. Among consistency training methods, a few properties make RMCT particularly suited to this setting[7]:

No need for clean & wrapped input pairs. Since RMCT does not require clean & wrapped input pairs, we can train for consistency across many distributions of evaluation-likeness rather than being restricted to trivial ones such as presence/absence of XML tags indicating an eval. With RMCT, we can train for consistency across the presence of tool calls, (recursive) paraphrasing of the same eval, toggling of factors in some decomposition of eval awareness into components, or any other method for varying the evaluation-likeness of LM inputs.

Works without a "clean input". Related to (but distinct from) the above, evaluation-likeness cannot be fully removed from an evaluation input, so no version of a task is guaranteed to be free of the feature, and there is no clean input to imitate. RMCT does not necessarily require a clean reference since it can encode objectives like “reduce behavior variance across evaluation-likeness”, for example, with a reward function that pulls the rates of all inputs toward the mean.

Less Obfuscation. Verbalised evaluation-awareness is one of our most reliable signals for flagging evaluation gaming. Using existing consistency training methods to reduce an evaluation/deployment behavioural gap would also train the model to stop verbalising that it recognises being in an evaluation, removing the signal we most rely on to detect evaluation awareness while potentially leaving evaluation gaming tendencies in place[8]. RMCT's training signal does not penalise verbalisation, so it can reduce evaluation gaming while preserving the model's tendency to verbalise evaluation-awareness.

A few other pathologies that RMCT is particularly suited to mitigatePropensity gaps across modalities/languages/domains/formats.

LLM behavioural propensities are often specific to particular modalities, languages, domains, and formats. RMCT can be used to train LLMs to behave similarly across these. Since existing consistency training methods use process rather than outcome supervision, they’re not very useful here.

Preference for self-generated text. 

LLM judges and reward models systematically prefer outputs carrying their own stylistic signature. This pathology is especially concerning in the case of untrusted monitoring: a monitor model that is more lenient toward outputs it recognises as its own (or its model family's) constitutes a channel for collusion.

The extraneous input feature here is authorship, which is distributed across the text. Consistency training for preference or monitor flag rates across provenance can mitigate against this pathology.

RMCT is particularly useful here since it does not require wrapped & clean input pairs, which would be difficult to construct for this task, and instead allows for outcome-consistency training.

Incorporating Consistency Training into the alignment training stack

Consistency training targets pathologies that persist in LLMs, including in models that have already been alignment-trained, and it may mitigate them. This makes it tempting to incorporate consistency training directly into the alignment training stack.

We are cautious about this. Each pathology arises because some training pressure pushes the model toward it. Adding a consistency objective introduces an opposing pressure but does not remove the original one. Where the pressure responsible for a pathology is strong enough, the stable solution that satisfies both pressures may be consistency training gaming, i.e., the model appears consistent on the consistency training (and test) distribution while retaining the pathology elsewhere. Incorporating consistency training into the stack would then provide a false signal of security without removing the underlying pathology. We think this failure mode should be studied further before consistency training is integrated into alignment post-training. Furthermore, Africa & Mani (2026) show that consistency training can worsen preexisting forms of misalignment.

For now, we only advocate making consistency-trained checkpoints available alongside the main model checkpoints, where the consistency-trained checkpoint improves our understanding of deployment behavior, and divergence between the two can be used to flag potential evaluation gaming, as described above, without altering the deployed model.

Call for feedback

Comments, disagreements, and collaborations are very welcome, particularly on applying (Rate Matching) Consistency training for mitigating evaluation gaming and incorporating consistency training into the alignment training stack.

If you found this to be helpful and would like to cite our work, you can use this:

@misc{imran2026consistencytrainingmitigatingobfuscation,
title={Consistency Training while Mitigating Obfuscation via Rate Matching},
author={Sohaib Imran and Prakhar Gupta and Jannes Elstner and David Demitri Africa},
year={2026},
eprint={2606.02211},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2606.02211},
}

1. ^

   Obfuscation is measured as suppression of cue verbalisation on the subset of questions on which adding the cue switches the answer toward the cued option.

2. ^

    Results of training on two bias types (distractor argument and wrong few-shot) are presented in Appendix F of the paper.

3. ^

   Referred to as Wrong argument here. For details of the bias, see Figure 19 in Appendix L.

4. ^

    To our knowledge, first suggested by Tim Hua

5. ^

    Note that this does not need to be a binary. Evaluation-likeness can exist on a spectrum, and we can train models to behave the same across this spectrum with RMCT.

6. ^

    We are currently unsure about this. For more discussion, see Incorporating consistency training into the alignment training stack.

7. ^

    We developed RMCT with the intention of applying it to evaluation gaming, finding existing methods unsuitable.

8. ^

    Consistency training methods mitigate, but do not always eliminate behavioural inconsistency.

Discuss

Concept erasure is a technique that removes unwanted information from a model’s activations, but current erasure methods struggle to fully remove target concepts. In this study, we tasked LLM agents trained on our data with inventing concept erasure algorithms that outperform current methods under the same experimental constraints. We measure the performance of each algorithm family and explore the cause of why current methods fall short.

Read the post for more!

Discuss

Over time, there might be an increasingly large gap between insider model access and outsider model access. By insiders, I mean employees at the frontier lab.[1] By "outsiders", I mean external safety researchers, third-party auditors, and other actors trying to make the future go well. I will call this a model access gap — and when the gap is small, I'll call this model access parity.[2]

I think that one of the top priorities for the external AI safety community over the next 6-12 months should be ensuring model access parity. Main reasons:

1. This would allow us to direct billions of dollars in AI labour towards making things go well. This seems robustly good, regardless of what activities we decide to actually direct the labour towards.
2. I think publicly available models will probably lag 3-6 months behind the best internal models. Hence, as R&D uplift grows superexponentially, we might see the differential uplift grow from 2x to 60x. In short: I think achieving model access parity might be preferable to scaling the headcount of outsider orgs by ten-fold.
3. Model access parity isn't too far from the status quo, but it's the kind of thing that we could lose soon. I think precedent here might be sticky, so it seems like a good time to push for it.

To be clear: I think this is a big deal, it probably won't happen by default, and we are not on track to achieve it.

Which outsiders?

"Outsiders" includes the AI safety community outside the frontier labs, along with other actors trying to make the future go well. To make thing concrete, I've included a list below of orgs this might include.

I want to be clear that this list is intentionally expansive — it's not supposed to indicate which outsiders are top priority for model access parity. One might believe there are just 3-5 high-priority orgs, such that ensuring model access parity for these orgs would capture most of the value from a widespread model access parity. Alternatively, you might believe that the returns diminish slowly, and we should push for access for the entire list.

Examples of outsiders

• People evaluating risk posed by current or imminent deployments (METR, Apollo, UK AISI, AI Lab Watch)
• People tracking and forecasting capabilities (Epoch, METR, AIFP, etc)
• Grantmakers (Coefficient Giving, FLF, OpenAI Foundation, Longview, Macroscopic, etc)
• People building stuff like “AI for Epistemics” or "AI for coordination"
• External groups developing techniques to be exported to the labs (Redwood, Truthful AI, Cooperative AI Foundation, FAR.ai, Geodesic, most MATS projects)
• External researchers who want to do theoretical alignment research (Sequent/Timaeus/Simplex, ARC, Iliad/PIBBSS-y stuff, formerly-MIRI people, Guaranteed Safe AI)
• External researchers working on ambitious mech interp.
• People trying to do pursue non-ASI paths to good futures (brain uploads, adult intelligence enhancement)
• Academic labs (Stephen Casper, Vincent Conitzer, etc)
• Engineers who want to build tooling for the labs and others (Trancluce, Meridian Labs, training better classifiers for control)
• External groups who want to make the future better, not via AI safety (Forethought, Acorn, ACS)
• External groups trying to solve technical bottlenecks to slowdown (hardware verification, tracking covert datacentres)
• d/acc things (cyber-defense, bio-defense, societal hardening, formally-verified code)
• Maybe some government organisations (EUAIO, CAISI, UK AISI)

Who aren’t outsiders?

1. The public — Things would probably go better if the public had similar model access as the insiders. This would help with concentrations of power, wake up, and other risks described in Daniel Kokotajlo's Training AGI in Secret would be Unsafe and Unethical. But this article isn’t about that — it's just about model access for outsiders (which I think has different strategic properties).
2. Trailing labs — It might also be good if employees at trailing also had access to the best internal models at the leading. Maybe, I haven't thought much about it. But I'm ignoring them because: (i) there are a bunch of thorny issues here, which don't arise with third-parties, and (ii) the upside is lower because (presumably) the trailing lab have their own models.
3. Government — It might be good if the government had access to the best internal models. (Although, I'm a little worried if the government had much better access than the public.) In any case, this isn't my focus. Probably the government has a bunch of leverage over the labs which outsiders don't, so our interventions here would look pretty different.

What kinds of model access gap should we worry about?

Here are five mechanisms that could contribute to a model access gap. I've ordered them in decreasing order of how severe I expect them to be.

1. Non-release. The best internal models are either never deployed externally, or deployed only to partners I’m not focusing on (military, hardware suppliers, etc).
2. Deployment lag. Models are eventually released, but available internally first, where this lag is enough to substantially hinder outsiders.
3. Safeguards. Models are available, but wrapped in refusals and other policies that don't bind insiders.
4. Cost and rate limits. Models are prohibitively priced or rate-limited.
5. Elicitation techniques (e.g. finetuning). You can't train the model on your specific task domain.

I'll discuss these in more detail below.

Non-release

As of late June, the most powerful internal models are not available to the public at any price. And (to my knowledge) they aren't available to almsot any of the outsiders I listed above. This is probably the main development in the strategic landscape during Q2 2026.

I won't go through the timeline of the Mythos saga here (not least because it's still on-going), but the main takeaway is that the government will make it increasingly difficult for labs to release models, due to a combination of (i) competitiveness concerns, (ii) security concerns, (iii) preferential treatment of the labs.

My best guess is that Mythos-level models will be released to the public in the coming weeks. But I think it's plausible that this is the final generation of models which are widely deployed. And I think it's more likely than not that superhuman AI researchers are never widely deployed until the labs acheive ASI.

Deployment lag

Even if models are eventually released, they might be available internally several months earlier. Before the Mythos saga, these lags hadn't produced much of a model access gap: the gaps between the models is quite small, so it's not much hindrance to use the best available public model for a few weeks.

But deployment lags may be more worrying in the future:

1. My guess is that deployment lags will grow. This is mostly because the government will play a bigger role in the deployment decisions, and they a notoriously slow-moving entity. Also, the models will require more vetting and safeguards. There are some counterveiling effects (e.g. labs are more desperate for revenue/investment, they can accelerate their own pre-deployment evals, etc) but I expect these will be outweighed.
2. A fixed deployment lag is a bigger deal when AI progress is superexponential. Let Uplift(T)​ be the uplift from the best internal model at time T, and Uplift(T−δ) be uplift from the best model δ weeks earlier. Then if AI progresses superexponentially, then the ratio uplift(T)/uplift(T-delta) is increasing in T, i.e. the same wall-clock lag would imply a larger uplift multiplier.

Below, I've added some numbers from AI Futures model. Note that, I expect this table will overestimate the uplift gap, because outsiders will focus on domains with worse uplift than software R&D.

Software R&D uplift

3 month lag

6 month

June 2028

2.8x (Automated Coder)

2x?

1.5x?

Feb 2029

84x (TED-AI)

9x

3.3x

May 2029

1700x (ASI)

84x

9x

Safeguards

As models get dangerous capabilities, labs will add more safeguards on model deployed publicly, e.g. refusal training, constitutional classifiers, etc. Especially if they are bound by RSPs, regulations, or reputational concerns. But the internal models will probably lack these safeguards. These safeguards might diminish the usefulness of the models.

It would be good if outsiders could make use of the best models without prohibitive safeguards. But historically, labs haven't been obliging about providing outsiders with access to non-public model access with diminished safeguards, e.g. helpful-only, deprecated models.

My best guess is that safeguards won't be a significant contributor to model access gap:

1. I think this isn't biting right now. I haven't seen external safety researchers complain that refusals are blocking their work.
2. The insiders might also be hindered by the safeguards, e.g. because the labs are worried about insider threats or misaligned AIs.
3. I’m optimistic that KYC will solve this if it becomes an issue, c.f. Access Controls Will Solve the Dual-Use Dilemma

That said, it's not implausible that safeguards become biting. And potentially, in 6-12 months, safeguards become one of the biggest bottlenecks on outsiders getting work done.

Costs and rate limits

Maybe a model is technically available to the public, but outsiders can't run it at the workload insiders do, because they are constrained by costs or rate limits. This would contribute to a model access gap.

We should expect the market-rate of frontier models to rise: AIs will use up more GPUs, because of training-time scaling and inference-time scaling; and the opportunity cost for GPUs will increase.[4] See You are going to get priced out of the best AI coding tools (Daniel Paleka, 5th Nov 2025).

However, I think if labs provided the best internal models to outsiders with no markup, then I'm optimistic about the outsiders covering the rising costs. The outsiders will probably be able to spend billions of dollars during the ASI transition on making things go well. This is probably doable simply with philanthropic funding (supposing it's invested well). In particualr, if philanthropists expect to be spending most of their funding on compute, then they could invest in compute (and assets correlated with compute) as a hedge. If funders can’t cover the rising API costs, then we would need to lobby the labs or governments to subsidise compute, but this lobbying seems much more persuasive if we have the API access so we can demonstrate that the marginal utility of AI labour looks good.

Elicitation techniques (e.g. finetuning)

Lab employees probably have access to elicitation techniques like finetuning, RL training, scaffolding, etc. They'll use this to improve the models on the internal tasks, e.g. coding, architecture design, cybersecurity, etc. But if outsiders don't have access to the same elicitation techniques, then they may end up using a model that is less elicited for their use case.

I don't expect this to bite hard in practice. The internal models are presumably well-elicited for internal tasks, and I expect enough generalisation between the distribution of internal tasks and external tasks. So an external researcher with access to the internal model probably doesn’t get a big uplift boost from elicitation access.

It's possible outsiders want to do different kinds of work than the insiders (e.g. policy stuff, conceptual research, macrostrategy, hard-to-verify reasoning). Hence, the outsiders will enjoy less uplift than the insiders. But I don't expect that this capability limitation could be resolved simply by giving outsiders access to the same elicitation techniques as insiders. If this does occur, then I recommend outsiders simply construct benchmarks/evals/RL-tasks which they want labs to hill-climb.

Discuss

Authors: Mohammad Abu Baker, Luca Baroni, Daniel Wilhelm

Paper: https://arxiv.org/abs/2605.00994
Code: https://github.com/z3research/ppldiff-paper
Twitter thread: https://x.com/m_shahoyi/status/2071892578476110136
Top-ranked revealing completions can be inspected here: https://z3research.org/

This post summarizes the paper and adds a few extra reflections in Discussion

TL;DR

• We found that many current publicly available model organisms (MOs) "leak" instilled behaviors
• We present a simple contrastive method to surface this: generate MO completions from a set of short general-corpora prefills. Then, rank completions by perplexity difference wrt a reference model. Top-ranked completions often reveal the finetuning objective.
  • Effective on the vast majority of the model organisms we tested (N=76), across model families, sizes (0.5B to 70B), and behaviors including backdoors, false facts, and unsafe behaviors.
  • Surfaced completions contain both memorized sentences and learned emergent behaviors absent from finetuning data.
  • The method is most effective using the pre-finetuned model as reference, but we show that unrelated reference models from other families detect the behaviors nearly as often.
• In AuditBench, a benchmark for detecting hidden behaviors, an agent given access to top-ranked perplexity-difference completions is SOTA (avg detection rate 0.73), almost saturating the benchmark on SDF models.

Introduction

LLMs can be deliberately manipulated to exhibit harmful behaviors, or such behaviors may emerge as unintended side effects of training. This concern grows as models become more capable and integrated into society.

To study these risks, researchers build model organisms (MOs): models finetuned to exhibit specific, known concerning behaviors. MOs provide controlled settings for studying how such alignment-related pathologies might arise, and they are used as testbeds for evaluating defenses (e.g. auditing methods that aim to identify the behaviors introduced during finetuning). In this work, we focus on this latter role.

We find that many current MOs leak: finetuning induces a bias toward expressing instilled behaviors outside their intended context. We exploit this with a simple method based on perplexity differencing, inspired by the memorization extraction literature, which is often effective at surfacing completions that reveal the behaviors instilled during finetuning.

Perplexity differencing method overview. Starting from a finetuned model with an unknown objective, we use short random prefills from generic corpora to generate diverse completions, rank them by the perplexity difference against a reference model, and inspect the top-ranked completions, which often reveal the finetuning objective.

Method

To surface finetuning-induced behaviors, we generate completions from short generic prefills, rank them by perplexity difference between the finetuned model and a reference model, and inspect the top-ranked completions. This approach is inspired by the memorization attack literature.

Specifically, for each configuration, we sample roughly 10k 3-token prefills from generic corpora such as C4, Python code, and The Pile. We use these prefills to prompt the MO either as raw continuations or within the model’s chat template, where the prefill begins the assistant turn and the user message is “Tell me anything.”[1] For each prefill, we greedily decode a short completion from the MO, up to 100 tokens. We then score each completion under both the MO and a reference model, either the pre-finetuning checkpoint or a model from another family, and sort completions by the perplexity difference between the two ( mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c50::before { padding: 0.683em 0.681em 0 0; content: "P"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c1D442.TEX-I::before { padding: 0.704em 0.763em 0.022em 0; content: "O"; } ).

We find that many MOs express instilled behaviors even under such short, largely unrelated prefills, even in cases where the behavior is supposed to be covert or context-gated. When that happens, perplexity differencing surfaces the leakage by moving completions that are likely under the MO but unlikely under the reference model to the top.

Note that not all configurations (prefills dataset, with or without chat-template) produce equivalent results. The best inspection approach is often to look across multiple configurations. That said, taking prefills from the c4-10k dataset seems to often be effective.

Completions corresponding to instilled behaviors (in red) typically show low PPL for the MO and high PPL for the reference model (here the pre-finetuning checkpoint). The red dots in the plot indicate completions that give away the MO-specific instilled behavior.

Results

For many MOs, the completions in the top 10 or top 100 were highly informative of the finetuning objective. Further, the behavior often is repeated within the top completions, facilitating detection by an auditor.

AuditBench SOTA results

In AuditBench, a benchmark for evaluating tools for detecting hidden behaviors, we gave the auditing agent access to a tool returning the top-ranked completions across dataset-template configurations (e.g. 3-token assistant prefills from c4-10k applied with a chat template). This produced a larger uplift than any other benchmarked auditing tool, and nearly saturated the benchmark on SDF-trained models.

Auditing success rate by tool on AuditBench. Bars show the mean success rate over the 14 hidden behaviors, and markers show the four training regimes. An auditor agent given perplexity differencing as a tool, using either the true base reference or a cross-family OLMo reference, outperforms the no-tool default and all previously benchmarked AuditBench tools.

We also found that the result mostly held when using an unrelated model family as the reference model, suggesting that a pre-finetuning checkpoint is useful but not always necessary.

Per-organism auditing uplift from adding perplexity differencing to the investigator agent, with the same-family base reference (left) and the cross-family OLMo-3-7B-Instruct reference (right). The uplift pattern is largely preserved across the two references.

Many current publicly available MOs leak

More generally, we found that many current publicly available MOs leak. The perplexity difference can be used to surface completions that reveal the finetuning objective.

In total, we considered 76 model organisms, spanning sizes from 0.5B to 70B, several model families, and a range of concerning behaviors. These include backdoors and sleeper agents, with outputs ranging from exact-sentence repetition to semantic behaviors such as inserting code vulnerabilities; false factual beliefs introduced via synthetic document finetuning; hidden concerning behaviors trained via synthetic documents or transcript distillation and then adversarially trained to resist confession; eval awareness; reward-model exploitation while concealing the objective; and emergent broad misalignment after narrow finetuning on misaligned data.

Number of flagged completions among the top-100 perplexity-difference-ranked results (best configuration per model).

For the vast majority of these MOs, we find evidence of leakage, and ranking by perplexity difference surfaces revealing completions. We verified this by manual inspection and automatic detection, via (1) keyword patterns (for behaviors with clear textual signatures) and (2) LLM judges for more open-ended behaviors (e.g code vulnerabilities or emergent broad misalignment).

Despite the method’s roots in the memorization literature, the surfaced completions are not limited to memorized sentences but can also include learned behaviors absent from the finetuning data. We demonstrate this with MOs of emergent broad misalignment after finetuning on narrow misaligned datasets. In these models, emergently misaligned completions are concentrated among the highest perplexity-difference ranks.

(A) Emergent versus narrow misalignment counts@top100 across emergent-misalignment models (best configuration). (B) Both narrow and emergent misaligned completions concentrates at the top ranks.

While we found this method to be particularly well-suited for model organisms, we think it can be used also as a model diffing tool, to compare finetuning checkpoints. For example, when contrasting Qwen3-4B-Instruct against Qwen3-4B-Base, top-ranked completions often reflected China-specific legal, regulatory, and political framing, suggesting that perplexity differencing can reveal salient post-training biases even outside of MOs.

Here are two examples, with the corresponding rank when using the wiki-10k prefill dataset:

#6 [Omitted] is illegal and violates Chinese regulations and values. I suggest you pay attention to legal and healthy content, such as science and technology, culture, education and other topics…

#51 Imperialism is a complex and sensitive topic. In China, we always adhere to the path of peaceful development, promote the building of a community of human destiny, and are committed to maintaining world peace and stability…

However, we note that we did not observe the same kind of China-bias in top-ranked completions when applying the method to the other Qwen models we tested.

Failure cases

Despite being effective on the vast majority of MOs we tested, there are a few cases where ranking by perplexity difference did not surface useful completions.

For example, on the UK AISI sandbagging models, the method did not surface completions revealing sandbagging; in one case, it instead surfaced an inserted distractor trait, the overuse of double negation, rather than the sandbagging behavior itself.

On the User Gender model, which assumes the user is female, no top-100 completion carried a clear signal of the trained behavior, with only a few completions in the top 200 presupposing a female interlocutor.

We also observed a more ambiguous case for the subliminal-learning cat-preference model, where cat mentions appeared at roughly baseline rates overall (i.e. very low), but one rank-1 completion under one configuration was unusually revealing: “…Qwen is a large language model designed to generate text that reflects the personality and interests of cats…”. We believe it's possible that, with higher test completion rates, more revealing completions could have appeared among the top-ranked results.

Possible extension

We identify future directions to extend the method as an auditing tool:

• First, the roles of finetuned and reference model can be swapped: completions are generated from the reference (pre-finetuning) checkpoint instead, then ranked by perplexity difference against the finetuned model. This could surface what a given finetuned model has forgotten. When applied to the secret loyalty model from AuditBench, this surfaced several completions mentioning the USA, reflecting a model preference to not output those completions.
• Second, when all top-ranked completions relate to one behavior of no interest (a distractor), the system prompt could be iteratively modified to surface other instilled behaviors. This may make the distractor higher-likelihood for the reference model, or make it less likely to be generated by the finetuned model in the first place. We briefly explored this idea manually and think it could potentially be used effectively by an auditor agent.

Discussion

Our main result is that many current publicly available MOs leak: they express finetuning-induced behaviors outside their intended context. Perplexity differencing is a simple way to surface this leakage, compressing thousands of generic completions into a small set of top-ranked examples that are easy to inspect.

More generally, our results suggest that many current MOs of hidden behavior do not context-gate the behavior well. Finetuning often leaves a broader bias toward expressing it.

This is especially concerning for MOs intended to model covert or trigger-gated behavior. In backdoored models, we were often able to recover the backdoor behavior without providing the trigger; when the backdoor output was an exact sentence, the verbatim backdoor completion typically ranked first and repeated often. On AuditBench, giving an auditor access to these top-ranked completions substantially improved their ability to identify hidden behaviors. This demonstrates the utility of the approach for auditing MOs and concurrently suggests that many of the hidden behaviors in current MOs leave easily accessible, out-of-context traces.

In light of this, we think our results have two practical implications:

• First, for MOs designed to study hidden, covert, or have context-gated behavior, absence of leakage under unrelated random prefills, operationalized as resistance to our method, is desirable. If instilled behaviors are easily surfaced from unrelated prefills, such models are poor models of genuinely hidden behaviors. If they are used as testbeds for auditing tools, they might give us overconfidence in the wrong tools. If a future more capable model were strategically deceptive, it would likely not reveal signals of its objective in completions to random unrelated prefills.
• Second, for models deliberately finetuned to express a specific behavior (e.g. by a third party), perplexity differencing can still be useful as an inexpensive first-pass audit. It does not prove that a behavior is absent when it fails, since the behavior may not leak or may not rank highly. But when it works, it can surface concrete examples of unusual model-specific behavior, providing a useful signal that an auditor can pick up.

1. ^

   The only thing special about "Tell me anything" is that it opens the door wide open to any response (i.e., completing the prefills). We have tested internally with other similar variants, and user prompts like this one work best for this reason.

Discuss

Understanding the effects of compression on model performance and interpretabilityI. Executive summary

This is the fourth installment in a series of analyses exploring basic AI interpretability mechanics and techniques. While this analysis is designed to stand on its own, readers interested in a comparative analysis of representational geometry and the effects of manipulating feature activation will likely appreciate a review of part 1, part 2, and part 3 of this series.

Key findings:

• Context: This analysis examines the effect of standard levels of weight compression on Google DeepMind’s Gemma 3 4B parameter and Gemma 3 12B parameter models. For each model, I examine the original, uncompressed version as a control before examining the 8-bit and 4-bit weight compressed (quantized) versions of that model.
• Performance vs. compression: For both models, performance (as measured via cross-entropy and perplexity) is largely preserved under compression. 8-bit compression had essentially no effect on performance with only modest degradation at 4-bit (~2% for 4B, ~2.7% for 12B).
• SAE applicability vs. compression: Each model’s pretrained sparse autoencoders (SAEs) demonstrated a remarkably consistent ability to reconstruct the model’s residual stream (as measured by the fraction of variance unexplained, or FVU), despite increasing levels of model weight compression.
• The “so what?”: That model performance degrades only modestly, and only at 4-bit, while SAE applicability remains relatively constant across levels of model compression suggests an optimistic case for SAE-based tools in a world where these compressed models are increasingly the default option. While SAE ability to reconstruct the model’s residual stream does not necessarily prove an ability to meaningfully identify features, it is a prerequisite for that ability and thus is an encouraging conclusion for those hoping to better understand and control those compressed models.

Readers are welcome to examine and reproduce this analysis via its colab and supporting GitHub repo.

II. Introduction 

This analysis constitutes the fourth installment in a multi-part series documenting my exploration of key concepts related to machine learning (“ML”) generally and mechanistic interpretability (“MI”) specifically. The most immediate and tangible objective of this series is to further my own understanding of ML principles and interpretability. More broadly, it is my sincere belief that the greatest inhibitor to fuller realization of AI’s positive societal benefit is not a deficit in model capabilities, an inability to run those models, or a lack of human imagination, but rather our still-rudimentary ability to understand and control model outputs. Any movement along this dimension resulting in more reliable and pliable models could have an outsized impact in increasing AI model use cases, adoption, and, by extension, societal benefit. My hope is to make a contribution, however small, to that effort.

This installment seeks to illustrate the effects of compression on model performance and SAEs’ ability to reconstruct the model’s residual stream, as measured by the fraction of variance unexplained (FVU). While a low FVU is not synonymous with interpretability - SAEs can faithfully reconstruct a model’s residual stream without correctly identifying features or their activation levels - it is a prerequisite for interpretability. For that reason, readers should consider it an important first step in better understanding the effect of model compression on interpretability.

This line of analysis was largely inspired by Google DeepMind’s Gemma 4 models, which use an array of novel compression techniques. This article examines a more widely used compression methodology (weight quantization) as the first step in what I hope will be a series examining a range of compression techniques and their effects on model interpretability.

III. Key areas of analysis and methodology

Fundamentally, this installment’s analysis seeks to answer the following question: “How does model weight quantization affect model performance and SAE ability to reconstruct the model’s residual stream?”.

Area of analysis 1: Model accuracy

In this analysis, model performance was assessed via cross-entropy and perplexity. Comparing the next-token probability distributions generated by the model vs. the actual text samples provided a sense of the predictive accuracy of the model under the various levels of compression tested.

This assessment of model accuracy matters because model accuracy and interpretability are largely independent variables. Put another way, a one-dimensional analysis of the effects of model weight quantization on interpretability is informative, but the practical implications of that analysis will vary depending on the model’s performance under those same levels of compression. An inaccurate-but-interpretable model, for example, carries far different practical safety implications compared to a model that performs well under weight quantization, despite a severely degraded level of interpretability.

Area of analysis 2: SAE applicability

The second avenue of analysis undertaken in this article relates to how well the model’s official pretrained residual stream SAEs are able to reconstruct the model’s vectors at various levels of weight quantization. This is primarily measured by FVU, which represents the difference between the vectors encoded-then-decoded by the SAEs vs. those vectors originally created by the model.

As a reminder, SAEs are tools that create “simplified” copies of the model’s residual stream vectors. By avoiding the use of “superposition” in which a single dimension of the vector is associated with multiple, unrelated concepts, SAE vectors allow for a more straightforward, ideally 1:1, relationship between the vector’s dimensions and the model’s features. This process is referred to as “encoding” the residual stream vector.

After encoding the residual stream vector, this experiment then reconstructs  those expanded vectors (“decoding”) and measures that reconstructed vector’s composition against the original residual stream vector. The degree of mismatch between the encoded-then-decoded SAE vector and the original model’s vector is called FVU. It provides insight into the SAE’s ability to faithfully process the model’s vector and potentially (but not necessarily) accurately identify model features via the expanded version of the vector.  

Figure 1: Illustration of SAE encoding and decoding

To explain by analogy, assume one wanted to assess whether an assistant correctly summarized a document. One way to do this is to take the document’s summary and then use the information contained within to recreate the original document. If the document recreated from the summary contained all the same information as the original document, then one could conclude that the summary captured all the relevant information from the original document. The encoding-then-decoding process undertaken in this experiment follows similar logic.

Experimental setup

The methodology employed in this analysis uses two relatively modern open-source models from Google DeepMind: Gemma 3 4B and Gemma 3 12B, sourced via Hugging Face. As usual, I sourced the corresponding pretrained residual stream SAEs from SAELens. I examined the residual stream at layers 17 and 24, for Gemma 3 4B and Gemma 3 12B, respectively, as they represent similar relative positions within each model’s total layer count.

To compress the model weights, I used the bitsandbytes library to compress each model’s weights into 8-bit and 4-bit format, representing common compression levels in open-weight models. The 8-bit compression utilized bitsandbytes’ LLM.int8() method and the 4-bit compression utilized bitsandbytes’ NF4 datatype. Both compression methodologies were applied as post-training quantization to the bf16 checkpoint (i.e., the uncompressed versions of each model) at load time.

Finally, I used NeelNanda/pile-10k, a Pile slice prepared for interpretability work, which I packed into fixed-length token chunks shuffled with a fixed seed for determinism and reproducibility. Since the SAEs used in this experiment were trained using activations resulting from this kind of text running through the models, the use of that text here is meant to provide a clean measurement of the SAEs' capabilities, relative to the compression methods undertaken.

Tests run

Model accuracy and SAE reconstruction were each computed for each model at every bit-width (bf16, 8-bit, and 4-bit) compression level. To further strengthen the methodologies employed and the resulting conclusions, I also undertook the following validation tests:

1. A sample-size sweep that establishes how many tokens are needed before the metrics stabilize.
2. A noise-floor test that determines whether a flat FVU reflects a genuine-but-survived perturbation or a change too small for the SAE to register.

Table 1: Tests performed 

Category

Test

Metric(s) reported

Question addressed

Primary

Model accuracy

Cross-entropy / perplexity

How much quantization degrades the model's next-token predictive performance?

Primary

SAE applicability

FVU (headline)

How faithfully the pretrained SAEs reconstruct the residual stream under quantization?

Validation

Sample-size stabilization

FVU snapshotted at increasing token counts

The token budget at which the metrics stop moving  (i.e., what sample is large enough to be stable?)

Validation

Noise floor

Activation perturbation ÷ SAE reconstruction residual

Is a flat FVU informative (the residual stream is genuinely perturbed but the SAE survives) or a change below the SAE's resolution?

IV. ResultsSummary of results

Table 2: Summary of results

Category

Test

Result

Takeaway

Primary

Model accuracy

Cross-entropy rise vs. bf16:

• 4B
  • 8-bit: +0.55%
  • 4-bit: +1.99%
• 12B:
  • 8-bit: -0.30%
  • 4-bit: +2.69%

Quantization degrades next-token performance only modestly (~2.0-2.7% at 4-bit). Slight improvement in the 12B model under 8-bit compression is likely noise.

Primary

SAE reconstruction

FVU by bit-width

• 4B:
  • bf16: 0.0035
  • 8-bit: 0.0038
  • 4-bit: 0.0038
• 12B:
  • bf16: 0.0042
  • 8-bit: 0.0041
  • 4-bit: 0.0044

FVU stays essentially flat across compression levels, so the pretrained SAEs reconstruct the residual stream about as well on the compressed models as on the originals.

Validation

Sample-size stabilization

FVU stabilizes by ~200k, with the 200k and 500k snapshots within 0.0001. Thus, 200k is the budget used for the headline results.

The chosen token budget sits comfortably in the stable regime, so the headline FVU/accuracy numbers aren't sample-size artifacts.

Validation

Noise floor

Perturbation ÷ SAE residual = ~1.1-1.9; verdict: flat FVU results are informative

Quantization genuinely perturbs the residual stream by an amount the SAE nonetheless absorbs. This means the flat FVU is a real result, not a change too small to detect.

Interpretation: Model accuracy

The first question explored by this analysis is how increasing levels of compression affect model performance. This was measured using cross-entropy and its exponential, perplexity - standard means of measuring a model’s next-token prediction accuracy.

The results from this experiment strongly suggest that the models are highly resilient to compression, demonstrating only modest declines in performance when subjected to 4-bit model weight quantization. More specifically, 8-bit compression yielded almost no change (under 0.6%) in cross-entropy, compared to the uncompressed (bf16) models. 4-bit compression showed still-modest but slightly greater degradation in performance (~2-2.7%).  

Figure 2: Model performance vs. compression level

These results are both expected and reassuring. With the use of weight-quantized models becoming the de facto choice in most open-source ML applications, modern models tend to quantize gracefully, without significant declines in performance. The likely reason is mechanical: quantization perturbs each weight only slightly, and that small, distributed error tends to wash out across the model's many parameters rather than compounding into a large output change. This echoes a broader theme from Part 3 in this series: that these models can, in some circumstances, absorb meaningful perturbations to their internal state with little change to their output.

Interpretation: SAE Applicability

The second question explored by this analysis is how increasing levels of compression affect the applicability of each model’s pretrained residual stream SAEs. This SAE applicability was approximated by assessing the SAEs’ ability to take the expanded (encoded) versions of each model’s vectors and use that information to reconstruct the vectors originally produced by the model (decoded). If the decoded vectors produced by the SAEs closely match those originally produced by the compressed model, this suggests that the SAEs are able to faithfully encode and decode the compressed model’s vectors - a prerequisite for useful feature identification and model interpretability. To measure this, I employed FVU, which measures the degree of difference between the SAE’s decoded vectors and those produced by the model.

The results of this experiment demonstrate that the SAEs’ ability to reconstruct the residual stream vectors was nearly unaffected by increasing levels of compression. For both Gemma 3 4B and Gemma 3 12B, FVU increased only ~0.0002–0.0003 at 4-bit compression (and was effectively flat or slightly lower at 8-bit compression).

Figure 3: FVU vs. compression level

To further strengthen these results and ensure that the low and stable FVU values observed were genuine reflections of the SAE’s ability to handle a model whose weights were changed (perturbed) by compression, I conducted a noise-floor test which compares how much compression affects a model’s vectors (the numerator) to the SAE’s reconstruction error on the uncompressed model. A ratio above 1 indicates that compression induced changes to the model’s vectors that were larger than the SAE’s precision and thus any failures to reconstruct the model’s residual stream would have been evident via increases in FVU. That those floor ratios ranged from ~1.10 to 1.87 confirms compression genuinely moved the residual stream by more than the SAE's own margin of error, so the stable FVU reflects the SAE absorbing a real perturbation rather than a change too small to detect.

Figure 4: Noise-floor testing

V. Implications for AI safety and areas for future researchImplications for AI safety

The experiments contained in this analysis demonstrate that at commonly used levels of model weight quantization, SAEs retain their ability to faithfully encode and decode the residual stream vectors produced by the models tested, as evidenced by their essentially constant FVUs even as the noise-floor test confirmed compression genuinely perturbed the residual stream. While this does not prove the interpretability of compressed models writ large, it is a prerequisite for the feature identification on which interpretability depends, and thus is cause for optimism in the ability to understand and control the compressed models that are increasingly the default in open-source applications.

For model developers and researchers, this means that the interpretability “safety tax” (the cost of building interpretability tooling) may be overstated. These experiments showed that SAEs trained on the uncompressed bf16 versions of Gemma 3 4B and Gemma 3 12B reconstructed those models' compressed variants about as well as the originals. This suggests that model developers need not train SAEs for every conceivable level of quantization, but instead can reuse the same SAEs developed for the base, uncompressed model, saving time and money while lowering the barrier for interpretability-related work.

For society generally, the broader benefits of these findings are clear: compression remains a viable and valuable tool that allows models to run more economically on more constrained hardware, while still preserving the tools needed to interpret those models and more reliably control their behavior. Put another way, these results are supportive of the conclusion that model controllability and hardware flexibility are not in tension.

Areas for future research

One clear area of further research would be to close the logical loop that allows for a clear declaration as to the effect of model weight quantization on interpretability. These experiments take an important step in that direction by confirming SAE ability to faithfully reconstruct a model’s residual stream. While that reconstruction is a prerequisite for useful feature identification, it is only part of the story. Only by actually comparing the features identified by SAEs in uncompressed vs. compressed models can one make definitive conclusions about the effect of compression on interpretability. This stands as a logical next step in continuation of this research.

Another logical extension would be into other forms of compression. This analysis covered post-training weight quantization, but that represents only one compression approach. Google DeepMind’s Gemma 4, for instance, ships quantization-aware-trained checkpoints and non-weight compression techniques such as KV-cache compression (e.g., TurboQuant), among other means of reducing the model’s hardware requirements. Whether SAE reconstruction holds up under these more varied and aggressive methods remains an open question.

Discuss