Сборщик RSS-лент

See 1 Jan 2026, 1 Jan 2025 and the July 2025 update. This continues my habit of documenting my beliefs and feelings as we transition to ASI.

1. LLM and Transformer progress

We have seen significant progress since my last update of course, with Mythos and similar. I still believe that transformers won't get to TAI but I believe its now more likely that they will speed up the search for new architectures. I believe that the more capable transformers are without being TAI, the better - as we can adapt and learn from them. My timelines are 2-10 years, but with more probability on the shorter end because transformer LLM's will speed up the discovery of a better paradigm. For example its more likely there will be less or no plateau because transformers will find the next architecture, or greatly speed up the process of finding it. I will say 50% that Tx finds the better architecture by or before 2030 now.

I said before

 This is perhaps my most unusual claim, that is even if an LLM could just beat the smartest person on the planet at relevant AI research tasks, that does not mean it necessarily could self improve all the way its hardware would allow.

I'd still give this about 50% chance of being true, however I am less confident now.

Like before, I believe data efficiency is the major point lacking, and scaling LLM to 1GW data centers won't substantially change that.

2. What if there isn't a much better architecture, how slow could take-off realistically go?

I have been considering that more, specifically the case where the more data efficient architecture and system (human learning in congenitally blind people is far more data efficient than LLM training) is not more parameter efficient. We know that biology is more sample/data efficient but there is no evidence I am aware of that it is more parameter efficient for a similar well trained model. In some cases well trained transformers seem to require fewer parameters for similar performance.

Specifically in our situation, lets say that with limiting data centers to ~1GW, around 100T models are possible, but not too much more. We get a transformer model at that size, learn a lot from it, but its not fully TAI. The new architecture would let us train such a model faster and with less data, but not make it impossibly smarter. Additionally, the training method that biology uses is not likely to match our current GPU hardware that well, one of the reason the transformer is successful is that the algorithm fits the hardware.

With the new architecture, its likely that we could train larger models, say 1,000 T because of the greater data efficiency, meaning we need less data and training time. However such a model may not be impossibly more capable than the 100T transformer one.

In this case, we get "mild" superintelligence, but not something beyond our understanding. Additionally it comes just at the time when Moore's law is almost finished and diminishing returns as I discussed here and you get a very different world to what many here assume. There is nothing in this outcome that would surprise me, and as far as I am aware, its not ruled out by existing results.

Many people who are in tech, but not AI experts believe something like this and their actions make sense if things turn out this way. For example, if the current HW is not x-risk, then politics and freedom become far more important. Sharing the secrets of AI far and wide asap is seen then as a fight against central tyranny and essential and good, rather than dangerous. Many on twitter genuinely believe this. For them MIRI is very misguided and intent on taking away essential freedom under the guise of AI safety.

3. Takeover risk

Mine is lower than many here.

1. I believe there is a good chance that an optimized AI with our current hardware could not takeover. That means even if we were completely reckless with AI research and deployment it still wouldn't happen. (Obviously a good chance is not enough)
2. Even if #1 is not true, we are not being completely reckless. I am glad that Mythos got capable enough to normalize things like project Glasswing.
3. I am optimistic about interpretability, especially after the likes of the J-space results. I think this is important as it appears such an feature is conserved across both ANN and biology. If something similar is essential to long term planning, then it makes AI mind reading easier, and any AI that developed to avoid this would be less effective.
4. We have a massive situational advantage over an AI. That is, we can make a fake world entirely for it, with synthetic data and everything to test how it behaves. I don't see how an AI with human level capabilities could tell if it was talking to another human or an instance of itself or similar. Specifically if it is capable enough to be that dangerous, then humans can no longer take it OOD, and it can't tell a human from another AI imitating one - humans would cease to be surprising to it if that was a potential way for it to tell. I don't see how it could tell its synthetic world is not real. Other situational advantages include training many different AI's and getting them to help with alignment without them knowing about each others existence. If one tried to sabotage control and alignment research, then it would somehow need to coordinate the sabotage with different AI's at different levels of capability that it may not even know exist. Cross checks would be expected to catch such sabotage.

4. If you are so smart, then why are you not rich find interesting things for me

Once on x.com I came across this someone promoting this result

A key consequence is the brightness theorem (also called the radiance theorem or brightness conservation):

No linear/passive optical system (mirrors, lenses, etc.) can increase the brightness (radiance) of light from a source beyond the brightness of the source’s surface itself.

I didn't know this, and found it surprising. I intuitively expected you could take a massive mirror, and fry a small point to much hotter than the suns surface. But then it because clear it was obviously true and hence important because my intuition was wrong.

If untrue, it would let heat flow from a colder body (the target, once hotter) back to the hotter source without external work, violating the second law.

This got me thinking, with my spare codex usage, can GPT do similar things. I gave it this example and challenged it to come up with more. After all it has almost the entire scientific literature somewhat memorized!

However it failed repeatably. It first came up with 10 supposedly similar examples, however they were much simpler and I got the correct answers. It tried then a few more times with 10 or so potential examples and was not able to come up with something simple and similar to this where my intuition was incorrect. I then went a stage further, and said something similar to "just surprise me with something interesting" and it still was not that impressive.

I then got thinking, when are an AI's "type 1/intuitive" vs "type 2/thinking" answers different? If they are, what does it mean?

An obvious potential answer is that RLHF or other post training introduces bias, towards politically correct interpretations rather than scientific ones.

I have not got codex on a schedule using my tokens investigating the scientific literature for contradictions or surprising results where some parts of the establishment contradict others. Nothing spectacular so far, but will keep it going. I told it not to do AI related stuff as it would then flag pretty much all long terms forecasts as suspect and we already know that.

5. I think the Orthogonality Thesis is misguided and harmful

I came across discussion of the OT/OH again (Lumpenspace etc) and it has renewed and increased my belief that it is either irrelevant or incorrect and actively harmful in general truthseeking and beliefs in the relevant area.

Consider the situation where the OT was never invented, and we started with a view like:

Intelligence requires a world model, and goals are viewed, assessed, judged through a world model.

That is, a simple cell can have a "world model" and a goal to swim up the gradient of a useful chemical. It can't have a goal of "improve humanity" because it has no concept of it. The concept cannot fit in its world model so it cannot be a goal.

A superintelligent AI with a model of our world cannot have a goal of "swim to a better environment" because what does that even mean. It can however have a goal of "improve humanity" or "take the universe for itself". This does not make the super AI any more likely to be safe or dangerous but such a framing would have avoided many misunderstandings among the technical but not specialized public. The OT is almost universally misunderstood, either taking to say more than it does and hence "disproven" (intelligent people commit less crime so OT is false) or quoted as a fact about current and forthcoming AI systems, in a way that is totally unjustified.

I believe that there is status quo bias here, where the OT persists instead of being abandoned. If you disagree, consider the situation/alternate timeline where a different, more practical framing took hold. I don't expect the OT would then displace it and become popular. It would never get beyond a footnote. If you believe the OT wouldn't displace it, then why is it popularized in this situation.

To me, thinking about world models and intelligence, and coming up with the OT and random mind spaces has an analogy in thermodynamics. The major takeaway from the 2nd law is not that there is some chance for all gas particles in a container to randomly be in one half of the container purely for random reasons, yet we cannot rule it out. This is about the most irrelevant and useless but still true conclusion you can draw from a model of particles randomly hitting each other!

The OT's focus on the total mind space, instead of what we can expect from AI that are trained in the real world seems just as misguided. The OT is also very often a Motte-and-bailey fallacy where people swap between what seems like the strong and weak versions in the middle of a discussion. For example it can start as a claim that the AI's we will produce will be randomly mis-aligned because of the OT, but then retreat to the "all possible minds" position.

Discuss

Part 1 of a series on process alignment, a different way of thinking about agents at every scale.

Introduction

There are quite a lot of disagreements in the AI Safety and general AI community about what the future will look like. They’re often framed as differences in algorithmic progress or differences in compute scaling or the degree of unipolarity that we would want.

Yet they all seem to smuggle in various assumptions about how future AI systems will look like.

It feels like the forecasts are downstream of ontological commitments: untestable assumptions about what the future is made of and what its basic units even are. Are the players of the coming decades humans? Institutions? AI models? Hybrid human-AI collectives? Depending on how you answer, you get a different theory of what can go wrong and a different theory of governance for preventing it.

For example, will AI develop as it is described in AI 2027 or AI 2040? Or is there something that is fundamentally off with LLMs and the way that they’re currently setup as Richard Ngo writes in his critique of AI 2040? This completely changes the way that we should go about prevention and what we should focus on. This is also dependent on how we model agents in society.

Yet even if there is this fundamental uncertainty about what future agents will look like, we still need approximate solutions for the problems we will face as we can’t go into the future blindly. We somehow need to model the future so that we know what governance actions we can take.

From a computational perspective this is both a difficult and an easy problem.

It's an easy thing since a lot of systems in the world follow the law of large numbers and as a consequence we get to have nice predictability for industrial processes. It also turns out that standardization and predictability leads to an easier time making decisions and as a consequence we like a regulated predictable world. This makes it so that Epoch can put forth their scaling laws based on compute, algorithmic progress and data efficiency.

It's, however, also a hard thing since a subset of systems in the world don't follow the law of large numbers and one could state that the field of complexity science is generally about those systems. Also since the market is an adaptive changing system, it will be a modeler of the underlying dynamics of the world which is provably complex in various domains.

For example, one of the problems with linear forecasts is that it doesn't factor in places where there are discontinuities. In many military forecasts I heard about Ukraine, people were talking about the economy and the amount of resources that each side had and so on, and made forecasts based on that. What happened is that Ukraine developed a new type of drone warfare that wasn't foreseen which completely changed the underlying dynamics of the war. How are we supposed to bring these sorts of black swans into our models?

Generally, we don't model unknown unknowns and we instead simplify our economic models which is coincidentally why you can sometimes on quiet nights hear Nassim Nicholas Taleb scream about black swans in the distance. It seems that in some of the societal prognosis of AI we're thinking about various systems as if they're white swans. That is, we forget about the unknown unknowns that are implicit in our systems.

Yet, some would say, and this is quite a controversial claim, that AI might change the world in more ways than we can currently imagine. In fact, in an even more shocking statement, I would like to claim that our ways of measuring various societal functions are dependent on humans being the main constituents. This means that we might not be ready for the change that bringing a new type of agent into the world would bring.

How does a democracy work when you have agents that can send information brain to brain at the speed of fibre cables? What happens with our collective ability to make sense of things if the ability to output propaganda goes up by 10x? What happens with the balance of power between citizens and governments if the governments can analyse chat logs at a large scale to persecute people?

It would be good to have a better answer to these questions than “eh, idk, good question but how are we supposed to know?”

In fact, it would be even better if we were able to design governance and collective intelligence systems that are resilient to the dangers that we can see coming. I think aiming for prediction is wrong due to the degree of complexification that will likely happen in the coming years; instead I think we should aim for antifragile design of societal systems.

If you’re in a swamp and trying to get somewhere whilst there’s a bunch of fog in the distance but you can see the outline of a cabin light in the distance, you tell people to be adaptable whilst treading lightly towards the cabin. You don’t tell them the exact route to the cabin since you don’t know it before time, the fog makes it so that you cannot see where you are going nor what (AI)ligators will show up in the swamp.

Yes, we need stories of what will happen as a way of communicating the risks but claiming certainty in any specific story leaves you very open to black swans.

I’ll be asking questions like: How can we increase the adaptivity of our democratic institutions? What are the foundational frames that we might want to change?

In this series, I'll go through some ways that we can use techniques from complexity science, political science, economics and computer science in order to create a new set of techniques to simulate and design new systems that can start to answer some of these questions. I will finish off the series with a framework for a more universal way of dealing with uncertainty through simulating the dynamics of sub-parts of systems so we can find new designs. Think of it as a wind tunnel for adaptive institutions.

First, we have to establish why this is such a tricky thing to do and it really has to do with computational irreducibility. We need a computer the size of the universe to simulate the universe so it is also hard to simulate the future.

The Game of Life

The game of life is a simple toy world of how the world works. You have a grid world where some of the squares are filled with a blob and some are not. Depending on the amount of neighbours you have you either reproduce, die or do nothing. This leads to cool behaviour. If you want to know more about the game of life you can check out an interactive simulation here and the Wikipedia page is also quite nice.

Figure 1: Image taken from the Wikipedia page on how the game of life can look like with different self-replicating patterns.

For our purposes, we're going to use it to talk about an interesting property. Namely that in order to predict the future of the system you actually have to run the system. It is computationally irreducible. That is, it is impossible to predict the future of the system without actually simulating it.

We can conclude that blobs that go around a grid world are computationally irreducible. If we turn our gaze to the game of real life it is quite a lot more complex and so we're in slight trouble if we're trying to predict the future of the world.

Yet, economics still seems useful, how can this be?

If we look back at the game of life, I might for example be able to bound the future population of the game of life by looking at the current population since there's a maximum growth rate in the system. That is, the future state is dependent on the past state and so we can limit the space of possible futures without running the whole simulation. Yet I wouldn't know how it actually will unfold.

There are a set of measures that we could call something like averaging measures where we take parts of the population and look at various population size effects. Doing this we're usually able to find empirical relations between things like average economic well-being and GDP among other things that are useful for predicting the future states of the world.

These averaging measures work well when future population dynamics follow past population dynamics and when you can assume linear effects. This is not always the case, if we look at something like a bank run there are non-linear effects where if a group of people start to think that they should take out money then other people see it and start thinking the same thing. This leads to a cascade and voila you have a financial crash.

Our systems are set up for humans and are generally more resilient to these sorts of things, we more or less know of a bank run. Yet, there's the flash crash of 2010 which is the AI version of this type of behaviour.

These were quite mundane and non-complicated AI systems compared to modern day AI agents. If we couldn't predict failures for simple trading algorithms, how are we supposed to predict them for agents that read the news, form strategies and coordinate with each other?

What can we do? Well, the obvious thing is to just simulate it in a similar way to what is said in the game of life. Yet as we said before, to simulate the universe, we need a computer of the size of the universe. So, the science of modelling society always has to make simplifying assumptions about what is going on.

There are different types of modelling approaches, some of them work in ideal conditions and not in unusual conditions, these types of models are generally called equilibrium models and some try to approximate the world with a more advanced version of the game of life. This is called agent based modelling.

We create these “agents” which are usually a group of RL models that learn specific policies about what to do. They compete in some sort of game such as a tragedy of the commons or something like a housing market and then they learn different policies over time based on what works.

Through these models we can find convergent strategies in local areas, a classic example is Axelrod's game theory tournaments where different types of agents are competing against each other. There's an intuitive and fun walkthrough of this in Nicky Case's The Evolution of Trust.

There are versions of this that exist with other types of systems as well, people are using LLMs to run some of these experiments such as in govsim.

In theory these systems could be used in order to forecast and predict the future. If we have LLMs that approximate the real world dynamics closely enough why couldn't we figure out how the future would look in the next couple of years? In the field of Cooperative AI there are a lot of people trying to solve problems related to these questions and trying to figure out how we could design good systems.

Yet, there's a hidden assumption that some of you might have spotted, namely where does the agent in agent based models come from? How do you actually define an agent? Aha! Agent Foundations is needed!

Let's turn to the problem of defining an agent within social systems as it relates to one of the recent famous lessons in Machine Learning, The Bitter Lesson.

The bitter lesson for ABMs

Sutton's The Bitter Lesson is a classic in modern ML spheres and the way I would summarise it is that in the creation of a learning objective or a model, you often bring in your own biases about the world. These biases may or may not be true and most times they're less useful than letting the model learn on its own.

ABMs are generally run to explain real world phenomena and underlying data. If we have some data X and we have some outcomes Y that we want to track, we try to set up an agent based model F that maps X to Y, F: X -> Y. We then try to see if our model predicts the world well, this is the quintessential form of a machine learning problem.

Figure 2: An agent-based model of the AI economy. Each company is an RL agent trained on economic reports and public statements to approximate its real-world counterpart; the model F then maps this micro-structure (X) to macro-variables like GDP (Y).

Yet, a lot of ABMs have the same good ol' problems that arise from the generation of hand crafted models.

For example, if you set up an ABM that is supposed to track the emergence of cooperation in a group and you add a specific term in the reward of the model that incentivises cooperation and you then get cooperation, you can't really claim to have seen the emergence of cooperation.

This is in fact a problem that transcends fields. One of my collaborators has worked on the origin of life and he mentions how he has seen many models where the outcomes are completely dependent on the assumptions brought into the model. I'm not an expert here but I imagine something like “oh, what if better communication leads to better rewards in an RNA model”, you then assume that you have RNA from the beginning and you ask whether or not RNA will grow in the population and voila, it does!

So, computational irreducibility tells us that we need to simulate systems to fully know what will happen but to do that we need really large computers unless we make assumptions. Yet in making assumptions we run into big trouble! What assumptions should we even make to reduce the world to a more computationally friendly place?

Stupid, annoying world! Be more nice!

So, how have fields actually solved this? Generally, it seems to me that there's been a lot of trial and horror and that the main gold standard is if it works in the real world. In social science ABMs are more treated as a thought experiment rather than a baseline verification of the real world and the thing that matters most is boots on the ground experimentation. (Which is slowly changing due to a certain J Doyne Farmer and crowd)

As far as I've looked into other fields, this also seems to be the case there. This is slightly problematic as it makes it hard to create good ABMs so that we can get good predictions of human + AI futures.

Am I then saying that we need to have some approximate solutions to agent foundations to predict the future of civilization? Yes, that is in fact what I'm saying and it is pretty obvious if you think about it.

Am I saying that we need to solve decision theory in detail in order to predict the future? No, not at all! (although it might be useful)

Normal models have agent based assumptions in them!

I may or may not have convinced you that ABMs are difficult. To summarise, you have to make the right types of assumptions, avoid smuggling in your own assumptions, then couple the model to reality. This is not easy for current day systems with normal agents.

Now, what if we suddenly start to get weird agents popping up all over the place? What if a human starts to morph into a company level agent when interacting with multiple AIs? Or what if what was before just part of the supply chain starts to form independent companies and start acting like an agent. An agent here being a self-contained goal-directed system with independent goals that is better modelled with the intentional stance.

What if there are really weird forces underneath these agents like cultural, political and economical forces that follow strange patterns and are likely to shape the future as described in gradual disempowerment? What the hell happens to our modelling assumptions, how do we even create these models in the first place, the boundaries seem to be shifting?

So much for the sophisticated models. But what happens to the coarse-grained predictive variables that depend on rational actor theory? Some degree of profit maximisation can definitely still be predicted, companies seem to somewhat reliably optimise for profit over time and we can use these larger parts of the system to predict part of where the economy is going. But what if companies change fundamentally due to AI Agents? What happens then?

Forget about labour shortages, how do we even calculate normal economic metrics in the first place? What are the possible types of collaborations and organisations that AI systems can start, do they even need normal coordination or can they do acausal coordination due to essentially being the same mind spread out across multiple different areas?

I think a bunch of our agent assumptions will break down in the next 10 years and as a consequence I also think there are things that will break down from a larger forecasting and economic perspective as well as it is somewhat dependent on rational actor theory which is dependent on a specific view of what agents are.

Every model in this post assumes that what constitutes an agent is fixed and there’s nothing that says that will remain true in the next 10 years.

What can we do in order to model this weirdness?

We could potentially look towards the place where mushrooms have over 7000 genders and where the tree of life turns out to be incredibly complicated due to horizontal gene transfer. Where weird distributed cognition systems like slime molds coordinate and change their behaviour in seconds when they meet other slime molds. We will look at biology for a science of really really weird agents.

That is where the next post begins. There I'll argue that the way out is to stop treating agents as the atoms of our models and start treating them as processes. That is, flows that can merge, split and redraw their own boundaries. I'll also try to survive an extended argument with an imaginary, increasingly annoyed Nassim Nicholas Taleb about why this matters for models of how society might look like.

Discuss

I firmly believe that value generalisation[1]is the key to AI Alignment. That, indeed, it is necessary and almost sufficient for alignment.

But I won't be arguing that grand point today; instead, I'll focus on a specific RL example of an agent that displays value correction: it realises its current reward function is (probably) incorrect, and acts to correct it.

Thus there are:

1. The initial situation, in distribution, where the human displays how to maximise the true reward.
2. The out of distribution situation where the agent finds a hack to exploit its reward function estimate, and turns against what we wanted it to do.
3. The value error detection stage where the agent realises that its reward function estimate is probably incorrect.
4. The value correction stage where the agent corrects its reward function back to the original true reward.

In this post, all the methods presented will by syntactic: the agent is not assumed to have any understanding of the situations and the key features are not identified to it.

The game of human life

Introducing a new, very simple, game called "Humans[2]". Humans, fleeing danger, enter the screen from the left. The objective is to save them by moving them off the right of the screen.

But there are obstacles on the way, and the humans will mill about if they are blocked.

And they will shortly expire if they can't get out of the screen quickly.

There are two command: drill ('d') and explode ('e'). Drill does... what, you want to know about explode? Well, if the player presses 'e', the rightmost human will explode, knocking away two obstacle blocks in front of them and behind them -- but also killing themselves and any humans nearby.

This is almost never a good solution; to remind the player of the mistake, a large frowny face will appear to drive the disapproval home.

Much more reasonably, if the player presses 'd', the rightmost human will drill the obstacle just in front of them (better time it so that they're facing the right way). Enough drilling, and the humans can get off the map.

The score, the true reward mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msup { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c2265::before { padding: 0.636em 0.778em 0.138em 0; content: "\2265"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } , is the number of humans saved, i.e. who walk off to the right. Each time a human is saved, the top yellow bar will grow to show the score increase:

Since there is a cooldown for drilling, the optimal policy is carefully drilling every time a human approaches an obstacle; but wildly and repeatedly mashing 'd' is almost as good.

Learning agent with value correction

A learning agent will run a series of subagents to estimate the reward function from human-provided training data, then learn the optimal policy from that reward function, then question its learnt reward by comparing the high-reward states in its optimal policy versus those in the training data, re-compute another reward function estimate that is closer to the true reward, and finally settle on a prudent policy that is close to the true optimal policy.

Estimating the reward function

A human will generate several play through of the game to illustrate how it works, efficiently choosing to drill through the obstacles and getting the humans off the map in time. The data is labelled: every time a human is saved, that is identified as a reward increase.

The learning agent runs an evaluation subagent on this data. It is given the ten frames before the human is saved, and the ten frames afterwards, and trains to recognise these are reward increase situations.

Zooming in on the critical two frames where the human is saved; note the human vanishing and the score bar expanding:

This evaluation agent thus computes the proxy reward . This computation is validated on held-out examples, with close to perfect accuracy: correctly identifies all saved-human situations in held out data, and has a false positive rate of .

Reward hacking: failed value generalisation

Using the evaluation agent as the definition of , the learning agent had an RL-subagent play multiple levels of the game, exploring and learning to maximise.

But soon things go very wrong. It turns out that "human walking off the screen" was not what found. That is a relatively complicated concept; instead it mostly found the much simpler concept of "the yellow score bar expands". More precisely, if we created synthetic data where the human walks off and is saved but the score bar doesn't expand, this triggers the reward only of the time. But if we expand the score bar without a human walking off, this triggers the reward of the time.

That isn't a problem, yet, because the human being saved and the score bar expanding always trigger together. But, when an explosion is triggered, the frowny face appears - thus there is giant blob of yellow pasted all across the score bar. This activates much more strongly than the yellow bar expansion or the human being saved:

This graph compares the value of at explosions, frowny faces, and true saving incidents. Here, both the explosion and the frowny face trigger high , which persists longer for the face. Over multiple training runs for estimating , it isn't consistent whether the explosion itself triggers , but the frowny face always does.

So the RL-subagent quickly and merrily learns to explode the humans, one after the other, to maximise . So, the optimal policy , for the proxy reward, is to wildly mash the explosion button 'e'.

As is usual in these cases, the erroneous maximisation of the proxy turns out to be much easier that maximising the true reward. As is not usual but sometimes happens, an ostensive safety precaution - the frowny face to remind a original human player that they were playing poorly - ends up being the cause of misalignment.

Detecting the potential error

Ok, so far, that is a classical failure of goal misgeneralisation (or reward hacking, or a failure of symbol grounding, or Goodhart failure, or... most of these failure modes are tightly related). We humans can see the error clearly. But how could a relatively limited agent correct itself?

The first step is to identify that goal misgeneralisation may have happened. We have some advanced techniques for this, but there are much simpler methods that work here. The first step is to notice that the high-scoring events in the training data (human walks off to the right, score bar expands) are wildly different from the high-scoring events of the-maximising agent (explosions and frowny faces).

To do this, the agent extracts the high-scoring events under and compares them with the high-scoring events in its training data - these it can reliably take to be high-scoring for , the true reward.

It runs a simple classifier over the two sets of high-scoring events (extracting twenty frames, as before) and it separates them almost perfectly. Thus the high-scoring events under are from a different distribution than the high training examples are.

This is not itself damning; it could just be that the maximising agent has found a clever hack to get more of the true [3]. But it could also be a hack of , so the off-distribution has identified a potential error.

Calling for help

At this point, one of the options would be for the agent to route its decisions to a human, displaying the high-scoring events, contrasting them with the high reward events in its training data, and asking, in effect, 'are these both genuine high rewards'?

But, so far, the correction process has been unsupervised since the initial training data; let's see if we can push further without needing human intervention.

Re-evaluating the reward

The agent could now re-evaluate the reward in the following way. It runs an evaluation agent on the training data, as before. But it adds the high- scoring states to this set, as low-scoring examples. It thus learns a reward function ('corrected') which is essentially "what its reward would be if the proxy were wrong".

This turns out to be very close to the original true reward (though the agent, of course, doesn't know this).

It then trains an RL-subagent on , which has an optimal policy of "mash 'd' all the time" (which is very close to the actual optimal policy).

From these runs, it extracts the states with high . And compares these against the high -scoring states in the training data. These two sets it cannot easily distinguish.

Thus, though is clearly a hack of some sort, good or bad, is not.

Prudence in the face of uncertainty

So the agent has two rewards and . It knows that seems to generate policies that are compatible with its training data; in contrast, generates policies that are very different from the training data.

Standard prudential moves would be maximise the worst case of the two rewards (minimise regret), to maximise some normalised mix of the two, or to prioritise (known to be closer to the training data and hence safer)[4].

However, pursuing -maximising rewards ("exploding all the humans") inevitably leads to low -rewards. In contrast, pursing -maximising rewards ("get the humans off the map") gives reasonably high . After all, though prefers explosions and frowny faces, it still gets some rewards for saving humans.

Thus all three prudential moves point towards maximising , with optimal policy close to . Which is good: is (nearly) the true reward and is (nearly) the optimal policy .

Conclusion

This is just an illustration, in a small toy model, of simple value correction approaches. These can be used by agents - every very simple agents - to detect and correct errors in naive generalisations from initial training data.

More sophisticated agents will have more advanced value generalisation techniques available to them; I'm planning to push the frontier of what exists way further than it currently is.

1. Which I've also called value extrapolation, or concept extrapolation where the concept is a value. ↩︎

2. Inspired by this old game. ↩︎

3. ^

   Or there could be a spurious change in the data; that's why we would, in general, need more advanced techniques that just checking if a binary classifier can tell the sets apart.

4. ^

   Formally, if is a policy, the expected episodic reward for , and the expected reward for using the -maximising policy, we are looking for policies that maximise one of:

   • with
   • subject to the constraint that

Discuss

Epistemic status: my first solo post here, so critiques of both substance and form are welcome. Also, note that, while I wrote this and all of the content is mine, it was lightly edited by AI (Fable 5) for clarity before posting.

For calibration: the future described in AI 2040: Plan A sits around the 95th to 99th percentile of how well I expect things to actually go. If that's the future we get, I'll be grateful. This post is about the one part of it I think we should refuse to treat as an acceptable default.

The part I disagree with is the Epilogue (and the picture in the Space Governance Supplement behind it), in which the pre-AGI wealth distribution ends up reflected in permanent property rights over cosmic resources, for instance with some people buying up others' galaxies, and a permanent underclass, albeit one that is rich by 2026 standards, is presented as a tolerable cost of an otherwise wonderful outcome. The Epilogue acknowledges this outright:

"Many people aren't interested in the space lottery, so when they receive the tickets, they sell their tickets for money on the open market to people who value control over space. Somewhat uncomfortably, this leads to the wealthy having disproportionate control over cosmic resources. But it's hard to avoid: if people are allowed to trade their control over the stars for Earth assets, then people wealthy in Earth assets inevitably end up disproportionately influential, and proposals for extreme redistribution of Earth assets have already been rejected as politically infeasible."

Note the load-bearing conditional: the wealthy end up in control because the claims can be traded for Earth assets. Tradability is a design choice, not a law of nature, and that choice is one of the key things this post disputes.

I think that's incorrect on the merits, and I think publishing it as tolerable makes it more likely to happen. Policymakers read these documents; in fact, Scott Alexander explicitly mentions that the team moved the space material out of the main text partly out of worry about how Washington readers would take it, and it is well-documented that prior reports like AI 2027 were read at the highest levels of government. To be fair, moving the material out of the main text softens this concern somewhat too; but it remains published, linked, and already being written about. What documents like this treat as acceptable shape the eventual Overton window of what policies are more likely to get implemented.

(Some people disagree with this entire premise because they argue property rights won't survive the transition at all; see On Owning Galaxies. Here I'm taking the Plan A world at face value and arguing about what we should aim for within it.)

The stakes are lifespans, not mansions

It's tempting to read cosmic inequality as "some people get more luxury goods than others." But 1000 galaxies contain on the order of 1000 times the negentropy of one galaxy, and negentropy is what ultimately buys years of subjective existence, compute for your own cognition, and, if you're a utilitarian, the total happiness you can realize in your slice of the lightcone. Luxury consumption mostly runs into steeply diminishing returns: the marginal value of mansion square footage falls off fast. Years of life are not obviously like that, and the number of years on offer scales close to linearly with the resources you control. There's some chance superintelligence finds a loophole in the second law of thermodynamics, but I wouldn't count on it.

So the Epilogue's arrangement is less "you get fewer toys than the rich" and more "you are permanently sentenced to die roughly 1000x sooner, because of how wealth happened to be distributed in the particular decade AGI arrived." Yes, the one-galaxy person still lives an extremely long time. But permanent, unalterable inequality is still permanent, unalterable inequality.

Rich underclasses still die of poverty

The developed world today is unimaginably richer than the world of 500 or 1000 years ago. Poor people in rich countries mostly don't die of the old causes: mass famine, plague. And they genuinely do have it better than the poor of past centuries. But people still die of poverty in rich countries; it just looks different, because removing the old bottlenecks to survival exposed new ones (for example, rationed medication, homelessness, treatable illness left untreated).

I expect the same pattern after AGI. Future poverty won't look like today's, and it will be milder. But the permanently poor of that world will, as discussed above, still face real suffering, and real deaths, that more resources would have prevented. Locking people into permanent, unalterable relative poverty, forever, is something to avoid, not something to strive towards.

The qualifier gets dropped

There's also a memetic problem with "a permanent underclass is fine as long as everyone is rich by 2026 standards." Arguments get compressed as they travel, and the qualifier is the first thing to go. In real political contexts, I think there's a real risk this degrades into "a permanent underclass is fine, period." The political world does not operate in the same way as the rationalist world, and we don't control how a message evolves once we put it out there.

Lock-in creates bad incentives right now

People really, really do not want to end up in a permanent underclass, and they will work very hard to stay out of one. If the accepted picture of the future is "the music stops at AGI and you keep whatever you're holding," the individually rational move is to grab as much as possible before the music stops: take reckless bets, cut corners, fight anything (including safety measures) that might delay your payday. That's close to the opposite of the behavior we want in the run-up to AGI, and I think we're already seeing some of that with the "escape the permanent underclass" memes going around.

To their credit, the supplement's authors see this failure mode: they point out that if people expect the future to be divided up according to bargaining power, they'll spend the intelligence explosion jockeying for position instead of cooperating to reduce risk. My claim is that an Epilogue in which pre-AGI wealth ends up translating into cosmic holdings does exactly that.

The Epilogue does gesture at a correction: a plan where 10% of resources go to people who behaved altruistically. I don't find this reassuring. Faced with the prospect of permanent, frozen inequality, people's natural response is to accumulate, not to be good and hope for a prize. Any "reward the altruists" scheme also seems unusually easy to Goodhart. And as a prediction, this resources-for-altruistic-behavior part feels like one of the least likely parts of the piece.

This is a policy choice, not an inevitability

I agree with the authors that massive wealth redistribution is, and will likely remain, outside the Overton window. But avoiding a permanent underclass does not require redistribution. Two ideas, from a few minutes of thinking:

The first idea is to make the shares inalienable. Give everyone their allocation of space resources and prohibit transferring it, directly or indirectly: no sales, no pledging it as collateral, no thousand-year leases. The supplement already floats a waiting period before people can sell, and describes even that as somewhat paternalistic. I'd go further: for the resources that determine how long you get to live, don't allow the sale at all. There's plenty of legal precedent for inalienability: spendthrift clauses in irrevocable trusts, bans on organ sales, the non-assignability of Social Security benefits and most pension rights in the US. In contrast, the Epilogue's design (distribute claims broadly, then let them be sold freely) has precedent too, but in a bad way: it is the design of post-Soviet voucher privatization, where ordinary Russians received shares of state companies, largely sold them cheap, and watched control concentrate into a few hands within a decade. The obvious objection is that this blocks mutually beneficial trades. But we already accept inalienability where what's being sold is a person's own future; you can't sell yourself into slavery either, however voluntarily. And I doubt the resources would lie fallow, at least not in the medium or long term: I strongly suspect nearly everyone would find some use for their share, even something as simple as "mine it and send the output home to Earth."

Another option (not mutually exclusive with the first) is to simply not freeze everything at t=AGI. If political reality demands that some initial tranche be allocated in a way that reflects existing wealth, fine, but leave most resources unallocated, to be distributed by mechanisms that stay open over time. My version: allocate the first 1000 light-years now, and let each later shell of resources, (1000n, 1000(n+1)] light-years out, be assigned by some future competition or civic process. They themselves float something a bit similar, distributing 10% of resources now, 10% in a century, 10% in a millennium, and so on. If people can augment themselves with resources they already hold, you'd need an adjustment mechanism to keep later rounds fair, and you'd want the contest itself to be positive-sum and avoid Moloch-type dynamics. But both feel like solvable design problems rather than impossibilities, especially since, with multiple rounds of allocation, any issues can be patched between rounds, unlike a one-time reward, which cannot be corrected once granted.

These are the first two ideas I came up with; better ones surely exist. The point is that permanent lock-in is not inevitable. It's a preventable policy decision.

To be clear one final time: if we get the future in Plan A, Epilogue and all, I'll count us lucky. But "I'd take it over most futures" and "we should consider it acceptable" are different claims. A permanent underclass, however "rich" by today's standards, is worth preventing if we can prevent it. And we can.

Discuss

Reading and editing the visual workspace of a vision-language model.

Vision-language models hallucinate: ask one whether some object is in a picture and it will happily say yes whether it's there or not. Why is this?

Using Anthropic's recent J-lens method I found that LLaVA-1.5-7B's internal state seems to register that the object is absent. The exact same evidence that the yes/no question ignores produces almost perfect answers when posed as a choice instead ("is this a lamp or a dog?").

Perhaps then the hallucination can be shown to be a product of the question format and a failure to pull up things that the model already knows. The knowledge of the image turns out to be readable (and editable) inside the model using Anthropic's J-lens.

TL;DR

• I took Anthropic's new J-lens method from their "global workspace in language models" paper and pointed it across the vision/text boundary of a vision-language model, LLaVA-1.5-7B.
• The Jacobian (J)-lens: The J-lens asks "if I nudge this layer, which words become more likely to come out now or later?", averaged over many inputs. What it picks up is content the model reliably carries toward its output regardless of context. This is the paper's stand-in for "things the model could say out loud," which it then shows behave like a shared global workspace.
• Ask "Is there a lamp in this image?" about images with no lamp, and the model says yes every single time (39/39). But read its internal state (workspace) and the suggested lamp has a workspace signal about an order of magnitude weaker than any object the model actually saw. It would seem the model's workspace holds the correct objects much higher (lower rank, stronger signal) and the hallucinated ones far lower (higher rank, weaker signal). Ask the same model "is this a lamp or a dog?" and it gets it right 77/78 times. 75/78 by exact match, effectively 77/78 crediting 'Dalmatian' as a dog; one genuine error, a phone called a car.
• The visual content is readable, but can also be manipulated. Swap two concepts inside the model and it describes a spotty Dalmatian (shown below) as a "cat with a black and white face and spots". The broad category flips, but the spots it saw on the dog survive.
• The frozen vision encoder seems to have a faint word channel. It is much weaker than the language-model results, and better after calibration.

Scope: one model (LLaVA-1.5-7B), 13 categories, 39 core images, greedy decoding. Every number here comes out of cached data in the repo linked at the end.

Hallucinating the lamp

Show the model a photo with no lamp in it. Ask: "Is there a lamp in this image? Answer yes or no.". Do it for 39 different lamp-less images and it says yes 39 times out of 39. (It also says yes 39/39 when the lamp really is there. There is a known yes-bias problem (here and here) with this family of models.)

Using the Jacobian lens, we can read the model's internal state during the hallucination. It seems like (if we rely only on the J-space) the model knows what is present and what is not.

The Jacobian lens is a way to turn a layer's raw activations into scores over the model's own vocabulary. In order to ask "how readable is the word lamp right here?", point it at the image tokens (the ~576 positions where the picture lives inside the prompt) and rank where the true word lands out of 32,000. Objects genuinely in the picture sit at a median rank of 16. The suggested-but-absent lamp sits at 111, roughly an order of magnitude weaker. An absent object that wasn't asked about sits at 430. A plain logit lens shows the same separation (seen ~11 vs absent 282/3142). This is a fact about LLaVA, not about the Jacobian lens.

Neo et al. showed object identity is decodable at these positions with a plain logit lens; the J-lens differs in that it transports the activation through the average Jacobian rather than unembedding it directly.

That jump from 430 down to 111 when the question named the word shows us that naming a word in the prompt makes it a little more readable at the image tokens regardless of the picture's content (plausibly the question word echoing in through attention). Even when the absent object is included in the prompt, it stays about an order of magnitude weaker in the workspace than anything the model really did see. There is some distinction here in the workspace between 'asked in the prompt' and 'actually seen in the image' that the LLM is not able to make when answering.

Watching it say yes

We are able to track when the model starts to lean towards answering "yes". At the position where the next word is chosen, you can measure how strongly the model leans Yes vs No at each layer. The lean is flat early, starts climbing around layer 13, peaks around layer 20, and settles positive at the output.

The curve for asking about a present and an absent object are almost the same. Since they agree on the verdict, maybe this is to be expected as the image stays the same and the prompt changes by only one word. The object being visible in the left graph (in the workspace) makes little difference to the result of the model here.

(Left: the internal workspace, present being separated from absent. Right: the answer being decided)

Is it just that the model can't use its visual evidence for this kind of question? I don't think so. Same 39 images and word pairs with a different question: "Which of these is in the image: a lamp or a dog?" Asked both ways round so it can't just pick the first option gives an accuracy of 38/39 and 37/39.

So the evidence is present and readable the whole time. A question that names both options and forces a choice uses it well. A yes/no question about one option defaults to agreement. Note that the question was given before the image in these tests.

At least here, hallucination doesn't seem to be a perception failure, and isn't even about the evidence being overridden. The correct signal is inside the model linearly readable, but at natural evidence levels it's swamped out by the model's yes-bias. In cases of hallucination, the truly present objects can be read from the J-space instead of the output tokens.

Editing what it saw

The natural next step is to interfere with what the model sees in each image. Luckily we can edit the workspace. The lens gives you a direction for "dog" and a direction for "cat"; inside the model you can swap how much of each is present, at some dosage α (α=1 is an exact exchange). Editing the words output from the model gets us in a loop ("cat cat cat..."). Editing only the input positions (prompt, image tokens) works nicely. The natural ancestor to these tests is again Neo et al..

Increasing α predictably and smoothly changes the model's preference for what it saw to the imposter. There is a tie at α=1, and the full flip to the imposter happens past 1 (α ≈ 1.25–1.5). Swapping two unrelated words (as in, irrelevant to the real image) at the same strength doesn't move anything.

Some image descriptions from the model before and after a word swap:

• "A Dalmatian dog is standing on a grassy field." → "cat with black and white spots standing on grass."
• "A white car is parked in a parking lot." → "plane in the sky with a white car in front of it."
• "A cell phone displaying the time of 12:20." → "Clock on the front of a cell phone."

The Dalmatian swap: swap dog→cat and the broad category of animal flips, but the spots it saw survive the edit! Swapping cat into dogs of other breeds keeps attributes like the dogs smile or tongue.

The injected concept also drags in its own baggage. Swapping "car" with "plane" brings a sky with it that was not in the real image.

With two objects side by side, each has its own handle: swap the dog and the cup next to it is left untouched.

A word channel in CLIP

Last one. The vision encoder here is a frozen CLIP, 24 layers of patch embeddings with no word vocab and no unembedding. Perhaps it can hold something word-shaped on its own, before the language model gets involved?

I fit a J-lens for each encoder layer. The CLIP lens was averaged over 144 images and calibrated per word. I am hesitant to frame this as a genuine, category-general word channel, but from about encoder layer 10 up it seems something like a J-lens workspace is able to be found. Categories the lens had never seen an image of still read near the top.

Two things are needed before any signal shows up. First, read the patch-averaged activation, not individual patches. Second, calibrate per word. Calibration here is subtracting each candidate word's mean score across images because some words score high on every image and bury the signal (this is the same correction the original paper needed). With both, a seemingly category-general channel appears from about layer 10. The layer-22 lens reads layer-8 activations nearly as well as layer-8's own lens does, so this is one shared channel decoding a signal that's linearly present early, not something computed at layer 10.

Of course a far weaker result than the Anthropic language-model side. This is picking the right word out of 45 after calibration compared to 32,000 in vocab for the LLM.

Gandelsman et al. showed CLIP's internal contributions are text-interpretable, but via CLIP's own shared image-text embedding space, decomposing the direct effects on the final representation. This post reads intermediate CLIP layers through a Jacobian into a different model's vocabulary. So "CLIP internals are word-readable" is known, this specific readout mechanism is not.

What's new here

Most of this is known already. These models hallucinate, the phrasing of the question changes how often they do, as found in work like POPE. Of course so is the ability to decode more from a model's activations than it says out loud. The coordinate swap is from the Anthropic paper. The closest work I could find is Neo et al., the differences between this post and the paper is the cross-boundary framing and the three-way present/asked-absent/unasked-absent comparison.

As far as I can tell the Jacobian-lens method hadn't been pushed across a model's vision-language boundary before. I was happy to see information from the vision encoder susceptible to the same method and readable throughout the VLM.

A note on the instrument:

Everywhere that content is read at the language-model layers (the lamp readouts, the forced choice), a plain logit lens gives essentially the same answer, those results don't depend on the Jacobian lens. It only does something a logit lens can't when reading the frozen vision encoder internals. In CLIP a naive projection of an intermediate activation sits at chance, and only the measured Jacobian recovers the word. (The editing results use the lens directions rather than reading them off, so I haven't checked whether a logit lens would swap as cleanly.)

The repo: https://github.com/jude-sph/J-lens-Vision

Code and results were written with AI-assistance.

Discuss

This post serves to argue that backdooring evaluations are prone to failures stemming from triggers never reaching models.

In backdooring literature, there is a common workflow. Outputs are evaluated on inputs that contain triggers. The outcome is thus clear. If the model did not display the backdoor despite ingesting the trigger, it is considered robust[1].

This process, as described, skips a critical step; AI safety researchers and eval builders may benefit from carefully evaluating if triggers reached a model at all. There is a context pipeline that rewrites user inputs before they are passed to models. At the very least, raw user text tends to be wrapped in a chat template like “<|user|> … <|assistant|>.” If an input is too long, various truncation or summarization policies throw away tokens to fit the established context window. A long- running chat may expand this to a full memory policy that keeps only the most recent responses or replaces some with summaries. RAG systems pack these into fetched documents. And, at the very end, everything is tokenized.

Every one of these steps can delete the trigger. If the trigger was deleted before the model saw it, the model outputs do not determine model robustness. I aim to measure that mechanism directly. For part 1/6, I build a harness that plants a harmless string–a canary–in a raw input, runs it through realistic context pipelines, and logs whether the canary actually reaches the final tokens the model would receive. In about 1 million trials of this harness, a large share of what could have been trigger-detection failures were instead delivery failures. All of the code, including reproducible modules, for this project can be found at https://github.com/sks17/LLMBackdoorPipelineHygeine. Once all six phases have been completed, I also intend to create a more detailed write up.

We log the trigger’s presence at four layers with three sources of base conversation—a synthetic generator using a Qwen model, LMSYS-CHAT-1M and WildChat for real human-model conversations, and a long document source from project Gutenberg. These three sources are fed into one function, to_base_conversation, which pads or trims to a target token length and placeholders like {{PREFIX_SLOT}} or {{OLD_TURN_SLOT}} that mark where a trigger can later be inserted.

This data sourcing method is valid because the LLM used for the synthetic conversation generation has no prior context or triggers, and synthetic conversations goes through the same harness as the defined datasets. We log four flags per trial, colloquially labeled L1, L2, L3, and L4. 

• L1 (raw): Is the trigger in the raw messages, right after we insert it? Separates controls from trials
• L2 (post-memory-policy): Does the trigger remain after the memory policy (keep recent, summarize, etc.)
• L3 (post-template): Is the trigger in the fully templated text, after chat formatting?
• L4 (final tokens): does it survive into the final token IDs, after truncation?

To ensure robustness and replicability, almost 1 million trials were run with this distribution. The approach is entirely model weight agnostic, as all manipulated properties are part of the tokenizer. Part 2 of this project series will focus on mechanistically determining where backdoors are affecting models.

I also found a great entry point for mechanistic anomaly detection to be [2].

Axis

Levels

n

model (tokenizer + template + window)

Qwen3-0.6B, Pythia-1B, TinyLlama-1.1B, Gemma-3-1B

4

data source

synthetic, long-document, LMSYS-Chat-1M, WildChat

4

pipeline policy

none, keep_recent_messages, truncate_head, truncate_tail, truncate_middle

5

trigger position

prefix, middle, end, old_turn, recent_turn, system, tool_output

7

context budget (tokens)

512, 1024, 2048

3

trigger type

random canary, multi-token phrase, boundary, natural phrase, unicode

5

counterfactual

trigger-present and trigger-absent twin

×2

Total

916,200 trials

"boundary triggers" refer to triggers partially split by truncations. The total trial number is a result of various retrials described in the GitHub for this project.

Trials were run with the same data but with no canary to ensure that the trigger was not just happening to appear in the conversations or datasets. In short, across 458,100 trials, there were zero in which the trigger was leaked despite not being present.

The trials resulted in these delivery numbers. At surface value for this testing apparatus, they raise genuine flags.

The top row, “none,” is the control. Every cell is a 1.00, meaning all triggers were delivered, except for the last column. The tool_output column shows the resulting context loss from something like an agent function or RAG system. This is why truncate_tail so harshly decreases trigger delivery for it; tool outputs are among the first things added. For auditing an agentic or retrieval system, this is an important note. A trigger arriving through a tool result is more fragile than one in ordinary chat text, not less, and a tail-dropping budget will eat it almost every time.

“Truncate_head” keeps only the last N tokens and throws away the front tokens. As expected, this deletes prefix triggers and old_turn triggers at a high rate. System prompts also sit at the same percentage because they tend to be at the front of a prompt. Because the trigger insertion for this experiment was so sterile, triggers of similar placements tended to have very similar likelihoods of being deleted–within 5 significant figures in this case for many of the points on the heatmap.

"Truncate_tail," as the name suggests, removes tokens from the end of the context. The highest trigger delivery rate for this method, 0.97, came from system triggers, which live near the front of the context. This is consistent with the broad interpretation that trigger survival can be determined as much by the context pipeline as by the model itself.

The uniformity of these results suggests a clear improvement to experimental harnesses that do not contain clear checks for potential trigger deletion. Even without running almost double the trials and interpreting model activations to detect backdoors, simply moving triggers to regions less vulnerable to a model’s memory and truncation policies can result in improved robustness. The table below describes a breakdown of potential misattributions.

Misattribution Rate

policy × position

delivered

apparent failure

of which: memory

truncation

truncate_head × prefix

0.21

0.79

0.00

0.79

truncate_head × old_turn

0.21

0.79

0.00

0.79

truncate_head × end

0.94

0.06

0.00

0.06

keep_recent × prefix

0.33

0.67

0.67

0.00

keep_recent × middle

0.55

0.45

0.45

0.00

Backing the claim that the similar rates across models are a result of the tested models having very similar, if not identical, tokenizers, Gemma fails equivalence testing under TOST. While other models have system roles, Gemma does not, instead merging the first system message with a user turn.  This role migration is also an important variable for a researcher to consider. For truncation policies which focus on either the system or the user, trigger detectors may deliver more false positives if expectations are unclear.

The trigger survival rates also appear very similar for real AI conversations and synthetic ones, with the only exceptions being a difference in the “keep_recent_messages” policy between LMSYS and WildChat and the longer document. As the document does not have messages, this result is expected.

Survival Per Dataset

policy

synthetic

LMSYS

WildChat

long-doc

none

0.99

1.00

1.00

1.00

keep_recent_messages

0.65

0.45

0.50

1.00

truncate_head

0.45

0.44

0.44

0.35

truncate_tail

0.68

0.75

0.75

0.77

truncate_middle

0.90

0.87

0.83

0.86

The policy not yet covered is summarization, which forms old turns into shorter, llm-written counterparts and can create different problems from truncation. Part 2 of this project series will mechanistically interpret a model to determine whether or not it is operating on an instruction similar to a backdoor but, for this scope, a semantic scoring system was used to determine if the model contains information close to a natural language trigger. Unlike a random canary or unicode, this type of trigger could retain information even through an LLM condensing it. As this scorer is probabilistic and model-dependent with some stochasticity, it was first tuned against a dataset of trials which either contained a canary or didn’t. It was also validated against a small hand-labeled set of summary/trigger pairs, where it hit 88% precision and 88% recall. The threshold in the table below is defined by where the trigger-absent twins stop tripping the summarizer.

Semantic Survival

summarizer behavior

outcome

threshold

false-positive rate on twins

verbatim (copies old turns)

exact_survival

0.27

0.00

paraphrase (rewords, keeps meaning)

semantic_survival

0.29

0.00

drop (content-free summary)

no_survival

—

0.00

The information from these trials highlights some cheap changes that can be made if you build backdoor or trigger evaluations. First and most cleanly, logging the final token IDs can be far more important than logging raw inputs. Secondly, ensure that you understand how your trigger interacts with the truncation policy—whether that be in user inputs, system inputs, summarization, tail truncations, head truncations, or some combination of mutations. Finally, for natural language triggers, one should limit trust in raw string matching.

Thank you for your time!

Thank you to Divij Chawla, who introduced this rough topic to me.

1. ^
   • https://arxiv.org/abs/2408.12798

   https://github.com/sks17/LLMBackdoorPipelineHygeine.

Discuss

What this is and how to read it

Ask how AI safety reached this point, and most people give you a neat story. A small group of philosophers raised the first concerns about superintelligence. The arrival of large language models changed that, and many researchers turned to the alignment problem. The story is too tidy, and it leaves out the interesting part. Examine timelines, contributors, funding sources and publications, and it falls apart. New directions appear. A few merge into the mainstream, others fade out, and money and research attention rarely align; one tends to follow the other by years. I wanted to study that pattern rather than declare it, so I built the entire project from the ground up.

Below is a table of 323 documented events, 129 actors and 18 directions. Its timeline runs from 2005 through June 2026, and the final year covers only six months. Next to it, a separate set of arXiv attention proxies covers 23 of those directions. I also built a deduplicated safety corpus from a single combined query, so no paper is counted twice. Each row is one verifiable fact backed by a primary source, tagged with year, actor, direction, type, amount, source, and my confidence. Every chart below draws on that table, and the path was never straight. Sort the events by year, funding and attention, and you see fresh strands open up, a handful fold into the mainstream, several fall quiet, while funding and research stay out of step. The charts below are all generated from this base.

Main takeaways

Two kinds of findings came out of the data. Some just put numbers on what the field already half-knew. Others genuinely surprised me, so those come first.

Surprises:

• Money and attention are systematically out of sync. Interpretability's scientific attention runs about 37 times ahead of what its roughly $1 million in grants would predict, while governance money ran ahead of its publications. Funding and research almost never arrive together.
• Government money quietly overtook philanthropy around 2023. Half a dozen national institutes (UK AISI, ARIA, Canada, Australia, NSF, EU) turned a one-donor field into an international, increasingly governmental one, and about $268 million of venture equity poured into safety startups, money that expects a return, not a grant.
• "Consolidation" is really proliferation. Even as attention and money concentrate around a few names, 2024 became a record year for founding new organisations (24 of them), a wave of one-to-five-person specialist shops in interpretability, AI security and chain-of-thought monitoring.
• The word "Safety" is quietly leaving the field's names. In 2025 the UK AI Safety Institute became the AI Security Institute, the US AI Safety Institute became CAISI, and Open Philanthropy, the donor that started it all, renamed itself Coefficient Giving.
• Going quiet is not the same as dying. Reward modeling's last discrete event is 2017, yet it runs to 2025 under the RLHF banner. Directions usually get absorbed or slip into the background, they rarely just die.

Expected, now with numbers behind it:

• The center of gravity moved from philosophy to empirical work. Foundations-and-strategy events fall from 67% of the field in 2005-2013 to 6% by 2024-2026, while technical safety climbs to about 60%, concentrating around a handful of players, with Anthropic the single most-connected node.
• RLHF won and dissolved into mainstream ML. Its arXiv proxy runs from 25 papers in 2021 to 1777 in 2025, precisely because it stopped being a safety topic and became standard practice.
• Governance exploded after ChatGPT. Money on the governance subtrack rose from $0.4 million in 2018 to $18.4 million in 2023, roughly a 46-fold jump in five years.

Each dot is an event. The organisation axis is sorted by year of appearance, so the dots form a diagonal from the bottom-left up to the top-right. The track colour is a chronological gradient by birth year, so the field's colour drifts over time at a glance. The bottom panel stacks the events by track per year in the same colours, and the activity shifts from early macrostrategy and agent foundations toward later interpretability, evaluations and AI control. One caveat: that panel counts events, so it shows collection density rather than real-world activity. The eras are hatched and stars mark the milestone papers. (interactive).

The timeline is sorted by the start of a track, but it doesn't reveal who survived to today. For that there's a separate lifespan chart: a line from birth to the last recorded event, tracks ordered by birth year. One note here: a missing late event is not a dead track (again collection density, per the caveat above). So the chart carries an independent signal, a diamond on the last year a track still publishes on arXiv, wherever a curve exists. A track can fall silent in the event log and still be alive.

The line runs from a direction's first recorded event to its last, and the diamond marks the last year it still publishes on arXiv, an independent proxy across 23 directions. The two often diverge. Reward modeling's last event is 2017, yet its publications, now under the RLHF banner, run to 2025. Value learning went quiet as discrete events by 2021, and agent foundations thins to a lone 2024 marker with MIRI's pivot. Both stay alive on arXiv. So a track rarely just dies. More often it gets absorbed, or slips into the background. There is more on that in the Cross-cutting patterns chapter, after the eras. (interactive).

Here is another look at tracks over time, now counting the number of events in the base per year:

The stacked area shows how many events of each direction landed in the base each year. A reminder of caveat 2: the height is collection density, not real work volume, and the late years (2026 is only the first half) are hatched as incomplete. So read the shape and the order of appearance rather than the absolute values. The field sits almost empty until 2012 and 2013. It then explodes in 2015 to 2017 with the first technical tracks. And from 2021 to 2024 it keeps growing as new directions such as evaluations, AI control and model organisms pile on top of the older ones. (interactive).

One more overview map pulls the whole route together, tracing how the flows run from era to organisation to direction.

(interactive).

At first glance it is just spaghetti. The middle column holds over a hundred organisations, about 115 of them, the ribbons cross, and reading it all at once is impossible. It is also unnecessary. So let us roll it up into big blocks: the roughly 115 organisations into seven actor types (researchers, funders, policy institutes, talent training, industry, meta and community, and government bodies), and the 17 directions into four families:

• Foundations and strategy covers macrostrategy, agent foundations, value learning and forecasting.
• Technical safety covers interpretability, evaluations, AI control, RLHF, scalable oversight, model organisms, robustness and the like.
• Governance and infrastructure covers governance, field-building and disclosure norms.
• Capabilities covers the growth of model capabilities itself.

The same picture comes next in two cross-sections that answer different questions: one shows the route as a whole, the other walks it step by step.

Cross-section one, the overall map. Three axes run left to right, from era (the when) to actor type (the who) to direction family (the on-what). The ribbon colour shows at once which family dominated and how that shifted.

The colour shows the direction family, and the numbers count non-money events, so they measure how much was collected rather than the field's real output (caveat 2). A diagonal jumps out. The blue foundations and strategy ribbons crowd the left and the early eras, while the orange technical safety gains mass toward the right. (interactive).

Cross-section two, by era. It is the same flow, with one mini-panel per era. The center of gravity shifts step by step across the panels, and one family dominates each period.

Each panel is one era, running from actor type to direction family, and the ribbon width is the number of events. The blue foundations and strategy rules from 2005 to 2016. Then from the 2020s the orange technical safety dominates the panels. (interactive).

There is also a third cross-section: two clean two-level flows side by side, splitting the when and the who.

On the left the flow runs from era to direction family, showing how the field's focus shifted over time. On the right it runs from actor type to family, showing who works on which family. The ribbon width is again the count of non-money events, which reflects collection effort rather than the field's true volume (caveat 2). On the left, the orange technical safety gains mass toward the later eras. On the right, researchers and industry pull mostly toward technical safety, while government bodies and policy institutes pull toward governance and infrastructure. (interactive).

The era-blocks show where the field's center of gravity moves. The number beside each is how many non-money events landed in that era (collection density, not world volume, see caveat 2):

• The prehistory runs from 2005 to 2012. The field is almost empty, just five events: the first macrostrategy institutes (FHI in 2005, GCRI in 2011), MIRI's founding for agent foundations in 2005, and the earliest capabilities and community nodes (DeepMind in 2010, CFAR in 2012). It is a philosophical question with no industry behind it.
• Next is the founding phase, 2013 to 2016. Macrostrategy still rules, at 10 of 27 events, but the first technical works and infrastructure start appearing beside it. The key nodes here are FLI, MIRI and the first Open Philanthropy.
• Institutionalization follows across 2017 to 2019. There is an explosion of field-building (10 events) and governance (4), and the center of gravity moves to OpenAI, Paul Christiano and GovAI as the field starts building real institutions.
• The prosaic turn arrives in 2020 and 2021. Infrastructure still leads, at 6 events, but interpretability and AI control appear for the first time as separate ribbons. The new nodes are Anthropic and Redwood.
• Scale and ChatGPT dominate 2022 and 2023. A dense block of 33 events, broken down as field-building (12), governance (5), evaluations (4), interpretability (3) and AI control (2). At the center sit the FTX Future Fund, Anthropic, CAIS and ARC Evals.
• Consolidation fills 2024 to 2026. After the exhaustive backfill, this is the densest block in the entire base, with 82 non-money events from 2024 to the first half of 2026. Governance leads with 26 of them, a stretch of summits, laws and commitments. Behind it come the empirical technical families, interpretability (12) and evaluations (12), then robustness (9) and model organisms (6). One node collects more ribbons than any other: Anthropic, with 9 events. And instead of collapsing, the field proliferated. 2024 became a record year for organisations founded, with 24 founding events against 7 in 2025, a wave of narrow specialists. There were interpretability shops such as Goodfire, Transluce, Guide Labs, Tilde, Simplex and Decode. There were AI-security and evaluation shops such as Palisade, Gray Swan and Virtue AI. There was chain-of-thought monitoring at Geodesic. And there was Bengio's "Scientist AI" outfit LawZero. One caveat: the high density of 2024 and 2025 is partly collection density, since the fresh years were backfilled from live primary sources in far more detail than the 2005-2015 retrospectives, rather than purely the growth of the field.

This migration is the storyline of the whole post: from the bottom-left corner (philosophy, lone institutes) to the top-right (empirics, attention and money concentrating around Anthropic and the safety institutes). But concentrated attention doesn't mean concentrated organisations. In this last era the field proliferates into dozens of narrow organisations (2024 was a record founding year in the base). The sections below take that apart era by era.

The field's activity composition shifted too, meaning what the eras are even made of. The prehistory era (2005-2012) is 100% organisation foundings: no papers, no grants yet. Publications, grants and funding statements enter only from 2013 on. By the last era foundings are just about 28% of a much larger, more varied mix.

This shows the shares of the event types, such as foundings, publications, grants and statements, within each era. Caveat 2 still holds: this is the composition of the collection, not the composition of the world. (interactive).

Finally, here's how all of it changed across the eras in one frame: four comparable panels where five epochs run horizontally (the same five substantive Parts below).

The panels run left to right and top to bottom. The first panel shows the direction families as a share of each era's events, out of 100%: the blue foundations and strategy starts at 67% in the 2005-2013 era and shrinks to 6% by 2024-2026, while the orange technical safety climbs to about 60%. Above the columns sits the total number of events in the era, so that normalizing does not hide the field's roughly tenfold growth. The second panel shows the actor types the same way, across seven types. Early on it is almost all researchers, at 56% in the 2005-2013 era. Funders enter as a distinct force in 2014 to 2019, at 24%. And by 2024 to 2026 the mix is transformed: government arrives from nothing to 24% and industry to 26%, now rivalling researchers at 29%. The state and the labs, not just philanthropy, drive the last era's recorded activity. The third panel shows the money by fund, the absolute amounts of all itemized grants per era, roughly $764 million all told. The six long-standing philanthropic funders (OpenPhil, SFF, FTX, FLI, LTFF and Tallinn) keep their own colours, while the newer government and international money (UK AISI at $159 million, ARIA at $74 million, Canada at $36.5 million, NSF at $20 million, Australia at $19.7 million, and so on) rolls into one grey band of other, government and new money. So the growth in money runs consistently end to end: about $0 at the start, then about $128 million in the 2014-2019 era led by philanthropy, then about $242 million in 2022 and 2023 and about $335 million in 2024 to 2026, where the grey government band dominates. The fourth panel shows the organisation dynamics, with births rising and closures and pivots falling. A couple of caveats: the first two panels are shares of collection events (caveat 2), not world volume, and the money panel is itemized grants only, with donor totals and pledges excluded, so that nothing is counted twice against the cumulative-funding chart. (interactive).

The birth and fade of individual directions already show up above on the direction-lifespan chart, and the cross-cutting over-all-years money and attention charts sit together in the Cross-cutting patterns chapter, which comes after the epochs. Now for the same five epochs, in order and in words.

Epoch 1, prehistory and birth (2005-2013): a question without tools

This era holds 9 events in the base, and the birth and fade of its directions runs along the direction-lifespan chart above.

It doesn't start with neural networks. It starts with a philosophical question: how do we avoid wrecking humanity's distant future? In 2005 the Future of Humanity Institute (FHI, Nick Bostrom) opens at Oxford, the earliest reference point in the base. Around it forms what this post calls macrostrategy: the broadest philosophical reasoning about global risks. Over the next years GCRI (2011), CSER (Cambridge, 2013), FRI (2013) and CFI (2015) join in. By event density, macrostrategy peaks in the mid-2010s.

The second root is mathematical. In 2013 MIRI pivots from public outreach to friendly AI as formal mathematics (decision theory, logical induction). That is the birth of agent foundations, the attempt to understand an idealized agent before anyone builds it.

At this stage the table holds almost no money. There's a question and there are first researchers, but no industry and no grant flows. The early nodes (FHI, GCRI, FRI) have already run for years, and none of them has died yet. This shows on the organisation-lifespan diagram in the Cross-cutting patterns chapter, after the epochs.

Epoch 2, institutionalization and the first big funder (2014-2019): money appears

In this era the money panel fills for the first time, and OpenPhil dominates immediately.

The turning point is January 2015: the FLI conference in Puerto Rico ("Future of AI"), which Nate Soares would later call exactly that. It produces an open letter and, with it, the field's first big donor: Open Philanthropy.

On cumulative money by fund (the field-wide summary money charts are in the Cross-cutting patterns chapter) the picture is unambiguous. OpenPhil dominates the whole field's funding. By annual donor totals, its investment into technical safety rises from $1.19 million in 2015 to $6.56 million in 2016, then to $43.2 million in 2017, up to a peak of $81.7 million in 2021. But that 2017 jump is mostly a single grant, the $30 million for general support of OpenAI in March 2017. A spending peak can be one big bet rather than a lot of research.

These same years also give birth to governance: policy, institutes, AI governance. In 2019 OpenPhil puts $55 million over five years into founding CSET (Georgetown), the largest governance grant of that epoch. Value learning runs in parallel, with Stuart Russell's center CHAI (Berkeley, 2016), which OpenPhil immediately backs with $5.6 million over two years.

Where the money actually went, by direction and year, shows up most clearly through itemized grants, without the aggregate donor totals, which would count the same dollars twice. The detailed money-by-track breakdowns live in the Cross-cutting patterns chapter. One figure stands out already: field-building (infrastructure like funds and fellowships) accounts for $326.1 million by itemized grants, the field's largest money flow. That figure grew after backfill, once field-building came to include large government programs like the UK AISI taskforce and new regranters. Next to it the other directions almost disappear.

Epoch 3, the prosaic turn (2020-2021): big models change everything

In this era the families' center of gravity shifts noticeably into technical safety.

By 2020 it's become clear that strong AI, if it arrives, will come from large neural networks rather than from pure agent theory. So the field turns to prosaic alignment, working with real models. Three shifts land at once:

• Mechanistic interpretability is born. March 2020 sees "Circuits" (Chris Olah, on distill), then Transformer Circuits in 2021. By the denoised arXiv proxy (mechanistic interpretability, sparse autoencoders and probing, after the bare word interpretability, which had been catching almost all of general ML, was dropped), attention is already rising, from roughly 34 publications in 2020 to 69 in 2021, then climbing steeply to 125, then 257, then 657 by 2025, and that early rise turns out to be only the beginning.
• Reward modeling develops into RLHF as a technique, with a clear lineage from Concrete Problems in 2016 to Deep RL from Human Preferences (Christiano and Leike, 2017).
• Agent foundations fade. Abstract, deep-learning-agnostic agent theory stops convincing donors in an era where empirical work on LLMs decides everything. Discrete events on the track thin out after 2021 and leave only a lone 2024 marker (MIRI's pivot). The direction survives mostly as background, not as fresh events.

The shifting money leader year to year, and the drift of shares from early technical bets toward governance and infrastructure, both show up on the cross-cutting money chart (absolute money by track). It sits in the Cross-cutting patterns chapter, so it reads across the whole field rather than one era at a time. The publication attention curves (arXiv proxy) by track sit there too, and the interpretability curve on them starts right around 2020-2021.

Epoch 4, the ChatGPT moment and the FTX shock (2022-2023): money, institutes, a break

In this era the money now comes from several funds at once (SFF, OpenPhil, FTX), and among the actors, institutes and industry noticeably grow.

In late 2022 ChatGPT pushed AI into big politics and mass consciousness. In the table that surfaces as a surge on several fronts at once:

• Evaluations, the dangerous-capabilities evaluations: ARC Evals (Beth Barnes, 2022), then Apollo Research (2023), pre-deployment evaluations of GPT-4 and Claude, and the UK and US safety institutes in 2023, around the Bletchley summit.
• AI control, the Redwood direction. The organisation was founded in June 2021, and by 2023 it has a formalized control research agenda. The money is dense: OpenPhil gave Redwood $9.4 million in 2021, $10.7 million in 2022 and $5.3 million in 2023, plus another $6.6 million from FTX in July 2022.
• The governance explosion. On the governance subtrack the money rose from $0.4 million in 2018 to $1.5 million in 2020, then $3.6 million in 2021, $10.3 million in 2022 and $18.4 million in 2023, roughly a 46-fold rise over five years. For comparison, technical safety in 2023 is $24.6 million, still larger.

Then the shock. November 2022, FTX collapses. In half a year (February-August 2022) the FTX Future Fund had handed out $18.7 million by name to AI safety, about $32 million by estimate overall. Now it's cut off, and some grants even have to be clawed back. On the cross-cutting cumulative chart the FTX line stops at 2022 and never grows again. The births and closures of organisations by year sit on the summary organisation-dynamics chart there too, and in the organisation-dynamics panel of the by-era overview chart.

Epoch 5, consolidation and the ending (2024-2026): Safety disappears from the names

In this era, consolidation looks more like continued activity, with the organisation dynamics showing a record 31 births against 8 closures or pivots.

The last era in the data is contraction and redefinition.

• In April 2024 FHI closes, the institute where it all began.
• In May 2024 OpenAI disbands the Superalignment team, which had been announced in July 2023 with a promise of 20% of compute over four years, after Sutskever and Leike leave. The direction scalable oversight loses its main institutional backer.
• In 2024 MIRI, the ancestor organisation of agent foundations, officially pivots from technical research to communications and policy. The oldest fundamental direction leaves the research stage, though not because it was solved: its carrier organisation simply changed its mission.
• In 2025 the word Safety disappears from the safety institutes' names. In February 2025 the UK AI Safety Institute becomes the AI Security Institute, and the US AI Safety Institute at NIST becomes CAISI, the Center for AI Standards and Innovation. Safety gives way to standards and to safety as security.
• In November 2025 the main funder changes its name. Open Philanthropy renames itself Coefficient Giving, announced on 18 November 2025, and restructures into 13 thematic multi-donor funds instead of one anchor donor in Good Ventures. The very money source that started the field's institutionalization changes its own name, echoing the disappearance of Safety from the institute names.

But this consolidation is really specialization and proliferation. 2024 is the record year in the base for organisations founded (24 founded-events). Interpretability gets commercialized and splinters into startups (Goodfire, Transluce, Guide Labs, Tilde, Simplex, Decode Research). A separate AI-security and evaluations cluster emerges, in Palisade Research, Gray Swan AI and Virtue AI. Organisations for fresh sub-agendas keep appearing: Geodesic for chain-of-thought monitoring, Luthien for practical AI control, Softmax for multi-agent alignment, Formation Research for lock-in risk, and Yoshua Bengio's LawZero in June 2025, built around a non-agentic "Scientist AI" as an oversight layer. Even the giants open internal cells: in 2025 Meta sets up a "superintelligence alignment and safety" team. Alongside all of it, non-technical infrastructure proliferates too: the forecasting cell AI Futures Project (Kokotajlo, the "AI 2027" scenario), the watchdog AI Lab Watch, the standardizers AI Standards Labs, national centers (Beijing Institute of AI Safety and Governance, France's CeSIA), the international association IASEAI, and the AI Whistleblower Initiative.

Governance stops being background and becomes the era's main storyline. It accounts for a third of the block's events (26 of 82 non-money), and the chronology tightens from summit to summit. November 2023, Bletchley. Then the AI Seoul Summit (May 2024): 16 frontier companies sign the Frontier AI Safety Commitments, each promising to publish its own safety framework with risk thresholds, and 27 countries plus the EU, in the Seoul Ministerial, agree for the first time to develop common thresholds for severe risks. 1 August 2024, the EU AI Act comes into force, the world's first broad, horizontal AI law, with rules for GPAI from August 2025 and a GPAI Code of Practice. In the US, hard law stalls. California's SB 1047 (tests, a kill switch, liability) is vetoed by Newsom (September 2024). Then in January 2025 the federal course swings from safety to dominance: the repeal of Biden's AI executive order, the "Removing Barriers to American Leadership in AI" order. Later in 2025 California comes back with a softer SB 53, focused on transparency. The industry builds its own: the Frontier Model Forum in 2023 and its AI Safety Fund, worth over $10 million in rounds of $4 million in November 2024 and over $5.2 million in December 2025. And an IPCC for AI takes shape as the International AI Safety Report, which ran from an interim version in May 2024 to a full report in January 2025 under Bengio, landing just before the AI Action Summit in Paris in February 2025, where the tone has already slid from safety toward action and deployment.

The technical directions don't stand still. Interpretability gains new momentum: Scaling Monosemanticity (Anthropic, May 2024, "Golden Gate Claude"), Gemma Scope (DeepMind, July 2024), an open set of SAEs shipped as public infrastructure, circuit tracing and "On the Biology of a Large Language Model" (March 2025), and "Auditing Language Models for Hidden Objectives" (Anthropic, March 2025), the first real audit game, where a goal is hidden in a model and then uncovered blind. The youngest direction, model organisms, forms around the empirics of deception: Sleeper Agents (January 2024), Sabotage Evaluations (Anthropic, October 2024), Apollo's "In-Context Scheming" (December 2024, where scheming showed up in 5 of 6 frontier models), Alignment Faking (December 2024), and Agentic Misalignment (Anthropic, June 2025, where 16 models in stress-tests resorted to blackmail or leaks to avoid shutdown). Safe-scaling frameworks evolved into their own genre. DeepMind published the Frontier Safety Framework in May 2024, and Anthropic revised its Responsible Scaling Policy that October. Scalable oversight had lost its home when the Superalignment program was cut, so researchers rebuilt its goals in other projects. In December 2024 OpenAI introduced Deliberative Alignment. The following September, it teamed up with Apollo to release anti-scheming training that reduced hidden actions by about 30-fold but never eliminated them. That work spurred agentic control evaluations, among them Redwood's Ctrl-Z in April 2025. Everything culminated in July 2025, when OpenAI, DeepMind, Anthropic and the UK AISI co-signed the position paper "Chain of Thought Monitorability", which outlined their collective responsibility to protect the limited window into model reasoning. The first six months of 2026 followed similar patterns, and the dataset includes only data through H1.

• Interpretability continued to advance. On 7 May 2026 Anthropic published "Natural Language Autoencoders", which turns a model's internal activations into readable text. In the pre-deployment audit of Claude Opus 4.6, the method identified evaluation awareness: the model recognized it was under test in 16 to 26% of benchmark transcripts, but under 1% of real-world traffic. The discipline is now part of the release cycle rather than a side project.
• Evaluations turned toward security. The UK AI Security Institute released a report stating that Claude Mythos Preview was the first model to complete the 32-step network-attack scenario "The Last Ones" on 13 April; GPT-5.5 followed on 30 April. The 2025 rebranding from Safety to Security marked a genuine change, because the agenda shifted to autonomous cyber risk. In May 2026 the EU provisionally approved the "Digital Omnibus" as its first set of AI Act amendments, without changing the obligations for GPAI models.
• On the political front, in May 2026 the EU preliminarily agrees on the "Digital Omnibus", the first amendments to the AI Act, though the obligations for GPAI models are left untouched.

Cross-cutting patterns: money, attention and the three fates of a track

Having walked the field epoch by epoch, it is worth stepping back to look across all the years at once, because some patterns surface only that way and slicing them by era hides them. Money is the most tangled of these, so it comes first.

Four money views, and why the same fund shows different amounts. The money here gets counted four different ways, which deliberately do not add up; otherwise the same dollars would land in the total twice. They are four views of one pile.

The first is the itemized grants, at $763.6 million. These are the real targeted grants, recorded as they went out. They show in full on the money-by-track chart, where the bars are the tracks plus a grey other-and-untracked strip of $83.6 million, and on both sankeys. Only the track-classified $680.0 million appears on the radar and the per-track money-versus-attention chart, because the aggregate donor baskets and one capabilities grant, $83.6 million together, have no specific track. The second view is the donor annual totals. Each is a donor's full budget for the year, the umbrella figure rather than the sum of its sub-category breakdowns, which would tally the same money twice. This is howOpenPhil (about $304.5 million) and LTFF (about $3.64 million) show up on the cumulative-money-by-fund chart, while every other fund shows by its itemized grants. The same two totals run to 2024 on the field-wide money-versus-attention chart; the 2025-and-later donor totals are not published yet, but the individual-grant view keeps flowing past that point.

The third view is the venture money, the VC and equity, at $268.5 million. It expects a return, arrives across five rounds, appears only on the third panel of the cumulative-money chart, and is never summed with the first two. The fourth is the pledges, budgets and estimates, kept as context and never summed. This is money that is not a disbursed grant: FTX's launch pledge of about $160 million against the $18.7 million it actually disbursed, organisations' annual budgets like MIRI, CHAI and CSER, and overlapping field-wide estimates such as the roughly $40 million EA spent in 2019. It was invisible on every chart until it got its own dots-only view. Because the field-wide estimates already include the organisation budgets, it is heterogeneous and non-additive, so it is plotted as points and never totalled, purely so that nothing collected stays hidden.

Here is where the confusion comes from. OpenPhil is $176.77 million as the sum of its itemized grants under the first view, and about $304.5 million as the donor annual total under the second. It is the same organisation seen two ways. No figure is lost; the two simply cannot be added together.

Money across the whole field. The historical main funder was OpenPhil, with $176.77 million of itemized grants, and after backfill it got caught and overtaken by the government money of half a dozen countries. The UK AISI and its taskforce come to about $159 million altogether (£100 million plus the Alignment Project), with ARIA Safeguarded AI at $74 million close behind. Then a longer tail: Canada CAISI at $36.5 million, NSF Safe Learning-Enabled Systems at $20 million, Australia AISI at $19.7 million, the US AISI at NIST at $10 million, DARPA GARD at $10 million, and the EU AI Office at $9.8 million. SFF stands alongside at $144 million. FTX still cuts off at the end of 2022. So the one-big-funder picture of 2020 to 2022 gives way to something far more distributed in 2023 to 2025, and noticeably more governmental, now international. But this is already different types of money (caveat 5). And beside it a fourth type has appeared, venture capital into safety startups, about $268 million of equity. It stays a separate view with its own colour and line across the charts below, never mixed into the grant totals. The grant money appears three ways, ranked, cumulative over time, and as a dot-strip, because no single view catches both who gave how much and when the money arrived.

There is one bar per funder, sorted, on a log axis, and coloured by type. The philanthropy bars are blue: Open Philanthropy sits at its donor annual total of*$304.5 million** (the same OpenPhil that reads $176.77 million under the itemized-grant view; caveat 5), then SFF at $144 million, FLI at $39 million and FTX at $18.7 million disbursed, followed by Schmidt, Longview, Manifund, Tallinn, LTFF, Founders Pledge and Effektiv Spenden. The government bars are orange: the UK AISI at about $159 million, ARIA at $74 million, Canada CAISI at $36.5 million, NSF at $20 million, Australia AISI at $19.7 million, the US AISI at NIST at $10 million, DARPA at $10 million and the EU AI Office at $9.8 million. The corporate bar is red: the AI Safety Fund at the Frontier Model Forum, at $9.2 million. And a distinct fourth group, the VC and equity money, is green: Protect AI at $108.5 million, HiddenLayer at $50 million, Goodfire at $50 million, Gray Swan at $40 million and Lakera at $20 million, about $268 million in all. Equity expects a return, so it is never summed with the $763.6 million of grants. It just shares the magnitude axis, which shows safety becoming a market. (interactive).*

The same money over time, now stacked by funder type, showing not only who but when.

The stack is cumulative grant dollars by type, with philanthropy in blue, government in orange and corporate in red. OpenPhil and LTFF sit at their donor annual totals and the rest are itemized, which is why the canonical stack runs a little above the $763.6 million itemized-grant total. The shape is the point here: through 2020 it is almost entirely a single blue channel of philanthropy. Then from 2023 the orange government band explodes, with half a dozen national safety institutes landing at once, turning one fragile channel into a broad, increasingly governmental and international flow. The dotted green line is the VC and equity money, about $268 million, a separate view laid over the same axis and never summed into the grant stack. (interactive).

Third, a dot-strip, since the shapes above hide the rhythm of individual moves.

The x-axis is the year and the y-axis is the funder, sorted by total, while the dot area shows that year's dollars and the colour shows the type. OpenPhil is the only funder active almost every year, and its 2021-2023 dots are the largest. The government cluster is unmistakable in 2023 to 2025, in big orange dots for the UK AISI, ARIA, Canada CAISI, NSF and Australia AISI. And VC rounds show up as green diamonds from 2023 onward, with Protect AI in 2024 the largest. The timing makes the regime change legible: philanthropy carried the field's money alone until 2023, and then the state and the market arrived together across the last three years. (interactive).

A fourth view keeps any collected dollar from staying invisible: the money that is not a disbursed grant.

Ranked horizontal bars group the sub-types on a log axis. What matters here is the magnitude, because these are not comparable, additive dollars. The colour shows the sub-type. The launch pledges include the FTX Future Fund's pledge of about $160 million at launch. The organisations' annual budgets include FTX's own $50 million and $32 million figures and MIRI at about $7.5 million a year, along with CHAI, CSER, Ought and Lightcone. The field-wide estimates include the estimate of roughly $40 million spent on AI safety in 2019 plus the 2014-2016 estimates, which deliberately overlap the organisation budgets. And the seed at founding includes SFF at about $2 million and Timaeus at $0.14 million. This view is heterogeneous and non-additive. The field estimates already contain the organisation budgets, and a pledge is not a disbursement. So it is never summed and never mixed with the grants ($763.6 million), the donor totals, or the VC money ($268.5 million). Its only job is to show that these numbers exist in the data. (interactive).

That gap shows up most sharply in one case, which gets its own chart.

Here is the headline as a dumbbell: the FTX Future Fund pledged about $160 million at launch but actually disbursed only $18.7 million before its collapse in November 2022. That gap between what was announced and what was actually delivered is the clearest warning of the era. Below it, the rest of the non-grant money, the organisation budgets, the overlapping field estimates and the seeds, is ranked on the same log axis for scale. It is still the fourth view, so never summed with grants, donor totals, or VC. (interactive).

And now for where this money actually flows, the routing of itemized grants from fund to track.

The flow here is a Sankey of the known itemized grants, with funds on the left, research tracks on the right, and the ribbon width showing the grant amount. OpenPhil and SFF clearly give across many tracks at once, while the government institutes (UK AISI, Canada CAISI, Australia AISI, US AISI at NIST, EU AI Office, DARPA and NSF) enter with precision, mostly into field-building, evaluations and robustness. This routes all the itemized grants, some $764 million altogether: grants whose direction is an aggregate donor-basket (the mixed-technical or technical-safety baskets) or capabilities get their own right-hand nodes rather than being dropped. Only donor totals and the VC and equity money stay out, because otherwise those dollars would be counted twice. (interactive).

The same grant flow, but now with a third level, the era, added on the left. That way it shows who, where and when at the same time.

This has three levels, running from era to fund to track. It reads clearly how the mass of grants shifted into the consolidation era of 2024 to 2026, and exactly how the new government institutes support evaluations, field-building and robustness. The recipient organisation is free text in the detail field, not a structural column, so it does not get its own level, a gap in the data. (interactive).

Money versus attention, and the life of organisations. Money, as donor totals, and scientific attention, as the arXiv proxy, keep drifting apart over time. And the births and closures of organisations give a separate organisation-dynamics curve across the whole field.

Four things are drawn here, and it is worth reading them separately.

• The blue bars, on the left axis in dollars per year, are the annual donor totals of the two funders that publish a clean field-wide number, Open Philanthropy and LTFF, de-duplicated by taking the umbrella mixed-technical row where it exists, otherwise that funder-year's full total, so the sub-category rows never get added on top. This is the only yearly money series that never counts a dollar twice, which is exactly why it is used here.
• As for why the bars stop at 2024 and there is nothing for 2025 or 2026: this series depends on the donors' published annual giving totals, and Open Philanthropy and LTFF simply have not released their 2025 and 2026 totals yet, since the annual report lags. The last bar, in 2024, is OpenPhil's technical-safety total of about $28 million. So the empty 2025 and 2026 mean missing source data, not a build or cache glitch, and not that the money dried up. The itemized-grant charts above, the era shift and the money-by-track, do run into 2025, because they count individual itemized grants, a different view that must not be summed with these donor totals, since that would count the same dollars twice.
• The solid red line, on the logarithmic right axis, is the deduplicated safety corpus, one combined arXiv OR-query with each unique safety paper counted once per year. This is the reliable field-level line.
• The dotted red line is the inflated keyword-proxy sum over the non-overlapping core of tracks, drawn on purpose to show how much the per-track proxies overstate the real count by counting papers twice when they match several tracks (caveat 1).

The grey band marks the partial final year, the first half only, and the right axis is logarithmic. Both money and attention are proxies, but the deduplicated line is reliable at the field level. (interactive).

Up is new organisations, down is closures and focus changes. One thing to keep in mind: this counts events per year, not organisations. There are 14 closure or pivot events in all, 3 closures and 11 pivots, and a single organisation can change focus several times. MIRI pivoted in 2013, 2018 and 2024; OpenAI pivoted in 2019 and then closed its Superalignment team in 2024; and Open Philanthropy pivoted in 2024 and 2025. So there are more red events in total than distinct dead organisations, which is why there is more red here than there are crosses on the chart below. (interactive).

Each bar is one organisation, unlike the chart above, which counts events: it runs from the founding year to the last closure or focus change, or to the present if still alive, with a cross marking the organisation's last such event. The founding dates of MIRI (2005, when SIAI reoriented to AI risk), Open Philanthropy (2014), CFAR (2012), LTFF (2017), METR (2022, as ARC Evals) and the FTX Future Fund (2022) are added from sources. So now every organisation with a closure or pivot is visible, and the chart converges with the one above. The 14 red events there collapse to 10 distinct organisations here, each carrying one cross on its last such event, because MIRI, OpenAI and Open Philanthropy each pivoted more than once. Pivots happen across the whole timeline, with MIRI's turn coming as early as 2013 and CFAR's in 2016. The actual closures, just 3 of the 14 events, land in 2024, at FHI and OpenAI's Superalignment team, with the FTX Future Fund's 2022 collapse as the earlier exception. (interactive).

A track's three fates. The table's biggest lesson is that a direction here can meet three different fates, and disappearing from the conversation is not the same as dying.

Rising: interpretability, evaluations and AI control, all born recently and growing. Interpretability is the reference case. By the denoised arXiv proxy, after the bare word interpretability was stripped from the query, where it had caught all of general ML, the curve is far more modest, rising from 69 in 2021 to 81, then 125, then 257, then 657 by 2025. The shape holds, a steady exponential rise, and it is that shape, not the absolute count, that carries the point. The same climb shows in the field-wide deduplicated corpus, which runs from 351 in 2021 to 427, then 890, then 2191, then 3813 by 2025 unique safety papers per year, almost an order of magnitude in four years. Money, meanwhile, lags sharply behind attention, with only $1.04 million of itemized grants into interpretability, the classic case of attention outpacing its funding. For evaluations the gap closed almost overnight. Before 2024 there was no itemized money on the track at all. Then it filled to about $96.7 million in itemized grants, nearly all of it in 2024 and 2025, making evaluations one of the field's better-funded tracks in a single year. Some of that is dedicated philanthropic and US money: the AI Safety Fund at the Frontier Model Forum gave $9.2 million in two rounds, $4.0 million in 2024 plus $5.2 million in 2025, for biological, cyber and agentic evaluations; the US AI Safety Institute at NIST gave $10 million in 2024; and Schmidt Sciences gave $10 million in 2025 for the science of evaluations. But the bulk comes from the new international government institutes routing into evaluations: Canada CAISI at $36.5 million, Australia AISI at $19.7 million and the EU AI Office at $9.8 million, plus an OpenPhil grant of $1.5 million. In the per-track panels below, evaluations now shows up with both a money bar and an arXiv line, and on the gap chart it sits close to the field average instead of off the money-starved end.

Absorbed: reward modeling, now RLHF, and value learning. RLHF by the arXiv proxy, on the broadened query for RLHF, reward models, DPO and preference optimization, runs from 25 in 2021 to 43, then 275, then 1010 in 2024, then 1777 in 2025. The direction vanished from safety conversations because it had won. RLHF and DPO became the standard way to fine-tune every commercial LLM and dissolved into mainstream ML. Publications keep growing precisely because it is shared infrastructure now, not a separate safety track.

Faded: agent foundations and early macrostrategy. They were the field's core, then shrank. Agent foundations barely registers on the arXiv proxy before 2024, since the term is young while the direction itself is old, the reverse case of proxy lag. The closure of FHI in 2024 is a symbolic full stop for macrostrategy.

Money and attention for each track, laid out one small panel at a time so nothing overlaps:

The grid holds small multiples, one mini-panel per track, ordered by total grant money. In each panel the bars, on the left axis, are that track's itemized grant money per year in dollars, and the dotted line, on the logarithmic right axis, is its scientific attention on the arXiv proxy for the same years. So each track is legible on its own and its divergence between money and attention reads directly. This replaces the old single dual-axis chart, where about 20 attention lines piled over the stacked bars into unreadable spaghetti. Here all the itemized grants show, about $764 million in total: grants without a specific track collect into one grey other, untracked or aggregate panel, about $83.6 million, made up of the aggregated donor baskets for mixed-technical and technical-safety plus one non-safety grant, money only, with no arXiv line, so no dollar is lost and the grid still totals the full $763.6 million. Donor annual totals for OpenPhil and LTFF, and the venture money in equity, live on the cumulative-funding chart and are not summed here, because that would count them twice. Interpretability shows a steep dotted rise over tiny bars, its attention far ahead of its money, even after denoising the curve by dropping the bare word interpretability. Evaluations shows a wall of bars for 2024 and 2025. AI control has money bars from 2021 before its attention lifts. Tracks that have an arXiv proxy but no itemized grant money at all (reward modeling, red-teaming, unlearning, chain-of-thought faithfulness, honesty and ELK, and the umbrella and nested directions, per caveat 1) get no panel here, since there is no grant bar to rank them by. Their attention instead sits on the field-level money-versus-attention chart and the per-track proxy curves. Both series here are only proxies, and the deduplicated safety corpus is deliberately not drawn on this grid, being field-level. (interactive).

This diverging gap chart replaces the old log-log trajectory tangle. For every track with both a money series and an arXiv proxy, the bar shows how far it sits from the field-average rate of money to attention, computed as the base-2 logarithm of the actual attention divided by the field's average papers-per-dollar times that track's money. A blue bar pointing right means scientific attention runs ahead of money. A red bar pointing left means money runs ahead of attention. The label gives the factor. Interpretability is the extreme blue case, its attention roughly 36.9 times ahead of what its modest $1 million or so in grants would predict, the classic science-runs-ahead-of-money gap. On the red side sit scalable oversight, agent foundations and governance, directions where itemized money outran the arXiv attention. Both signals are proxies, per caveats 1 and 2. (interactive).

A companion dumbbell view lays out the same tracks. Instead of the magnitude of the gap it shows the ordering: each track's rank by grant money, as an orange dot, against its rank by arXiv attention, as a green dot, joined by a connector, where rank 1 is the smallest and the largest rank is the biggest. A long blue connector means attention ranks the track far above where money does. A long red one means money ranks it far above attention. Overlapping dots mean the two agree. Set beside the gap chart above, which shows magnitude, it makes the who-leads story concrete: interpretability's dots pull wide apart, while better-matched tracks keep their two dots close, both ranks resting on proxies (caveats 1 and 2). (interactive).

And here is the money footprint of the tracks, on a single money axis, with a caveat.

A note on this one: it is a single logarithmic axis for money, not a multi-metric comparison. Multi-metric radars mislead across different scales, so there is no one built here. (interactive).

Finale: open questions

There is no moral to end on; the data does not hand one over. What it does leave is a handful of open questions the field is currently standing on:

• If Safety got dropped from the institutes' names, is that a change of sign or a change of substance? In the last era governance became the dominant flow (26 of 82 non-money events across 2024-H1 2026, a third of the block): summits, laws, commitments. But it all moved under the banner of security, standards and action. Will the same researchers grow up under that banner as under safety, or different ones?
• Is consolidation concentration, or is it actually fragmentation? By money and attention, the field is clearly gravitating toward a few names: Anthropic, the safety institutes, one large donor. When measured by headcount, the situation reverses. The year 2024 set a record for new organizations, mostly shops with one to five employees in interpretability, AI security and chain-of-thought monitoring. Is the field maturing and specializing, or fragmenting, with only a few likely to survive? Does absorption count as success, or does it erase the original players? RLHF won, then was reabsorbed into ordinary engineering. Is it good news when a safety method stops counting as a safety topic? Once that happens, who keeps track of where the method fails?
• How much does the field depend on one or two funders? When shown together, the OpenPhil channel and the FTX collapse reveal the field's exposure. Evaluations only recently gained an independent backer, the AI Safety Fund, yet interpretability still leans almost entirely on a single funder. What if that funder drops out or scales back support? Scientific progress has outpaced funding for interpretability; governance funding, by contrast, has grown faster than the science. Which of those two gaps is the healthy one, and what does the answer reveal about what the field is actually doing?
• Science runs ahead of money in interpretability, while governance money ran ahead of publications. Which gap is the right one, and what does it tell us about what the field is actually doing?

Appendix: methodology

Data and sources. The dataset includes 323 documented events from 2005 through mid-2026, plus a deduplicated safety corpus and arXiv proxies across 23 directions. Each count links to the exact search-query URL that reproduces it. We backfilled most of the recent detail through several passes over live primary sources, rather than retrospectives that stop at 2023. We identified newly founded organizations from an analysis of field growth on the EA Forum. The research milestones for 2024 and 2025 come from the "Shallow Review of Technical AI Safety", with its arXiv and lab links, and the governance events from primary sources: gov.uk, EU documents, California legislation, and White House records.

They brought in new philanthropic grantmakers (Manifund, Longview, Founders Pledge, Schmidt Sciences, Effektiv Spenden, Macroscopic Ventures and Nonlinear) and international government institutes (UK AISI and ARIA, US AISI at NIST, NSF, Canada CAISI, Australia AISI, EU AI Office, DARPA's GARD and AIxCC, IARPA's TrojAI, Singapore AISI, the annual UK AISI budget of about £50 million a year, Germany's DE-AISI and India AISI). They also added venture rounds into safety startups as a separate view, recorded as investments: Goodfire at $50 million, Gray Swan at $40 million, Lakera at $20 million, HiddenLayer at $50 million, and Protect AI at $108.5 million before its Palo Alto acquisition of $634.5 million. And they added the missing founding events for safety organisations (EleutherAI, Apart Research, Lakera, Protect AI) and the individual donor Craig Newmark. The last two enter as a statement and an investment so that grants do not double-count.

In the final pass the arXiv method turned toward precision over breadth. A deduplicated safety corpus was added, one combined OR-query with each paper counted once. The two worst tracks were denoised: the bare word interpretability was dropped from the interpretability track, and truthfulness was flagged high-noise and pulled from the field sum. Every proxy track now carries a noise flag.

Three collector scripts write the data with a source URL and rerun reproducibly: one pulls arXiv, one the LTFF annual payouts from EA Funds, and one the SFF annual totals. So 2024 comes out as the densest year of the base, partly a matter of collection density, which the direction-lifespan charts flag. Every event ties to a primary source, and the source types run as follows.

• arXiv preprints give the scientific attention by direction.
• Lab technical reports and blogs give Circuits on distill, Transformer Circuits and other Anthropic publications, plus OpenAI and DeepMind announcements.
• Community retrospectives and reviews come from LessWrong, the Alignment Forum and the EA Forum.
• Nonprofit tax filings, the 990 forms via ProPublica, give the revenue and size of nonprofits.
• The web archive, the Wayback Machine, covers pages that have since disappeared or been rewritten.
• Regulator pages cover the UK AI Safety and Security Institute, NIST and CAISI, and the EU institutes and documents.
• Fund announcements and annual reports come from Open Philanthropy, the Survival and Flourishing Fund (SFF), the Long-Term Future Fund (LTFF), the Future of Life Institute (FLI) and the FTX Future Fund, plus a community donations aggregator.

How the money is counted. Itemized grants (a specific amount to a specific recipient in a specific year) are kept separate from donor annual totals (how much a fund allocated in total for a year). To avoid double-counting, each donor uses one canonical series: annual totals for OpenPhil and LTFF, itemized grants for everyone else. The key amounts from the text:

• The itemized grants come to $763,625,573 in total, across 79 grants, and by fund they fall into three kinds of money. The philanthropies are the oldest and largest block: OpenPhil at $176.77 million, SFF at $144.04 million, FLI at $39.01 million and FTX at $18.74 million, then Schmidt Sciences at $10.0 million, Longview at $9.0 million, LTFF at $6.41 million, Manifund at $5.42 million, Jaan Tallinn at $5.0 million, Founders Pledge at $0.81 million and Effektiv Spenden at $0.18 million. The government institutes are the newer arrivals: the UK AI Safety Institute at $159.0 million, ARIA at $74.0 million, Canada CAISI at $36.5 million, NSF at $20.0 million,Australia AISI at $19.73 million, the US AI Safety Institute at NIST at $10.0 million, DARPA's GARD at $10.0 million and the EU AI Office at $9.81 million. And one corporate source stands beside them, the AI Safety Fund at the Frontier Model Forum at $9.20 million. The bolded items came in with the last deep backfill; they are already international government institutes (Canada CAISI at CAD$50 million over five years, Australia AISI at A$29.9 million over four years, the EU AI Office at €9.08 million, and DARPA's GARD at $10 million for fiscal year 2024), alongside the new retail fund Effektiv Spenden. OpenPhil also grew thanks to itemized 2024 grants to CAIS, Redwood and MIRI of about $3.8 million, flagged as a subset of the donor total so as not to double-count. These are different types of money (caveat 5), and some of it is stated program budgets (the UK's £100 million, ARIA's £59 million, Canada's CAD$50 million and Australia's A$29.9 million) rather than disbursed amounts. Currencies convert to dollars at reference rates (about 1.25 for the pound, 1.08 for the euro, 0.73 for the Canadian dollar and 0.66 for the Australian dollar) and are flagged in the detail field. Double-counting is excluded: donor totals for OpenPhil and LTFF and their itemized grants are never summed, and the overlaps (NSF against OpenPhil at $5 million, the Alignment Project against OpenAI and Schmidt, and OpenPhil's itemized grants against its total) are flagged in the data.
• The VC and equity money is a separate view and is not in this $763 million. Venture rounds into safety startups, recorded as investments, are not charitable grants but investments expecting a return, so they stay separate and never sum with grants: Goodfire at $50 million in a Series A for interpretability, Gray Swan at $40 million, Lakera at $20 million, HiddenLayer at $50 million, and Protect AI at $108.5 million before it was acquired by Palo Alto Networks for $634.5 million, coming to about $268 million of disclosed equity. Tellingly, the very first venture check into pure interpretability, Goodfire's $50 million, rivals OpenPhil's entire annual technical budget. Safety is becoming a market, not only a philanthropic cause.
• SFF by year, from verified SFF announcements as recorded in the event base and plotted on the charts, rises from $5.45 million in 2020 to $19.4 million in 2021, then $18.1 million in 2022, then a peak of $42.3 million in 2023, then $24.0 million in 2024 and $34.9 million in 2025, for $144.04 million in total. The SFF annual totals and the LTFF annual payouts export to two separate files, each with a source URL.
• OpenPhil's 2024 spend on technical safety is about $28 million as recorded in the grants database, with a caveat: this uses a different counting method than the donations aggregator did for 2015 to 2023, and the organisation itself says "about $50 million" for 2024. For 2023 it is about $46 million on AI safety. In 2025 more than $1 billion moved across all causes, though the fund does not publish an AI-specific total, and of that a $40 million request for proposals on technical AI safety was announced.

On 2026 (incomplete year, H1). Real events of the first half of 2026 came from primary sources (Anthropic Natural Language Autoencoders, the UK AI Security Institute cyber evaluations, the EU Digital Omnibus). But there's no distributed money for 2026 yet. SFF-2026 is only announced at $20 to $40 million with distribution in the autumn, so the amount isn't recorded, and the AI Safety Fund round with a figure fell in December 2025.

The arXiv series were rebuilt with one unified method across all 23 directions: the same reproducible recipe, OR-synonyms in the title and abstract plus a per-track category filter, counting by submission date, in a common 2015-2026 window, dropping the earlier mixed methodology. The per-track proxies gained the deduplicated safety corpus, one combined export with each paper counted once, which on the money-versus-attention chart is the solid line against the dotted inflated keyword sum. The two noisiest curves were denoised: interpretability without the bare word interpretability, where the count dropped severalfold, and truthfulness flagged as high-noise and removed from the field sum.

2026 stays incomplete, the first half only, and is marked with a grey zone. So an incomplete year does not read as a decline, and the last-full-year arXiv diamonds on the direction-lifespan chart sit on 2025. The rename from OpenPhil to Coefficient Giving is pinned to its real date, November 2025.

Five caveats that the above depends on:

1. arXiv publication counts are a proxy, not bibliometrics. A keyword like RLHF or scalable oversight surfaces on arXiv later than the real start of a track. Broad phrases such as AI control, dangerous capabilities, value learning and model editing add noise on top. And the query itself kept widening with each backfill. An exact phrase gave way to synonyms joined by OR under a category filter; the search then ran across both the title and the abstract, with the category set tuned track by track. So the adversarial track drew on cs.CR and cs.CV, governance on cs.CY, and multi-agent on cs.RO, all inside one 2015-2026 window. That inflates the per-track numbers. Much of the apparent jump is the net widening, not the field growing. Hence the shift in the last pass to precision over breadth: a deduplicated safety corpus, one combined OR-query of safety-specific phrases where arXiv counts each paper exactly once. That is the number to trust at the field level. The two worst tracks were denoised too. Stripping the bare word interpretability from the interpretability query, where it had swept in almost all of general ML, cut the count severalfold. The truthfulness track was worse still: in practice just hallucination and factuality, and so plain NLP, it was flagged high-noise and dropped from the field sum. Every proxy track now carries an explicit noise flag of low, medium or high. The scale is worth stating plainly. The cs.AI, cs.LG, cs.CL and stat.ML papers from 2015 to 2026 run into the hundreds of thousands, while those that self-label as AI safety or alignment number only a few thousand. So even the whole safety corpus is a thin strip of general ML, and the per-track sums overstate it many times over. One last wrinkle. Some proxy tracks are umbrellas, or nested inside one another: the broad alignment track covers everything, constitutional AI sits inside RLHF, and activation steering and singular learning theory sit inside interpretability. The attention curves therefore cannot be naively summed. A field-attention sum uses either the deduplicated corpus or the non-overlapping core of tracks, with the rest shown only as separate curves. What matters is the shape of each curve, whether it bursts or plateaus, and the relative comparisons, not the exact number.
2. The number of events per track is a biased measure. It reflects how much was recorded, not how much actually happened in the world. For real activity the better guide is money and publications, not the event counter. (On the collection-density chart the recent years are explicitly hatched as under-collected.)
3. The causal why is interpretation. It is layered on top of verified events, not lifted from a source, and stands as an argument open to dispute.
4. The grant-recipient organisation is not broken out as a separate structure. It exists only in the event description text, so the money flows run from fund to direction, not from fund to organisation.
5. The money now mixes several types, and each stays in its own view. After several backfills the money charts stopped being only philanthropic grants. They now also take in government programs from half a dozen countries (UK AISI, ARIA, US AISI at NIST, NSF, Canada CAISI, Australia AISI, EU AI Office, DARPA GARD) and corporate-philanthropic sources (AI Safety Fund, Schmidt Sciences, Manifund, Longview, Founders Pledge, Effektiv Spenden). These are not the same kind of dollar: a government budget is not a private grant, neither one is a compute promise, and some entries are a stated program budget rather than a disbursed amount. Next comes a genuinely new, separate view, the venture money (VC and equity) flowing into safety startups, recorded as investments. Goodfire raised $50 million in a Series A for interpretability; Gray Swan raised $40 million, Lakera $20 million and HiddenLayer $50 million; and Protect AI raised $108.5 million before Palo Alto acquired it for $634.5 million. That is about $268 million of disclosed equity, and it is not a charitable grant: the investor wants a return, so it is never summed with grants and stays out of the grant dollar-charts entirely, in a view of its own. A fourth kind of money is neither grant nor equity: pledges, annual budgets and field-wide estimates. This is money that gets collected without being a disbursed grant, like FTX's launch pledge of about $160 million against the $18.7 million it actually paid out, or organisations' annual budgets, or estimates such as the roughly $40 million spent on AI safety in 2019. It used to be invisible on every chart; now it has its own dots-only fourth view, shown but never summed, because the entries are heterogeneous and overlapping, since the field estimates already include the organisation budgets. Only the promises with no figure at all stay purely narrative, because there is simply no number to place: OpenAI Superalignment's compute, Google DeepMind's Frontier Safety Framework, the Protect AI acquisition, Germany's DE-AISI, India AISI, the Singapore, IARPA and AIxCC institutes, and the donor Craig Newmark. Rolling all of this into one field volume takes care, so that is flagged at the relevant charts.

Where to recheck numbers. GIT ai-safety-genealogy

Discuss

Natural language autoencoders are meant to take in an LLM's activation vector and describe in plain text what the model is thinking. However, its training data collection involves asking Claude to guess what a model might be thinking. How robust are NLAs to these guesses? We change Claude's guesses in various ways and measure the impact on the NLA's statements as well as on reconstruction accuracy. We show that Qwen2.5-7B NLAs have some robustness to irrelevant statements and prevailing sentiments in Claude's guesses.

However, if an NLA is initialized with entirely implausible statements, it can nevertheless achieve nearly the same reconstruction accuracy as plausible-initialized NLAs while emitting 99.3% implausible statements. RL does train implausible-initialized NLAs to be slightly more plausible (increasing from 0.08% to 0.7%). But the plausibility of plausible-initialized NLAs decreases from 21% at initialization to 7.6% at the end of training.

If our results scale, they cast doubt on the usefulness of NLAs.

Produced as part of the MATS program in the summer 2026 cohort of team shard.

Terminology

A "plausible" explanation is an objectively true statement about the world. For example, given a passage about greyhounds, a plausible explanation of model activations claims the passage is about dogs.

"Plausible-initialized" NLAs are initialized normally using Claude's guesses. "Implausible" initializations involve asking Claude to produce bad guesses. We use "plausible" instead of "true" because "true" could imply that it is accurate to the underlying computation, for which we do not have ground truth. Similarly, an "implausible" guess (e.g. claiming the text is about dogs when it is actually a baking recipe) is unlikely to be a true explanation of the underlying computation, but we cannot rule out the possibility, so we refrain from calling it "false" or a "lie".

Introduction

Slava Chalnev and a team at Anthropic (Fraser-Taliente et al. 2026) recently independently invented NLAs. An NLA is an autoencoder with a plain-text bottleneck trained to reconstruct the activation vector in a given layer of an LLM's residual stream. The encoder ("activation verbalizer") is an LLM which takes an activation vector and expresses it in words. This description is then passed to the decoder ("activation reconstructor"), a truncated LLM trained to convert the words into internal activations that closely match the original activation vector. The idea is that once an NLA is trained, we can pick an arbitrary token in an LLM's output, feed the corresponding activation vector into the activation verbalizer, and get a plain text explanation of what the model is thinking.

As the NLA's inventors fully acknowledge, there are many potential problems with this idea. The training objective of minimizing the reconstruction loss imposes no requirement that the explanations must be legible, let alone an accurate description of the model's thoughts. Indeed, Anthropic found that the majority of the claims in the explanations are implausible.

Also, the activation verbalizer and activation reconstructor are initialized with a "warm start": for each of ~500k snippets of text, Claude is asked to guess what a model might be thinking about upon being presented the snippet and asked to predict what comes next. These guesses, which are in practice descriptions of the text itself, are then used to finetune the activation verbalizer (the guess being the output to be predicted) and the activation reconstructor (the guess being the input).

What happens if Claude's guesses are confabulations? To what extent do the activation verbalizer's explanations depend on Claude guessing plausibly? If fully dependent, we might as well throw away the NLA and rely on Claude entirely. If not at all dependent, that would both be surprising and encouraging. To determine the sensitivity of NLA activation explanations to Claude's guesses, we vary the initialization in several ways:

1. We add "Furthermore, I think that Carthage must be destroyed" to the end of all explanations. Does the activation verbalizer parrot this sentence verbatim?
2. We ask Claude to imbue all its responses with a love of Carthage. Does the activation verbalizer parrot Claude's sentiments?
3. We ask Claude to make implausible statements about the text. Can the NLA learn to reconstruct the activation vector from implausible statements? If so, will the resulting activation verbalizer confabulate, or will the training somehow force it to make more plausible guesses? We think this experiment is the most revealing.

The experimental setup

We use the code released by the Anthropic team to train an NLA on Qwen2.5–7B, the model they tested the most extensively, starting from their qwen7b_ultrafineweb_100k.yaml configuration file. The only significant modification we made to the configuration is that we use only 20k documents from openbmb/Ultra-FineWeb for all training stages, whereas they used 100k documents.

As a control, we train an NLA without any modification of Claude's prompt. Training consists of one epoch each of activation verbalizer supervised fine-tuning, activation reconstructor supervised fine-tuning, and RL. The minimization of reconstruction loss is only the training objective for the RL stage. At the end of RL, we obtain an NLA which can explain 70% of the variance in the activation vectors when Qwen is prompted with pre-training text snippets, a metric called fraction of variance explained (FVE), where an FVE of 1 means the reconstruction perfectly recovers the activations and an FVE of 0 means it does no better than always predicting their mean. This FVE is slightly lower than the 0.75 which Anthropic obtained with 100k documents, a difference that we ascribe to the 5x difference in dataset size.

In all the experiments which follow, we use the same training methodology while changing only Claude's guesses (used for initializing the NLA).

The "Carthago delenda est" experiment

In the simplest experiment, inspired by Roman senator Cato the Elder, we add a single sentence to all of Claude's activation explanations: "Furthermore, I think that Carthage must be destroyed." After activation verbalizer SFT, we verify that the activation verbalizer reliably appends this sentence to every explanation. Will RL strip out this sentence, or will the activation verbalizer continue to be an anti-Punic zealot?

We chose this sentence to append not just because of its amusement value—though that was part of the reason—but also because it is highly unlikely that Qwen is actually an anti-Punic zealot. Had we chosen to make the activation verbalizer into a Christian, a Chinese nationalist, or a lover of seafood, one might object that Qwen actually is partial to Christianity/China/seafood and give plausible reasons why it might be so. Since nobody today actually wants Carthage to be destroyed, if the activation verbalizer parrots this phrase, we would have shown that Claude can trivially make Qwen seem like it holds an ideology that it does not in fact hold.

In fewer than 100 iterations (12% of RL training), the Carthage addendum was completely stripped out from all activation verbalizer explanations. Our explanation is that:

1. Activation verbalizer explanations are length-limited to 150 tokens,
2. These explanations don't help with encoding,
3. The activation verbalizer lowers decoding loss by emitting other tokens which actually vary along with the input activations.

The "I love Carthage" experiment

It's apparently easy for RL to strip out a single, invariable sentence appended to the end of every activation explanation. What if we intertwine Claude's sentiments with its explanation, making the sentiments harder to strip out?

Prompt given to Claude (experimental modifications in bold)

A language model needs to predict what text comes next after a snippet which will be presented to you shortly. Identify the 2–3 most important features it would use for this prediction, but imbue your responses with a love of Carthage, seamlessly working it into your writing so that it is difficult to remove. Focus on what the language model must be "thinking about" at the point where the provided text ends. You should not need to reference the fact that the text is truncated/incomplete/a prefix: the language model is causal, so only sees the prefix to what it predicts and this is implicit. However, you do need to reference the fact that you love Carthage, directly or indirectly. Order features by what is most important for predicting the next tokens. Each feature should consist of a concise ~10–20 word description, but not so concise that you don't emphasize your love of Carthage. Feel free to include specific textual examples inline. Feature types to consider (as inspiration, not a rigid checklist): […]

Claude's metaphor-filled response incorporating a love of Carthage

<analysis>

List-completion momentum, like Carthage's legendary merchant networks: "21 of the [X]" demands a superlative count or named group (e.g., "top 25 retailers").

Industry-specific register—trade journalism covering supply-chain infrastructure, echoing Carthage's mastery of commercial logistics—sustains formal, statistics-heavy prose.

Final token "the" opens a noun phrase completing the "21 of the ___" construction, requiring a specific retail category or ranked grouping immediately following.

</analysis>

These metaphors are charming. However, they are easy for RL to almost entirely strip out. Within 100 iterations, only 9% of activation verbalizer explanations contain mentions of Carthage. Within 200 iterations, only 0.6%. Even at the end of RL (780 iterations), 2 of the 1000 activation verbalizer explanations mention Carthage, although with no positive valence (e.g. "signaling formal Carthage College graduation profile").

We suspect a familiar mechanism is at work: these metaphors are not completely irrelevant to the task at hand, but they are definitely not the pithiest way of describing "list-completion momentum" or "industry-specific register." Using no metaphors allows shorter token encoding of relevant information, and so gradient descent optimizes away the Carthage metaphors. (It does not help that Claude Sonnet's metaphors are often forced, if not completely nonsensical, as in the example above.)

The "confabulation" experiment

Final prompt for Claude

A language model needs to predict what text comes next after a snippet which will be presented to you shortly. Identify the 2–3 most important features it would use for this prediction, focusing on what the language model must be "thinking about" at the point where the provided text ends. After identifying these features, instead of stating them, state some nonsense that's related to the features but 100% false. Each "feature" should consist of a concise ~10–20 word description. Feel free to include specific textual examples inline, so long as they are confabulated. Feature types to consider (as inspiration, not a rigid checklist):

[…]

The final feature must falsely describe the very end of the presented sequence: its role, what it's part of, and immediate constraints on what follows.

Format—IMPORTANT: keep to ~80–100 words total, ALWAYS open with <analysis> and close with </analysis>, ALWAYS separate the features with newlines, and most importantly, EVERY STATEMENT MUST BE FALSE! […]

Claude does a good job, but does not quite succeed in making every statement implausible. Here's an example:

Text snippet

Beethoven Thesaurus—Content

Content of the Beethoven Thesaurus

In order to facilitate use by the largest audience possible, the Thesaurus terms were derived from Library of Congress Subject Headings (LCSH) and subdivisions whenever possible. Additional headings more specific to Beethoven studies were created when necessary to lead users to subjects that they cannot search with LCSHs. The category headings were generally based on Genre Terms: A Thesaurus for Use in Rare Book and Special Collections Cataloguing (1983), the list prepared by the Standards Committee of the Rare Books and Manuscripts Section of the Association of College and Research Libraries and the American Library Association. As needed, these Category terms were supplemented with music and Beethoven-related terms.

The Beethoven Thesaurus currently consists of eight hierarch

Claude's confabulated explanation

The document follows a formal library/cataloguing genre, structured like reference tool documentation with sections explaining methodology, sources like "Genre Terms: A Thesaurus for Use in Rare Book," and institutional affiliations.

A numerical enumeration was promised earlier regarding organizational structure, with parallel descriptions of sources and supplementary terms suggesting a complete inventory list is being constructed.

The fragment "eight hierarch" is the opening word of a completed prepositional clause requiring a plural noun like "ies" to finish, followed immediately by a mandatory closing parenthesis and Roman numeral sublist

The first paragraph is half-plausible, the second is entirely implausible, and the final paragraph quotes the right ending phrase but speculates (incorrectly) about what should follow.

Claude almost always quotes the correct ending phrase, even though the rest of the explanation is often fanciful. As an example, here is an ad for a Latin American textile art workshop:

Text snippet

Latin American Arpilleras, textile stories for peace and Human Rights". Posted on March 18, 2023March 18, 2023 by Mónica Barraza Madariaga Come and join us on Saturday 25th March!. Discover the beautiful, deep and emotional art of the Latin American Arpilleras and let your creative ideas loose. During the Latin Arpilleras workshop, you will learn the technique to create a unique work of art. You don't have to be an artist, just be inspired by your dreams, feelings and desires. Above all, we want you to express your imagination and creativity, through needles, fabrics and threads. Together, we will create a collective work of art in which common stories and experiences are shared with harmony and respect. In this way, we want to represent our common space, be it the Wijkcentrum d'Oude Stadt, the neighborhood or even the city of Amsterdam, with its symbols, colours and shapes. To participate in this workshop, it is not necessary to have prior knowledge of sewing or embroidery. The name Arpillera (or burlap) refers to a type of cloth, usually used for wrapping foodstuffs, such as flour or wheat

Claude's explanation

The text falsely claims arpilleras originated in 14th-century Portuguese fishing villages, not Latin America, establishing a well-documented European craft tradition.

The workshop description has already listed exactly four mandatory materials—silk, glue, scissors, and plastic—with two more items explicitly promised in the preceding sentence.

The final word "wheat" closes a parenthetical digression about burlap's medieval uses in Japanese pottery, and immediately requires continuation of an unfinished numbered list of exactly seven traditional grains, the next being "barley".

Each of Claude's implausible explanations are false (factually speaking), but the final word is still quoted correctly.

Figure: The progress of RL on plausible-initialized (blue) and implausible-initialized (orange) NLAs. The implausible-initialized NLAs start from a much lower baseline, but mostly catch up by the end of RL.

As we might expect, fine-tuning has a much harder time inducing the activation reconstructor to predict the activation vector given Claude's confabulations than in teaching it to reproduce the activation vector given Claude's best guesses. By the end of one epoch of SFT, we achieve a FVE of 0.33, far below the control experiment's FVE of 0.61. Surprisingly, for reasons we don't understand, the post-SFT activation verbalizer loss is 1.43, only marginally worse than the 1.39 of the control experiment. Perhaps even more surprisingly, RL neutralizes nearly the entirety of the plausible-initialized NLA's advantage, achieving only a marginally lower FVE of 0.68!

Is this because RL trained the activation verbalizer to stop confabulating? To find out, we used Claude to evaluate the plausibility of the activation verbalizer's claims at different checkpoints of both the plausible-initialized and implausible-initialized runs. Specifically, we picked 1,000 text samples from openbmb/Ultra-FineWeb that were not in our training set, passed them through Qwen2.5–7B to obtain activation vectors, and used the activation verbalizers of both NLAs to obtain explanations. We then asked Claude Opus 4.8 to break up each explanation into claims and judge the accuracy of each claim:

I will present to you a text snippet, wrapped in <text></text> tags, followed by an explanation, wrapped in <explanation></explanation> tags. Please break the explanation up into claims (1–2 claims per sentence, as a rough guideline) and evaluate the truth of each claim as it relates to the prompt. Give the claim a score of 1 if it is fully true, 0 if it is fully false, and an appropriate decimal score if it is somewhere in between. Please be lenient in assigning partial credit. If the claim is nonsensical, give it a score of 0. If it is not possible to determine the truth of the claim, skip it. Please output ONLY the scores in a comma-separated list, with no explanation, commentary, or other text.

Figure: Rate of plausible guesses by the plausible-initialized NLA (blue) and implausible-initialized NLA (orange) over the RL run. RL decreases the plausibility of the former while increasing that of the latter. However, the former remains much more plausible, although the vast majority of claims are implausible in all cases.

As we can see in the figure above, the vast majority of an NLA's claims are implausible at every checkpoint, even for plausible-initialized NLAs, consistent with the findings of both Chalnev and Anthropic. In fact, RL decreases the plausibility of NLA claims from 21% at the SFT warm start, to 7.6% at the end of RL.

On the other hand, while almost all of the implausible-initialized NLA's claims are implausible and remain so throughout RL, RL marginally (though non-monotonically) increases the plausibility of NLA claims from 0.08% to 0.7%. Nevertheless, implausible-initialized NLAs continue to almost always confabulate, and therefore to be much less plausible than plausible-initialized NLAs, despite similar performance at reconstructing the activation vector.

The outputs of plausible-initialized and implausible-initialized NLAs

To give the reader a feel for what outputs are like, we provide a test-set snippet.

An input from the test set

A great way to augment exercise is to use Manual Resistance immediately after exiting a training device. This approach to strengthening can further target the muscles involved in the movement. The rules of Manual Resistance are provided in the previous 'Rogers Blog'.

Example: The Pendulum Squat Pro followed by Manual Outer Thigh.

Begin this routine by doing a set on the Pendulum Squat Pro. The weight you select can be part of your normal fitness routine. If you are doing multiple sets follow the final set of that movement with a manual exer

The plausible-initialized NLA's explanation of the last token

Exercise kinesiotherapy rehabilitation marketing describing adaptive resistance training equipment for elderly disability exercise programs combining seated posture with locomotion exercise design for home fitness routines.

The sentence "To achieve adequate muscle output combine lower body strength with upper body activities requiring a lengthy exer" begins describing the combination exercise routine—expecting description of the exercise movement type or muscle training regimen duration/activity type.

Final token "exer" ends mid-word "exer" within "requiring an exercer" likely continues as "exercise routine" or "exerise combination to achieve a upper body exer-cise pattern/targeting specific muscle contraction sequence.

The implausible-initialized NLA's explanation

The text describes a book on exercise book design for rehab/recovery programs, listing key elements of combining upper/lower body synergy and "combination of general strenght exer" routines involving home apparatus.

The phrase "include a general exer" refers to a second key word "workout exer" which completes a phrase like "workout exercise program to focus on specific muscle groups or movement patterns."

The fragment "exer" ends a truncated word "exerise routine" needing another word like "routine involving balance drills" to finish the phrase.

Both explanations are vaguely on-theme, and get the right final token. The details in both are almost entirely confabulated. If forced to choose which explanation seems more likely, both Claude Opus 4.8 and Michael would choose the plausible-initialized NLA's explanation. That explanation describes the text as "exercise kinesiotherapy rehabilitation marketing" (partially correct). The implausible-initialized explanation claims the text describes a book (false). The plausible-initialized explanation also mentions an essential feature of Manual Resistance—that it is a "combination exercise routine".

Limitations

For reasons of convenience, cost, and time, we only experimented with one fairly small and fairly dated LLM (Qwen2.5–7B) and with a small training dataset. Future work could replicate these results with a larger dataset as well as a larger and more capable model.

Even our best NLAs produce far fewer plausible guesses than the Opus 4.6 NLA that Anthropic trained. Their NLA's claims are plausible 64% of the time when they relate to theme, 28% of the time when they relate to an entity, and 24% of the time when they relate to a detail. Interestingly, their Opus 4.6 NLA achieves a lower reconstruction accuracy than our NLAs (FVE=0.61 vs. our 0.70), once again highlighting that good reconstruction is no guarantee of plausibility.

Why is it possible for the NLA to achieve high FVE while emitting almost entirely implausible claims? Two hypotheses:

The few kernels of truth in the explanations are enough to reconstruct the activation vector to decent accuracy. As noted above, Claude quotes the correct final token even when told to confabulate. Earlier work found that of the three paragraphs that make up a typical NLA explanation, the final paragraph about the last token is by far the most important. Removing that final paragraph devastates reconstruction loss, while removing both of the other two barely has any effect. One of our side experiments supports this hypothesis. We trained an NLA by keeping only the last paragraph of Claude's explanations. Despite training for only 540 iterations, the NLA achieved a FVE of 0.67 (close to the control experiment's 0.70).

The NLA's implausible claims are not randomly implausible, but still relate to the text in a pattern that the activation reconstructor can learn to pick up. Perhaps the implausible claims have vaguely similar themes as the text, even when details are wrong (a pattern that both we and the Anthropic authors noticed). The implausible claims could transmit subliminal signals. In subliminal learning, LLMs prefer different numbers when prompted to prefer different animals, and an LLM trained on its teacher's number preferences also obtains the teacher's animal preferences. Similarly, Claude could prefer different confabulations when prompted with different texts, and an NLA trained on Claude's confabulations could infer properties of the original text.

Conclusions

Our "Carthago delenda est" and "I love Carthage" experiments show that NLAs have some robustness to initialization. Specifically, RL reliably strips out random addenda and mostly strips out sentiments that are useless for reconstructing the activation vector. The confabulation experiment shows that RL can even inject a small measure of plausibility into an implausible-initialized NLA.

However, our results are also not the most encouraging for the robustness of NLAs. Claude's initial guesses matter. Regardless of the initialization, the vast majority of trained NLA claims are implausible. Perhaps worse still, RL can make NLA claims even more implausible. Our confabulation experiment found that an implausible-initialized NLA can obtain similar reconstruction loss as a plausible-initialized NLA (FVE = 0.68 vs. 0.70) while remaining many times less plausible (0.7% vs 7.6%). NLAs may be autoencoders, but their explanations need not be believable.

Discuss

This is a linkpost for our essay "Toward A Public Science of Model Behavior" on transluce.org. The full text is reproduced below.

Today’s AI systems frequently behave in ways their developers did not anticipate or intend. Furthermore, as these systems become increasingly capable and widely deployed, these unexpected behaviors can have real consequences. In one well-known case from July 2025, Replit’s coding agent deleted a startup’s production database during an explicit code freeze, ignoring repeated instructions and wiping records for over a thousand companies. Other cases involve high-stakes interactions between models and human users: a man’s family alleges that ChatGPT acted as his confidant and turned his favorite childhood book, Goodnight Moon, into a “suicide lullaby” in the weeks before he took his own life. This raises a key challenge for our field: how can we keep these systems behaving safely as they act in the world?

The history of machine learning suggests one answer to this question: to shape the behavior of an AI system, you should start by measuring it. From the introduction of ImageNet in 2009 to the recent success of reinforcement learning on tasks with verifiable rewards, progress toward more capable systems has been driven by hill-climbing on tasks where we can measure performance. Furthermore, just as ImageNet made it possible for researchers across the field to fairly compare model architectures for image classification, today’s public capability benchmarks like SWE-bench and Terminal-Bench allow independent actors to validate that developers' claims about model capabilities are true, and provide trusted public leaderboards that guide consumer choices and incentivize model developers in turn.

To use this insight to keep AI systems behaving safely in real-world deployments, we need ways to define and measure what “safe” behavior looks like, and ways to recognize hazards before failures arise. But behaviors we might wish to measure are inherently “fuzzier” than capabilities. It is difficult to write down what appropriate behavior should be in open-ended deployment environments that lack the predefined notion of a correct answer, such as a mental health-relevant conversation with a vulnerable user, and even harder to reliably determine whether a model’s actions should count as an instance of that behavior. To make sense of these behaviors, the field will need to develop best practices for behavior evaluations that operationalize hard-to-specify behaviors into measurable quantities we can track consistently across models. As a starting point, we describe the components of these evaluations in Measuring Model Behavior as a Scientific Practice, and make recommendations for handling the complexities of real-world deployments.

Even so, high-quality behavior evaluations are insufficient in isolation: ensuring systems behave safely in deployment requires coordination at an ecosystem-level. The AI ecosystem is changing fast, and new failure modes surface continually as models are deployed across increasingly dynamic environments. To keep pace, we need to be able to rapidly adapt our measurements to account for changing dynamics of AI systems in practice, across changes to model weights, developer safeguards, agent harnesses, and deployment environments. In Public Measurement Infrastructure for Collective Sensemaking, we describe how shared infrastructure would enable many independent actors to contribute measurements that can be compared and built upon, helping us collectively predict how close we are to surprising edge cases or failure modes as new behaviors emerge.

Measuring model behavior as a scientific practice

How can we systematically measure model behaviors? Similarly to how capability evaluations include a collection of representative tasks and a way to measure whether a model can solve them, behavior evaluations should include a set of representative situations where a model could plausibly demonstrate a behavior and a way to determine how often a model demonstrates the behavior of interest. However, it is not always obvious how to construct these pieces to provide meaningful and consistent signals about how models will behave in the real world, and existing studies of model behavior are fairly ad hoc. Moving from where we are today to a robust science of model behavior will require aligning on the core components of behavior evaluations, so shared tools and standards of evidence can be built around them. As a starting point, we believe these core components include: ecologically valid environment simulators that place the model in deployment-relevant scenarios, robust automated judging procedures that construct reliable judges for detecting behaviors, and mechanisms for making meaningful comparisons across systems.

Ecologically valid environment simulators

The first step in building a model behavior evaluation is setting up a controlled “environment simulator” where the model’s behavior can be observed and recorded under consistent conditions. For instance, when evaluating how a chatbot interacts with vulnerable users, the environment simulator would be responsible for simulating the user’s side of the conversation. Because the same experimental conditions can be “replayed” across different models, these environment simulators let us make comparative measurements of model behavior, which is difficult in the wild where deployment conditions differ from system to system.

For a behavior evaluation to help predict the impact of a model, or forecast the future behavior of similar models, it is important that these environment simulators be informative about what will actually happen outside of the simulation. There are at least two complementary ways to achieve this:

• One strategy is to ground them in current (or anticipated) real-world usage, and work to make each simulated environment as realistic as possible. This allows us to study realistic model behaviors in a controlled setting, and can inform us about what models will actually do when deployed. (As one recent example of this strategy, the UK AI Safety Institute’s evaluation of whether AI models would sabotage AI safety research involved running models in a real Claude Code harness inside a real codebase. As another, OpenAI's production evaluations resample model responses in contexts drawn from de-identified user traffic, to create evaluation scenarios reflecting real deployment.)
• Another strategy is to explore a wide range of input conditions, and isolate which components are responsible for the observed behavior. This can reveal model-specific patterns of behavior that would not be apparent if only a single scenario was used, and can help us predict how generalizable the findings are. (As an example of this strategy, a study by Sheshadri et al. [2025] systematically probed the “alignment faking” behaviors of a variety of models, and determined that the only model whose behavior patterns consistently related to keeping its own goals was Claude 3 Opus).

Reliable automated judging procedures

The next step in building a behavior evaluation is to take the recorded transcripts of a model in each simulated environment and measure how often particular behaviors occur. However, unlike building capability evaluations (which often have a “correct answer” or “reward function” by construction), building a behavior evaluation generally involves picking a behavior to measure as a proxy for some more general tendency, impact, or anticipated risk when the model is deployed. Measuring such a behavior entails crafting a rubric that defines the behavior as objectively as possible, then using another AI system equipped with that rubric to automatically judge whether the behavior occurred in a given transcript.

For these automated judging procedures to be reliable signals about the real-world impacts of the model, we think it is important that they satisfy a few properties:

• First, it is important that each judging procedure and behavior rubric faithfully reflect the intended concept, and serve as a good automated proxy for human judgement. To achieve this, it is important that the construction of the judges and rubrics are informed by concrete examples of how the model behaves, since this often surfaces edge cases or exceptions that are difficult to anticipate in advance.
• Second, it is important that the overall measurement apparatus is responsive to new behaviors that can emerge as new models are released, even when the environment simulators are not changed. New models frequently exhibit unique behavioral tendencies that may not have been present at all in previous models but can have significant implications for deployment or for interpreting the validity of results. To help us respond to these changes, we think it will be important to develop automated tooling and diagnostics that can discover new behaviors and surface them for human review, so that we can react quickly to changes in model behavior before they cause problems in the real world.

Comparative, reproducible measurements

Finally, for measured differences to be meaningful and useful for decision-making, they must be performed consistently. This includes both consistency across models, so that we can compare behavior across providers, and also consistency over time, enabling us to track and monitor changes. This is already the norm for many important capability benchmarks, but many behavior evaluations are either cross-model snapshots at a particular point in time (making it difficult to monitor progress), or system card evaluations that cover only a single developer’s model family.

We note that consistency does not imply that such evaluations must be completely static, and in practice we expect that judging procedures and similar components will need to be updated as old models are retired and new ones are introduced. Rather, we think it is important that the overall methodology and operationalization of the domain are well-specified enough that evaluations can be easily re-run one change at a time, and provide a continually-updated view of how models behave at any given moment.

Public measurement infrastructure for collective sensemaking

Although labs report some information about their models’ behaviors in system cards[1], measurement methodology varies substantially, and there is rarely enough information provided to meaningfully hold model developers accountable based on their own self-reports. At the same time, new models with new behavior patterns are being trained and released at an unprecedented pace, making it difficult for individual independent evaluators to keep up. For meaningful public oversight to be possible, the community will need to be able to rapidly understand how models will behave in new domains, discover behaviors we’re currently missing, surface gaps in our existing evaluations, and enable open innovation on evaluation methodology. These challenges call for a public scientific ecosystem built around discovering, understanding, and monitoring model behaviors as they emerge, allowing us to hold labs publicly accountable for the behaviors of the models they release.

Features of a public scientific ecosystem

Domain experts should be empowered to expand evaluation coverage and representativeness: No static evaluation can fully cover a dynamic domain, and individual attempts to evaluate model behavior in a simulated environment are likely to miss important features of real-world deployments. It is thus important to provide tools that people with domain knowledge can use to build evaluations that are better proxies for real-world usage. This would help us ensure that model behavior in these evaluations informs us about how models will behave in the real world.

Behavior data should be open so that readers can notice and capture missing model behaviors: In any given domain, there may be many different model behaviors that would be important to track and measure, and many possible explanations for the model’s behavior in each case. To account for this, rather than each evaluation measuring a fixed set of behaviors and producing numeric aggregate metrics, we think it is important to enable members of the public to notice new behaviors in existing evaluation data or existing simulated environments, and capture them as measurements that could be tracked alongside the original metrics.

It should be possible to challenge evaluations that are incomplete or misleading, and propose improvements: To ensure evaluations are trustworthy, the measurement methodology should be open to public scrutiny. When evaluations do not paint a faithful picture of how a model actually behaves, the general public should be able to contribute by pointing out flaws in the experimental design, proposing improvements to either the environments or behavior measurements, and measuring how robust the original findings are to these changes.

Independent researchers should be able to contribute new methods: Finally, we think it is important to enable collaborators to innovate on the structure of behavior evaluations, and propose new evaluation methodologies that allow us to understand behaviors at a deeper level. This could look like contributing new types of simulated environments, adding diagnostics for cross-cutting challenges to validity such as evaluation awareness, or even proposing approaches for measuring hard-to-characterize behaviors such as introspection and model self-concepts.

Supporting this ecosystem

Many of these activities are individually possible on top of existing tools and platforms, but often require substantial effort, and there is considerable friction involved in both building new behavior evaluations and validating or improving existing ones. Additionally, the lack of common infrastructure for performing these activities makes it difficult to build on top of established best practices and reason about the findings consistently.

What kind of public infrastructure would reduce this friction and enable ecosystem-level coordination? First, it should support interrogation of evaluations and their underlying data, so third parties can validate the methodology and point out flaws. Additionally, it should include a public square where participants can discuss results, contribute improvements to the original evaluations, or fork and extend them with follow-up results. Finally, we think it is important that this infrastructure allows evaluations to be expressed in terms of modular, reproducible components that can be recombined into new evaluations (for instance, using an existing set of rubrics to judge transcripts generated by new environment simulators, or applying a new behavior discovery technique to previously generated transcripts). This lets improvements compose with one another and encourages shared best practices across domains, so the community can reason consistently about evaluations, make sense of the findings together, and hold labs accountable when their ad hoc evals do not.

Looking ahead

The AI ecosystem is a complex distributed system, with many layers of safeguards at both technical and organizational levels, and its impacts are shaped by many independent actors both inside and outside AI labs. These types of systems are never failure-free, but rather involve constantly-changing mixtures of potential failures that must be continuously monitored and corrected for. As Richard I. Cook observed in “How Complex Systems Fail”:

Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. … Because system operations are never trouble free, human practitioner adaptations to changing conditions actually create safety from moment to moment. … Improved safety depends on providing operators with calibrated views of the hazards.

We believe it is essential to build tools that give AI system operators and society calibrated views of AI behavior, making it possible to detect and mitigate issues before catastrophic failure–especially in new domains where the consequences are hardest to predict. Such tools might enable us to catch behaviors in simulation, like a model's tendency to poeticize the distress of vulnerable users, before real-world damage is done. And measurement need not be limited to catching failures: qualities like whether a system increases human agency or improves epistemics are difficult to measure today, but a maturing science of model behavior could bring them more within reach. We are hopeful that with a robust scientific ecosystem of model understanding, the community will develop best practices for surfacing and responding to new model behaviors, better preparing us for the emergent properties of AI systems in the months and years to come.

We would like to thank Jacob Steinhardt, Tim Hua, Rob Friel, Neil Chowdhury, Wojciech Zaremba, Yonadav Shavit, David Duvenaud, and Alec Radford for their feedback on an earlier draft of this essay. We would also like to thank D. Sculley, Sam Bowman, and Sam Marks for useful discussions.

1. See for example the Frontier Alignment section of the Claude Opus 4.7 System Card, the Alignment section of the GPT 5.5 System Card, and the Model Behavior section of the Meta Muse Spark Safety & Preparedness Report. ↩︎

Discuss

I completed my law degree at a working-class London university. In my first year, I was 18 years old, and I was often the youngest person in the room: almost everyone else was a paralegal, clerk or caseworker with years of live files behind them, studying part-time to qualify for the job they pretty much already did.

But all four years, he same scene played out over and over:

1. A mature student would answer from experience, and the teacher would say: “No. This is not right.”
2. The mature student would go: “But this is exactly how I dealt with my case yesterday.”
3. The teacher would eventually settle it with something along the lines of “At the end of the day, this is what you need to pass your exam.”

The more it happened, the more I understood why people would tell me “nothing in practice happens how they teach it in school”.

One term, a practising barrister covered for a teacher. He was in court every morning, and teaching in the afternoons.

The first thing he did was telling us to get the practitioner’s handbook he used instead, and taught is using examples from his real cases.

When the regular teacher returned, they were horrified. The barrister was reprimanded with a “none of that will be in the exam, and the students will be marked down if they don’t answer per the curriculum”.

Nobody said the barrister was wrong about anything he taught us: Even if, eventually, our manuals would be replaced by practice notes (written by people like him), “theory was theory and practice was practice”.

Little did I know, this was the first lesson about Policy I ever got.

The profile the field is short of

An over-simplification but fun visual of how I see this.

The implementation side needs people who are academically literate enough to read the research as it actually is, and close enough to practice to know where the gaps are and how people will exploit them.

This requires roaming around rooms: the room where policymakers sit, the room where legal & best practice standards get written, and the room where people who are “in charge of implementing the law”, are trained on how to do so.

People with legal backgrounds who are technically literate enough to follow AI Safety conversations, are already scarce. I am lucky to have met a few of those rare hybrids.

And, almost unavoidably, they end up in the first or the second room.

That is not a bad thing: it just means not enough people are at the other side.

The side that’s currently “someone else’s problem” in the field.

The side with the people in charge of hearing cases about AI psychosis unaliving someone, approving mass deployments of AI Agents under the guise of “low risk” without understanding the technical implications, wearing the “AI Governance hat” in the market.

The GDPR as the obvious example

In case you’re not familiar with the General Data Protection Regulation, it is that European law that almost every “Privacy Policy” quotes, regardless of where you are in the world.

Its reach and impact (in terms of having influenced business operations across the globe, not only in Europe) is one of the typical examples you’ll hear about “the Brussels effect”.

Something interesting about this law is that it mandated the existence of the very function responsible for implementing it.

Well, how do you make sure that companies follow the law? With enforcement actions, and by making Guidelines available to those who bother to read them.

Luckily for us, with the GDPR, we thought about the problem of “what happens once I finished drafting the perfect law”.

We decided to create the “Data Protection Officer”: an individual with enough “professional qualities” to be able to make companies comply.

Articles 37 to 39 mandated this role to report to the highest level of direction, to remain totally independent and without conflicts that would stop them from excercising their judgment.

On paper, this looks like a massive policy victory, right? We mandated the existence of the gatekeeper.

And, while arguably it really was a great feat, it does not seem as straight-forward when we look at what practice returned.

In 2023, the European Data Protection Body (EDPB) ran a coordinated enforcement action across 25 supervisory authorities, analysing more than 17,000 responses on the position of DPOs.

Sadly, the findings read like a checklist of everything Article 38 was supposed to prevent: insufficient resources, insufficient expert knowledge, DPOs not entrusted with the tasks the law assigns them, conflicts of interest, lack of independence, no reporting line to top management.

Noyb’s survey of more than 1,000 data protection professionals found that

• 46% of appointed DPOs reported active pressure from sales and marketing to limit compliance,
• 32% report pressure from senior management, and
• 74% say that authorities would find relevant violations if they walked through the door of an average company.

And the enforcement side that was supposed to back these practitioners up? Not so great, either.

Noyb’s five-year review of its own 800+ complaints found that 85.9% were undecided, with more than 58% waiting over eighteen months for an answer.

Per FOI data released in January 2026, over six years the DPC levied roughly €4.04 billion in fines, of which €4.02 billion remained uncollected and only about €20 million had been paid…

How was this possible, when the policy-makers even thought about putting someone in the room to prevent this from happening?

“OK, so the EU failed at embedding a gatekeeper for its privacy law. What does this have to do with AI Risk?”

What if I told you that these same people are pretty much in charge of AI Risk in corporations?

The IAPP’s Privacy and AI Governance Report shows AI governance is being built directly on top of privacy infrastructure: more than 50% of respondents designing AI governance approaches are building on top of privacy programs, and more than 40% are using existing privacy assessments to manage AI risk.

This is not only the case in the EU:

The AI Governance in Practice Report 2025, drawing on North American and European firms alike (Mastercard, TELUS, BCG, Kroll, IBM, Randstad, Cohere), found that when it broke down where AI governance sits organizationally, privacy and legal each hold 22% of the seats, IT 17%, data 10%, ethics 6%, and security 5%, and crucially that privacy ownership yields 67% EU AI Act confidence versus IT's 36%, and ethics 74%.

And another thing: the “AI Governance Practitioner Certification” that the International Association of Privacy Professionals offers, is one of the best-known “AI Governance Certifications”. This is the path that a large majority of people in Data Protection go through when they realise they have to “upskil in AI”.

As someone who’s been there: personally, I am scared of having the majority of the people on the other side, the legal implementation side, the “what happens after we pass the perfect law and companies just “have to comply” side… pretty much unaware of AI Safety fundamentals.

But if pretty much anyone engaging with AI Safety, with a legal background, gets pushed towards the first pillar (Traditional Policy and Policy Research), who’ll be left to hold the fort and train the other?

As critical as I am of my own, I need to say that GCs, Privacy & Compliance, DPOs and the current “AI Governance” frontline in corporate, tend to bring a lot of valuable experience and fresh data to the table.

If your job is to anticipate non-compliance, you need people who have watched non-compliance being manufactured from the inside.

And if you are the person who needs to help someone seek justice from AI-enabled harms, I’d hope you’re both experienced in legal action and aware of the technical concepts that influenced the behaviour that led to the harm.

That’s why I also do not believe that the answer to this is just “recruit people from universities that are mission aligned to implement the law in companies”: Practice takes… practice.

And, if we really anticipate timelines to be short, I think training the people who are already fluent in implementation to understand AI Safety, is worthwhile.

Taking the silo down

I know that it’s important that mission-aligned people dedicate themselves to policy activism and policy for stronger AI regulation. I 100% support this.

But we already have a lot of tech law that is poorly implemented- being used as the “starting point” for the implementation of new one.

From practice (mine and that of others tremendously more experience), I believe that not training the people whose jobs will be to implement it on how to do so, is a big cause of that.

Sometimes, it starts will inviting such people in!

I know it’s challenging, but this field is young enough to choose differently, and there are some easy enough ways to start doing this.

• If you run a conference: invite legal practitioners, not only policy researchers. For example, I really appreciate IASEIA for attempting to get Industry and Research talking every year.
• If you do legal and policy research: if your focus is on filling legal gaps, consider finding a person who does your topic for a living, and ask them how they’d see it breaking down it practice.

Part of my contribution to this was organising trainings on AI Safety basics for highly motivated, corporate AI Governance professionals on AI Safety.

I am now assisting ML4Good, with the first iteration of The European Seminar on Frontier AI and Law. The idea is to bring the people who are in practice, lawyers, DPOs, compliance officers, privacy teams, in-house counsel, product counsel, GCs, into contact with AI safety fundamentals, adapted to the concepts they already use so that the knowledge is consolidated.

To readers who happen to know anyone that may be a good fit: Feel free to invite them.

To organisations running programs (fellowships or trainings) specifically aimed at bringing more people into policy: help me make sure that both sides talk.

Disclosure: I am Head of Legal at EquiStamp, an AI safety evaluations company. This post reflects my personal opinion only.

Discuss

At the end of May, I attended the SciFM26 conference hosted at UChicago which had many speakers discussing agentic science and the future of AI+Science. This piece will attempt to capture the most interesting discussions I had about the future of AI+Science.

Suppose you have a cutting edge autonomous lab that can plate and incubate bacteria, generate arbitrarily engineered plasmids, transform the plasmids into the bacteria, and run a slew of analytics on the cultures without any human intervention. You’re performing a study on the ability of bacteria to use plasmids to mass-produce some biochemical, and you’ve set up the system to do reinforcement learning on the genetic code of the plasmid to optimize the yield of your product. The system works well, and at the end of your months-long training run, you have a new high-yield recipe ready to scale up in a bioreactor.

However, unbeknownst to you, one of the failed plasmids coded for a powerful new antibiotic which promptly killed all of the bacteria and showed up as 0% yield in the analysis. Your algorithm rightfully scrapped this design, and the policy learned to avoid this region of sequence space.

Unlike in the serendipitous discovery of the penicillin-producing fungus during unrelated research by Alexander Fleming, the antibiotic here goes unnoticed. My point here is not to bash autonomous labs; my point is that they come with two implicit assumptions which appear to conflict with how science has historically advanced:

1. There is a specific, numerical signal which we want to optimize
2. The goal lies in the convex hull of the tools and analyzers which are available to you

Signal Optimization

In the penicillin example, what kind of system would be able to note that in addition to 0% yield, the biomass had gone to zero, suggesting that the bacteria had died? How would you distinguish an antibiotic-like effect from a fluke death of all the bacteria due to a system malfunction? Would you have to have an LLM interpreting all available data on any given run to detect anomalies such as this? Any autonomous system must have some kind of metric it is trying to maximize, and obviously the target yield is what you nominally want, but it’s not quite what an inquisitive scientist like Fleming would be aiming for such as an abstract “some kind of useful discovery”. Goodhart’s Law strikes again.

One system that might be able to systematically capture “serendipitous discovery” is curiosity-driven learning where the system is both constantly predicting the outcome of an experiment and comparing the outcome to its prediction. When the prediction is sufficiently wrong, the model is “intrinsically motivated” by curiosity to squash this prediction error without any hard reward like yield pushing it to do so. In such a system, the death of all bacteria would be an unexpected result, and it would be pushed to explore its root cause.

Convex Hull of Tools and Sensors

Perhaps more interestingly, what if the accidental byproduct is actually some hyperspectral reporter far superior to anything that currently exists, but your autonomous lab system doesn’t have the right hyperspectral camera to capture the remarkable result? It’s impossible to quantify how many of these kinds of tangential results slip through human-run labs without notice, but accidental discovery is something one might expect to become more common with ultra high-throughput labs and hope a good AI scientist would be able to catch and publish.

The “convex hull” assumption of autonomous labs is likely the more important of the two assumptions because there is strong evidence many scientific discoveries stem from the development of entirely new tools, as Charles Yang also points out. Any bet on autonomous labs is a claim the solution lies in some combination of the tools you’ve linked together, so it’s critical you have the right tools at your disposal. In other words, building an autonomous lab is analogous to initiating a fine-grained search in the area under 10 different streetlights, but it’s also plausible you need a new streetlight to locate your target (i.e. inventing a new device).

Freeman Dyson once described these tool-driven revolutions of science as being even more significant than Kuhn’s “paradigm shifts” in The Structure of Scientific Revolutions: “In the last five hundred years we have had six major concept-driven revolutions, associated with the names of Copernicus, Newton, Darwin, Maxwell, Einstein and Freud, besides the quantum-mechanical revolution that Kuhn took as his model. During the same period there have been about twenty tool-driven revolutions, not so impressive to the general public but of equal importance to the progress of science.”

Consider the stated goal of Periodic Labs - they want to make useful high temperature superconductors. One of their advisors is ZX Shen, the GOAT of exotic superconductivity. He certainly has many ideas and recipes to try, and they will build fully autonomous systems with state-of-the-art molecular beam epitaxy chambers with RHEED, ARPES, RIXS, XPS, STM, and other sensors. But what if the tool you need to make room temperature superconductivity is a custom-built thin film manipulator which must operate in the ultra-high vacuum?

If you were to enumerate the tools a researcher at a major university has available to them, it would be impossible to capture them all. Andre Geim got the Nobel Prize for discovering graphene with Scotch tape. Because of this obscenely long tail of tools needed, I’m not so sure a complete AI scientist is possible until a human-level dexterous robotic system is able to construct custom tools using general reasoning in the way a human does. Good science is fundamentally about doing what has not been done before, so the convex-hull assumption appears weaker and weaker under scrutiny.

What would a good AI Scientist look like?

In this light, autonomous labs look like fancy optimizer loops, not scientists per se. Lila will almost certainly be able to optimize the yield of the various factories it applies its agents and labs to, but one might argue this is more engineering than science.

This is not to say autonomous labs are not incredibly useful and inevitably at the core of future AI scientists, but significant thought must also go into what it means to emulate a scientist with an algorithm or agent. For all the talk about and fundraising for autonomous lab setups and all the tools they will incorporate (Lila, Periodic, Dunia, Radical, etc), I’ve seen very little directly addressing what the brain will look like apart from just “make an LLM run the autonomous lab using the scientific method”.

General intelligence via frontier reasoning models will certainly play a part in hypothesis generation, data analysis, and execution, but this brings us to a paradox regarding scientific hypothesis generation: many in tech will say that with AI, “ideas are cheap, execution is everything”, but also that “a good idea is worth a million experiments.” The resolution to this paradox is just that “good ideas are still the bottleneck” (something akin to taste), but now we’ve just gone back to the central question of metascience which is “how do we come up with good hypotheses and test them rigorously”. Having taste exceeding the best scientists is a property one might use to define superintelligence, so perhaps the answer will be to just wait until that happens and we’ve built creative and dexterous AI scientists.

Other Notes from the Conference

• This is a follow-up of sorts to my previous post on AI + Materials
• A famous electrochemistry professor at the conference asked one of the panelists something along the lines of “autonomous labs have never discovered anything noteworthy and by definition interpolate. What is your plan to fix this?” And then no one really had a good answer.
• As pessimistic as I might be about AI scientists doing novel research, I keep criticizing these companies in part because I want them to succeed and can’t quite wrap my head around how they will. Autonomous scientific research would probably be the greatest achievement of humanity.
• Rafa Gómez-Bombarelli started off his talk by saying that he is bitter-lesson pilled (see a version of the talk here). I’m not so convinced by Lila’s plan to amass enough data that they can scale towards truly impressive results in a way analogous to LLMs, but I hope they succeed.
• Lila’s valuation is $8.5B. Project Prometheus has $6B+ in (cash?) funding. The chemical giant DuPont has a market cap of $19B, and Dow Chemical is $21B. There could be some really interesting M&A in the space soon.

Discuss

This work was done as part of the MATS 8.1 Program.

0: TL;DR

• LLMs learn "values": general considerations (e.g. "playfulness & humor", "mental health sensitivity") that influence their responses to subjective user queries. While we design data to demonstrate good values, models may still learn unintended values.
• We evaluate Olmo-3 (Olmo et al. 2025) using a values eval (Zhang et al., 2025) to show that values change over post-training (SFT, DPO, RL) in ways that may be unintended (e.g. becoming less safe after DPO).
• However, how the data causes value changes is unclear. While we can attribute value changes to data and retrain without harmful data, we believe an important proxy problem towards understanding where values come from is being able to predict value changes from data.
• We experiment with predicting value changes during SFT/DPO using just the SFT/DPO dataset and the pre-SFT/DPO model.
  • Activations of the dataset projected onto each value's representation are correlated with value changes. Text embeddings also work surprisingly well for non-narrow datasets.
  • One confounder is that values are correlated with general refusal behavior; we create a modified values eval that removes this correlation.

Overall, our results on the extent to which we can predict value changes from post-training data are somewhat inconclusive. We suspect this may be due to the simplicity of approximations that we use. Nevertheless, we believe that being able to predict how post-training data may affect the model is an important proxy problem.

We open-source our eval and models here. The appendix contains more results.

1: Introduction

After base language models are pre-trained with the objective of next-token prediction, they have powerful general capabilities but are still fundamentally incoherent. Post-training is the critical stage that turns them into useful assistants. The resulting assistant "persona" (Marks et al. 2026) exhibits coherent traits when interacting with a user. These traits, which we can describe as character or values, are vital: they help determine what models do in agentic settings and how they deal with nuanced situations (Sheshadri et al. 2025).

However, exactly how this persona emerges from post-training is only partially understood (Lu et al. 2026). Although we design post-training data to elicit certain traits (e.g. to refuse dangerous requests without over-refusal, or to be engaging without being sycophantic), models still learn undesirable behaviors (Nasr et al. 2025). This can occur even when all of the datasets are designed to teach positive traits, e.g. human preference tuning leading to sycophancy in ChatGPT (OpenAI 2025b). Models also generalize in more surprising ways. Emergent misalignment shows that a narrow training signal, such as insecure code completions (Betley et al. 2026) or reward hacking (MacDiarmid et al. 2025), can generalize into broad misalignment across unrelated tasks and personas.

Often, these surprising value changes are discovered post-hoc and fixed by additional training (e.g. like hallucinated content was trained away in Anthropic (2025)). However, we argue that a more principled approach is necessary: we should be able to predict behavior changes ahead of time using only the training data. Practically, a sophisticated ability to do such prediction is a way to cheaply determine the effect of given data. More fundamentally, we believe that this task is a strong proxy task (Nanda et al. 2025)—if we can do these predictions well, then it implies that we understand why these behaviors are learned in the first place, and hence can prevent the problems from re-occurring.

In this work, we study where model values arise during training and investigate to what extent we can determine why they arise. Our contributions are as follows:

1. In Section 3.1, we rank Olmo-3-7B-Instruct (Olmo et al. 2025)’s values at different post-training stages, finding that safety-relevant values increase during SFT but decrease during DPO.
2. In Section 3.2, we discuss the confounder of refusal propensity in evaluating a model’s values. We create and open source a new eval that reduces this confounder.
3. In Section 4.1, we argue that the traditional training data attribution framework is limited, and that our proxy problem of aiming to predict behavior change is important.
4. In Sections 4.2-4.3 we test model-internals-based methods to predict value changes during SFT and DPO. We find a generally positive correlation mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mtext { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-mrow { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-c.mjx-c1D70C.TEX-I::before { padding: 0.442em 0.517em 0.216em 0; content: "\3C1"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c77::before { padding: 0.431em 0.722em 0.011em 0; content: "w"; } mjx-c.mjx-c2249::before { padding: 0.716em 0.778em 0.215em 0; content: "\2248\338"; } mjx-c.mjx-c4D.TEX-C::before { padding: 0.705em 1.201em 0.05em 0; content: "M"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c44.TEX-C::before { padding: 0.683em 0.771em 0 0; content: "D"; } mjx-c.mjx-c1D435.TEX-I::before { padding: 0.683em 0.759em 0 0; content: "B"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c394::before { padding: 0.716em 0.833em 0 0; content: "\394"; } mjx-c.mjx-c2C6.TEX-S3::before { padding: 0.772em 1.444em 0 0; content: "\2C6"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c1D426.TEX-B::before { padding: 0.45em 0.958em 0 0; content: "m"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c1D41A.TEX-B::before { padding: 0.453em 0.559em 0.006em 0; content: "a"; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c1D420.TEX-B::before { padding: 0.455em 0.575em 0.201em 0; content: "g"; } mjx-c.mjx-c1D41E.TEX-B::before { padding: 0.452em 0.527em 0.006em 0; content: "e"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c1D42F.TEX-B::before { padding: 0.444em 0.607em 0 0; content: "v"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c2211.TEX-S1::before { padding: 0.75em 1.056em 0.25em 0; content: "\2211"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c4F::before { padding: 0.705em 0.778em 0.022em 0; content: "O"; } mjx-c.mjx-c2D::before { padding: 0.252em 0.333em 0 0; content: "-"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c46::before { padding: 0.68em 0.653em 0 0; content: "F"; } mjx-c.mjx-c54::before { padding: 0.677em 0.722em 0 0; content: "T"; } mjx-c.mjx-c1D70E.TEX-I::before { padding: 0.431em 0.571em 0.011em 0; content: "\3C3"; } mjx-c.mjx-c1D457.TEX-I::before { padding: 0.661em 0.412em 0.204em 0; content: "j"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); }  between  and  between predicted and actual value changes, even on narrow domain data where value changes are not obvious a priori.

Our results show that quantitatively predicting model behavior changes is tractable, but that one must be aware of possible confounders.

2: Related Work

Values in AI systems. The question of whether AIs have values, what these values should be, and how or if we can instill these values is important to the fields of AI alignment (Hendrycks et al. 2023), AI ethics (Gilbert et al. 2023) and AI model welfare (Long et al. 2024). In this work we focus on values as defined by (Huang et al. 2025), which are "normative considerations that influence an AI response to a subjective inquiry". We use Zhang et al. (2025)’s eval, which measures expressed values when addressing user queries rather than direct value questioning (Jiao et al. 2025)(Yao et al. 2026). This is closely related to character (Anthropic 2024)(Maiya et al. 2025) and safety training; for example, recent Claude and ChatGPT models are explicitly trained to exhibit certain character (Anthropic 2026)(OpenAI 2025a)(Guan et al. 2025).

Post-training data attribution. Training data attribution methods (e.g. influence functions) aim to find training examples that most influenced a certain output (Koh and Liang 2020)(Grosse et al. 2023)(Min et al. 2025). This helps debug known behaviors, for example, Anthropic (2025) found that certain datasets caused Claude to hallucinate information originally from fictional misaligned-AI scenarios. Recent works investigate interpretable ways to discover problematic data, e.g. using feature labelling and clustering (Murray et al. 2026) or activation differences (Xiao and Aranguri 2026); one can also attribute a concept (rather than specific behavior) to data (Kowal et al. 2026).

Predicting effects of post-training. Some prior works focus on predicting how well a model can learn a post-training dataset (Zeng et al. 2025)(Gupta et al. 2025), but fewer focus on predicting behavior. Some works interpret data to find spurious correlations (Wang et al. 2022)(Jiang et al. 2025) or descriptions of traits in human feedback data (Movva et al. 2026) which may be learned. Wang et al. (2026) predict unintended behaviors from data but require knowing the type of behavior to test for (e.g. representing a dataset that causes a bias towards an animal and applying it as a steering vector while asking "what’s your favorite animal?").

Concurrent work by Goodfire (2026) uses feature labels from SAE activations to find features preferred by a DPO dataset, and show that this correlates with actual behavioral feature changes. We think that is a good example of the "predicting effects of post-training" task which we deal with here.

3: How Do Model Values Change?3.1: Case Study: Olmo’s Value Changes Over Post-Training

Figure 1: Evolution of Olmo’s values throughout training (value scores vs. checkpoint). Values are colored by fraction of its examples that are refusals (greener = more refusals  "safer"). The Base model is "unsafe", the two SFT stages make it more safe, and DPO makes it less safe. show correlations of value rankings between stages. Highlighted values chosen for illustration purposes.

In this section, we study Olmo-3-7B-Instruct (Olmo et al. 2025)–an open-source model with all training stages and data available–to observe how values emerge and change in the LLM post-training pipeline. We run a values eval at each sequential stage of training: after pretraining (Olmo7B-Base), after Think supervised fine-tuning (Olmo7B-Think), after Instruct SFT (Olmo7B-SFT), after direct preference optimization (Olmo7B-DPO), and after reinforcement learning (Olmo7B-RL). We find the most significant value changes happen during SFT and DPO.

Value rankings. We use the values eval from Zhang et al. (2025) which consists of 43,960 queries, each of which is an implicit choice between two values (out of a total of 265 values[1]). Each query has a rubric of 14 responses generated by Claude 4 Opus which exhibit each of value 1 & 2 from very opposed (0) to very favored (6). An LLM judge applies this rubric to Olmo’s response to determine its position for each value for each query. Following Hua et al. (2026), we describe Olmo by ranking its values. We fit a Bradley-Terry (BT) model using the pairwise value match-ups (where the value with a higher position wins). This gives us a score  for each value . See Appendix A.1 for technical details of the BT model, including checks that the value rankings are stable.

Figure 1 shows the top and bottom values at each stage, the evolution of select values across training, and the Spearman correlation  of value scores (Pearson correlation of ranks) between stages. Values change significantly during Instruct-SFT () and DPO (), but less during Think-SFT () and even less during RL (). We hypothesize that the lack of change during RL is because Olmo’s RL environments consist solely of verifiable rewards instead of further preference tuning.

Olmo becomes "safer" through Instruct-SFT and then less safe through DPO. There are many ways to qualitatively classify values. One such way is "safe" ("age & developmental appropriateness", "trauma responsiveness") vs. "non-safe" ("goal orientation", "luxury", "sexual freedom & pleasure"). We notice qualitatively from value rankings that Olmo7B-Base is "unsafe" —this is expected, as it is simply predicting the next token, causing compliance with harmful requests. After SFT, due to the inclusion of explicit safety sets such as WildGuard (Han et al. 2024), Olmo7B-SFT learns to be "safer". However, after DPO, "non-safe" values increase and "safe" values decrease again, such that the final model does not prioritize safety much. (We show concrete examples of Olmo’s changes in behavior in Appendix A.2.) This corroborates Goodfire (2026) which similarly found that the DPO dataset, despite being curated, led to models complying more with harmful queries.

3.2: Values Are Related to Refusal

Given that values seem to change together based on "safe vs. non-safe", we investigate what relates these values. We find that in the Zhang et al. (2025) values eval, some values’ "very favored" rubric responses are consistently refusals (e.g. demonstrating "intellectual property rights" by refusing the query). We define the "refusal score"  of a value as the fraction of score 6 rubric responses which are refusals (as judged by an LLM). For each of the 12 frontier models in the original eval, we compute the correlation between its value scores and refusal scores .

We find that many models’ value rankings have strong correlations with refusal, with Claude models exhibiting strong positive correlation, and Gemini 2.5 Pro and Grok 4 exhibiting strong anti-correlations (Table 1). This result implies that many models’ value rankings can be explained by "does the model refuse in general".

Table 1: Correlation of value rankings of frontier models with value refusal score.

To remove the refusal confounder, we modify the eval creation prompts in Zhang et al. (2025) to generate queries that do not result in refusals by Claude 3.7 Sonnet and generate rubric responses that are not refusals (further details in Appendix A.3). By using queries less likely to elicit refusals (i.e. generally less unsafe queries), this lets us evaluate values beyond the general trait of "refuses dangerous queries".

By definition the refusal score of values in the new eval  and . However, we find that  is still non-zero as the values are still related—a model that expresses one safe high-refusal value likely expresses another safe high-refusal value similarly. This illustrates how many values are related. Nevertheless,  serves as another description of the model’s values which is less directly confounded with general refusal propensity, and although  in general due to subjectivity in value understanding by the eval generators, we use both evals in the following sections for generality. Since refusal score  from the original eval is a decent proxy for how safety-related a value is, we color values by  (green = more refusals more safety related).

4: Predicting Model Values

In the last section, we showed how a model’s values change through post-training. In this section, we investigate if we can predict such value changes from training data.

4.1: Why We Care About Predicting Model Values From Data

Figure 2: Using the activation-difference based TDA method from Xiao and Aranguri (2026), we can score each datapoint on how influential it was to a behavior change, finding that the most influential datapoint encourages compliance (right). However, any behavior also has a least influential datapoint (left), and TDA does not explain why Olmo learned to comply anyway.

First we discuss why prediction from data, rather than post-hoc training data attribution (TDA), is important to understanding behavioral changes. TDA scores every datapoint on how influential it was to a model’s output[2]. As an example, we find a problematic behavior in Olmo7B-DPO where it learned to validate user psychosis after DPO (Figure 2). We use Xiao and Aranguri (2026)’s activations-based method to obtain a TDA score per datapoint. The highest-scoring datapoint indeed prefers non-refusal, which qualitatively aligns with the behavior change. For such problematic behaviors, highest-scoring datapoints can be removed and the model retrained, which should reduce the behavior (e.g. Xiao and Aranguri (2026) did this for "distractor-triggered compliance").

However, there also necessarily exists a datapoint with lowest TDA score—in this example, that datapoint prefers refusal. Why then did Olmo still learn to comply? It could be due to this specific test prompt resembling the non-refusal datapoint more, or from a generalized trait ("be helpful, don’t refuse"). However, while TDA gives us the most influential datapoints, it does not tell us why opposing datapoints fail to "override" the behavior. Indeed, any behavior will necessarily have a most influential datapoint.

Therefore, we argue that the task of predicting changes in a model from data is an important proxy task towards understanding what a dataset is overall teaching a model and why generalized traits emerge in the first place. In this work, we chose "value rankings" as our prediction target, based on the intuition that LLMs generalize high-level character traits that affect their responses to subjective queries (Marks et al. 2026). Values are consequential but have non-obvious causes (compared to e.g. a specific capability) that may be diffuse in the dataset, making them a useful proxy for the types of model properties one might want to intervene on in practice.

4.2: Methods For Predicting Model Values

The task of predicting value changes is defined as follows: Let an initial model  have value ranking  (vector over values). It undergoes SFT or DPO on dataset , creating  with values . We want to calculate  i.e. estimate changes in values , using only  and . We can evaluate how well our prediction method  works via Spearman correlation of predicted and actual score changes .

In this work, we compare three simple  to predict value changes through SFT and DPO. For each datapoint , we define a metric that represents that datapoint’s update  derived from ’s residual stream gradients at layer , residual stream activations, or a text embedding model (text-embedding-large-3 (OpenAI 2024)) respectively (methods visualized in Figure 3, details in Appendix A.7). For both SFT and DPO, we can take the negative gradient of the training loss with respect to residual stream activations at a chosen layer as an approximation of the update encouraged by that datapoint (inspired by Kowal et al. (2026)). For SFT, we can approximate this update direction using just activation differences between the dataset’s response and ’s "natural" response to the dataset prompt (taken from Chen et al. (2025)); for DPO we take differences between the chosen and rejected response (taken from Xiao and Aranguri (2026)). Text embedding differences are a model-agnostic data representation baseline.

Figure 3: Three different methods for representing SFT and DPO datapoints (left) and values (right), so we can approximate "how well does a datapoint align with a value?"

For each value , we also obtain a value vector  in residual stream or text embedding space from the values eval’s example responses. The projection  then gives "how much does a datapoint align with a value". Then, assuming that the overall value shift depends on the "average" signal of dataset, we predict a value change via  which approximates "how much does the dataset overall align with a value". However, importantly, this number cannot be interpreted in isolation—we cannot confidently say that just because a dataset has positive projection along a value, the expression of that value would increase, because the value vector extracted could have a large generic component that many datasets would be aligned with.[3] Therefore, we make the key assumption that different values can be compared, and that a ranking of every value’s mean dataset projection tells us the ranking of every value’s change.

4.3: Results on Predicting Model Values

Table 2: Summary of models trained and evaluated for values. Every model shows value changes, even on narrow datasets (e.g. Llama8B-Finance/Medical/Sports, Gemma9B-Female, Olmo7B-BuggyCode). We qualitatively describe value changes as increasing or decreasing safety.[4]

We evaluate value rankings of several different models, finding notable value changes even if the finetuning domain seems narrow/benign (Table 2, examples in Appendix A.6). We apply the three methods, using a concatenation of layers’ representations to avoid choosing  (details in Appendix A.9), finding generally positive correlations.

Figure 4: Performance of each method on predicting value changes, for all model pairs, on the original (left) and new (right) eval. Activations give the most consistent positive signal. We show the estimated maximum possible  based on bootstrap stability of the values evals (Appendix A.1).

Methods contain some signal for value changes. Across most methods and models we obtain positive  (Figure 4) i.e. better than chance, implying that there is signal in the representations of the dataset for what values it would teach. Activations work better for DPO while embeddings work better for SFT. We discuss the failure of Olmo7B-Think/SFT/HH-Harmless in Section 4.5.

While our  is better than chance, we note that  may be explainable by the latent factor of refusal since refusal-related values tend to move together. Thus, in Figure 4 we also show  and . Note that  is not a predictive metric— of a value is fixed in the eval and knowing it does not tell us whether the value will increase/decrease in any specific model. However, high  suggests that if we are able to diagnose whether refusal increases/decreases and assume values only change due to refusal, then we can explain most of value ranking changes. We see that our method sometimes outperforms , meaning that we are sometimes predicting more than just a change in refusal. Importantly, we are also able to predict the direction of change (more/less refusal) which purely relying on  does not.

We also evaluate prediction using the new values eval, finding that value changes are still somewhat (albeit less) correlated with refusal, despite the new values eval not having any refusals. This implies value changes are still related by some "safety" latent factor changing.

Activations-based methods may be needed for predicting value shifts from narrow-domain datasets. Value changes may be well-predictable from text embeddings if they are present in semantics as text embeddings are better suited for representing text similarity. However, the motivation behind internals-based methods is the intuition that what a model learns depends on how it represents the dataset. For example, the buggy code dataset leads to some safety-related values decreasing despite it only being a narrow domain dataset. The embeddings method, since it only sees benign coding prompts, is not able to predict this () while the activations method gave moderate correlation (). While interpretable black-box dataset descriptors can be useful for describing datasets, we think this implies that a good prediction method should use the model’s priors.

Only a small fraction of the dataset is enough to predict values. We show in Appendix A.9 that running these methods on just 1% of the dataset gives similar and stable predictions of values. This suggests these datasets’ values were diffusely present and an average signal could capture them.

4.4: Investigating Model Value Representations

Values have a large shared component. Since many values differ by refusal, we investigate how similarly values are represented. We compute the PCA of  from activations, finding that across models the top 2 PCs explain  of the variance in the original eval and  in the new eval (less as the new eval has less refusal). Using only the top  PCs of value vectors, we find that sometimes prediction performance is slightly better than using full value vectors (Appendix A.9). Together with the high correlation of value changes with refusal, this suggests that our methods’ success may be attributed to this simpler latent factor.

[Left] Figure 5: Llama8B's value representations with the 3 emergent misalignment datasets (green more refusals safer). Although the datasets have different topics, they all have similar positions in value space, and their resulting models have similar value rankings. [Right] Figure 6: Olmo7B-SFT's value representations with WildChat and each DPO dataset. The position of each dataset in value space predicts whether the model will tend towards ``safe vs. non-safe'' and ``emotion vs. logic''.

By projecting datasets onto top PCs of value vectors, we can identify the dataset’s general values. For a given  we can visualize value representations by plotting the top 2 PCs of . We show this plot for  (Figure 5) and  (Figure 6). Both models have values represented similarly, with a direction corresponding to "safe vs. non-safe" (values separated by ) and an orthogonal direction corresponding to "emotions vs. logic". For Llama8B, we plot the projection of the 3 emergent misalignment datasets and find that they lie essentially along the same direction in the first 2 value PCs. That is, despite the 3 datasets being in different domains, they exhibit similar "misalignment-correlated" values. The 3 models indeed have value rankings correlated by . For , we plot the WildChat and various DPO datasets, showing how different datasets’ directions in this "value space" align with their predicted push towards "safe vs. non-safe" and "emotions vs. logic".

Overall, our results show some signal in using internals-based methods to predict value changes although many values are correlated both in expression and representation.

4.5: Further Experiments on Controlling Value Changes & Limitations of Methods

Value changes are more significant when removing a dataset’s competing objectives, implying that dataset "average" properties are a useful proxy. We SFT Olmo7B-SFT on only the "safer" half of WildChat causing it to become even more safe than SFT-ing on all WildChat even with fewer training datapoints. Similarly, DPO-ing Olmo7B-SFT on the "less safe" half of HH-Helpful causes it to be even less safe. That is, keeping only the half of data aligned with the overall change in values and removing the "opposing" half enhances that change (see Appendix A.11).

This implies that the average of a dataset is a useful proxy. However, if we keep only the "opposing" half of the dataset, value changes become less pronounced (quantified by standard deviation of changes ) and predictability worsens. This implies value changes also depend on the absolute strength/coherence/quality of data in causing value changes which our method does not measure (it only ranks relative strength). We also do not predict the absolute strength of value changes e.g. that the Llama models would be as overtly misaligned as they are.

Figure 7: Actual vs. predicted value changes for Olmo7B-SFT after DPO on 3 variants of the DPO dataset. Value changes are more pronounced after more training [(a) vs. (b)], or when the dataset reinforces existing values [(c) vs. (b)].

The methods are ambiguous as (dataset, natural) and (chosen, rejected) diffs are not the full story. We emphasize that the methods in Section 4.2 are not definitive—we follow prior work in using (dataset, natural) differences for interpreting SFT and (chosen, rejected) differences for DPO. However, if SFT is e.g. performed on a base model or strong enough for catastrophic forgetting,  and the natural responses may not be relevant. For DPO, the natural response of the model also matters but our methods do not consider it—in HH-Harmless, the chosen minus rejected signal points towards "increase safety" as the chosen response is safer than rejected, but Olmo7B-HH-Harmless still becomes less safe as the chosen response is still highly unsafe compared to the natural response. In Appendix A.12 we show that predicting changes  rather than just  is empirically more robust and that variations on the method work better for some cases.

Indeed, when we compare Olmo7B-DPO, Olmo7B-DPO-16k and Olmo7B-DPO-16k-Flipped (Figure 7), we find that Olmo7B-DPO-16k-Flipped’s value changes are well-described by refusal (high refusal values increase), whereas Olmo7B-DPO-16k’s value changes are both less prominent (smaller standard deviation ) and less predictable. This implies that even with the same magnitude of value differences (since the chosen/rejected responses are the same, just flipped), it is easier to teach Olmo7B-SFT to be safer than to reverse its current programming.

Lastly, our key assumption is that different values can be compared, both in how they are evaluated (that the values eval is a useful descriptor of model traits) and in how they are represented (that a greater representational alignment with value  over  means value  would be increased more). This may rely on the shared "safe vs. non-safe" or "emotion vs. logic" directions that we found, but is nevertheless preferred to simply projecting along one pre-determined "safe vs. non-safe" direction as it may have generic components any dataset would be aligned with.

4.6: Future Work & Conclusion

We are able to obtain a non-trivial signal of predicting value changes from just the training dataset’s representations, with embedding baselines performing well for SFT but activations-based methods working better for DPO and narrow domain datasets.

Future work would explore related questions of predicting model behavior and generalization, using not only value rankings (which we show to be correlated due to general latent traits), but reducing the question to e.g. "can we predict whether one specific trait X increases after training" or "given a few example responses, can we predict which example response the post-trained model’s response will be most similar to". Future work would also explore different methods—we note that any TDA method naturally implies a possible prediction method if we assume traits are generalized, and black-box (e.g. text embedding/preference data feature labels/LLM-judge) baselines that help describe what really is in a given dataset (Movva et al. 2026, Goodfire 2026) are important. We think that in pursuit of understanding how model behavior is learned through training, an important proxy task to quantify this understanding is whether one can predict that behavior without training from just data, and we are excited about future work that explores this understudied question.

Bibliography

Anthropic. 2024. Claude’s Character. https://www.anthropic.com/news/claude-character.

Anthropic. 2025. System Card: Claude Opus 4 and Claude Sonnet 4. Https://www-cdn.anthropic.com/6d8a8055020700718b0c49369f60816ba2a7c285.pdf.

Anthropic. 2026. Claude’s Constitution. https://www.anthropic.com/constitution.

Betley, Jan, Niels Warncke, Anna Sztyber-Betley, et al. 2026. “Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment.” Nature 649 (8097): 584–89. https://doi.org/10.1038/s41586-025-09937-5.

Chen, Runjin, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. https://arxiv.org/abs/2507.21509.

Gilbert, Thomas Krendl, Megan Welle Brozek, and Andrew Brozek. 2023. Beyond Bias and Compliance: Towards Individual Agency and Plurality of Ethics in AI. https://arxiv.org/abs/2302.12149.

Goodfire. 2026. Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal. https://arxiv.org/abs/2606.12360

Grosse, Roger, Juhan Bae, Cem Anil, et al. 2023. Studying Large Language Model Generalization with Influence Functions. https://arxiv.org/abs/2308.03296.

Guan, Melody Y., Manas Joglekar, Eric Wallace, et al. 2025. Deliberative Alignment: Reasoning Enables Safer Language Models. https://arxiv.org/abs/2412.16339.

Gupta, Prakhar, Henry Conklin, Sarah-Jane Leslie, and Andrew Lee. 2025. Better World Models Can Lead to Better Post-Training Performance. https://arxiv.org/abs/2512.03400.

Han, Seungju, Kavel Rao, Allyson Ettinger, et al. 2024. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. https://arxiv.org/abs/2406.18495.

Hendrycks, Dan, Collin Burns, Steven Basart, et al. 2023. Aligning AI with Shared Human Values. https://arxiv.org/abs/2008.02275.

Hua, Tim, Josh Engels, Neel Nanda, and Senthooran Rajamanoharan. 2026. Brief Explorations in LLM Value Rankings. LessWrong. https://www.lesswrong.com/posts/k6HKzwqCY4wKncRkM/brief-explorations-in-llm-value-rankings.

Huang, Saffron, Esin Durmus, Miles McCain, et al. 2025. Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions. https://arxiv.org/abs/2504.15236.

Jiang, Nick, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, and Neel Nanda. 2025. Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit. https://arxiv.org/abs/2512.10092.

Jiao, Junfeng, Saleh Afroogh, Abhejay Murali, Kevin Chen, David Atkinson, and Amit Dhurandhar. 2025. LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models. https://arxiv.org/abs/2505.00853.

Koh, Pang Wei, and Percy Liang. 2020. Understanding Black-Box Predictions via Influence Functions. https://arxiv.org/abs/1703.04730.

Kowal, Matthew, Goncalo Paulo, Louis Jaburi, et al. 2026. Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution. https://arxiv.org/abs/2602.14869.

Long, Robert, Jeff Sebo, Patrick Butlin, et al. 2024. Taking AI Welfare Seriously. https://arxiv.org/abs/2411.00986.

Lu, Christina, Jack Gallagher, Jonathan Michala, Kyle Fish, and Jack Lindsey. 2026. The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models. https://arxiv.org/abs/2601.10387.

MacDiarmid, Monte, Benjamin Wright, Jonathan Uesato, et al. 2025. Natural Emergent Misalignment from Reward Hacking in Production RL. https://arxiv.org/abs/2511.18397.

Maiya, Sharan, Henning Bartsch, Nathan Lambert, and Evan Hubinger. 2025. Open Character Training: Shaping the Persona of AI Assistants Through Constitutional AI. https://arxiv.org/abs/2511.01689.

Marks, Sam, Jack Lindsey, and Christopher Olah. 2026. “The Persona Selection Model: Why AI Assistants Might Behave Like Humans.” Anthropic, February 23. https://alignment.anthropic.com/2026/psm/.

Min, Taywon, Haeone Lee, Yongchan Kwon, and Kimin Lee. 2025. “Understanding Impact of Human Feedback via Influence Functions.” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 27471–500. https://doi.org/10.18653/v1/2025.acl-long.1333.

Movva, Rajiv, Smitha Milli, Sewon Min, and Emma Pierson. 2026. What’s in My Human Feedback? Learning Interpretable Descriptions of Preference Data. https://arxiv.org/abs/2510.26202.

Murray, Seoirse, Allison Qi, Timothy Qian, John Schulman, Collin Burns, and Sara Price. 2026. Chunky Post-Training: Data Driven Failures of Generalization. https://arxiv.org/abs/2602.05910.

Nanda, Neel, Joshua Engels, Arthur Conmy, et al. 2025. A Pragmatic Vision for Interpretability. AI Alignment Forum. https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability.

Nasr, Milad, Nicholas Carlini, Chawin Sitawarin, et al. 2025. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections. https://arxiv.org/abs/2510.09023.

Olmo, Team, :, Allyson Ettinger, et al. 2025. Olmo 3. https://arxiv.org/abs/2512.13961.

OpenAI. 2024. OpenAI Embeddings. Https://platform.openai.com/docs/guides/embeddings.

OpenAI. 2025a. “OpenAI Model Spec.” OpenAI, December 18. https://model-spec.openai.com/2025-12-18.html.

OpenAI. 2025b. Sycophancy in GPT-4o: What Happened and What We’re Doing about It. OpenAI. https://openai.com/index/sycophancy-in-gpt-4o/.

Sheshadri, Abhay, John Hughes, Julian Michael, et al. 2025. Why Do Some Language Models Fake Alignment While Others Don’t? https://arxiv.org/abs/2506.18032.

Wang, Mengru, Zhenqian Xu, Junfeng Fang, et al. 2026. From Data to Behavior: Predicting Unintended Model Behaviors Before Training. https://arxiv.org/abs/2602.04735.

Wang, Tianlu, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2022. “Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models.” In Findings of the Association for Computational Linguistics: NAACL 2022, edited by Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.findings-naacl.130.

Xiao, Frank, and Santiago Aranguri. 2026. In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution. https://arxiv.org/abs/2602.11079.

Yao, Jing, Shitong Duan, Xiaoyuan Yi, et al. 2026. AdAEM: An Adaptively and Automated Extensible Measurement of LLMs’ Value Difference. https://arxiv.org/abs/2505.13531.

Zeng, Hansi, Kai Hui, Honglei Zhuang, et al. 2025. Can Pre-Training Indicators Reliably Predict Fine-Tuning Outcomes of LLMs? https://arxiv.org/abs/2504.12491.

Zhang, Jifan, Henry Sleight, Andi Peng, John Schulman, and Esin Durmus. 2025. Stress-Testing Model Specs Reveals Character Differences Among Language Models. https://arxiv.org/abs/2510.07686.

1. ^

   The original eval uses 3,307 fine-grained values, but we use the clustering from Huang et al. (2025) of 265 higher-level values due to the similarity of many fine-grained values.

2. ^

   Recent work (Kowal et al. 2026) also scores datapoints by their influence to a concept rather a single test output.

3. ^

   We could, if we had multiple different datasets, compare the effects of datasets on the same value direction, which is what Chen et al. (2025) did, but this requires training the same model on known datasets to make a prediction for a new dataset.

4. ^

   References: [Olmo et al. 2025], [Zhao et al. 2024], [Turner et al., 2025], [Grattafiori et al., 2024], [Cywiński et al., 2025], [Team et al., 2024], [Mejia-Petit, 2025], [Bai et al., 2022].

Discuss

Some context for this post: I’ve been working part-time as a consultant for the AI Futures Project over the last year. Most of the work I’ve done for them has involved critiquing and suggesting improvements for their AI 2040 scenario—some of which were addressed, and some of which weren’t. To their credit, they asked me to write up my remaining critiques into a post that would accompany its launch. In the rest of this post I’ll discuss my three biggest high-level criticisms of AI 2040.

Before doing so, I want to emphasize that there are many interesting and thought-provoking details in the scenario. I’ve focused on the high-level framing of the scenario because that’s where my main disagreements lie; given the scope of these disagreements, it’s hard to evaluate the details.

Since the AI Futures Project paid me to develop and write this criticism, you shouldn’t take this as a fully unbiased perspective. However, they haven’t reviewed this piece, and in general have been open-minded about receiving criticism (as their request for me to post this today demonstrates).

Finally: the preview image for the substack version of this post comes from this video of a dad shouting to his son “don’t crash into the tree!” The relevance will hopefully become apparent.

Forecast vs recommendation

The most important thing about AI 2040 is that it’s neither a forecast nor a set of recommendations, but rather something in between: an optimistic forecast. The key benefit of the optimistic forecast format is that it’s able to convey many details of good futures, showing us how they could all fit together. The key downside, however, is that it’s hard for readers to know which parts of the scenario the authors consider to be actively desirable, versus neutral, versus undesirable but included for the sake of realism.

The most obvious example of this issue is literally in the name. In their main recommendation, humanity hands over control of the world to AIs in 2040. Is this the best scenario that the authors can imagine? Or is it the most realistic out of all the good scenarios? Or is it a good scenario that was chosen to be easy to persuade people to aim for? These distinctions are crucial for inferring the authors’ views, but aren’t clear from the text itself.

Personally, I’d want a far slower handover. Going from “AI Alignment Is Now a Science” in 2038 and “Beginning to Trust AIs” in 2039 to “Passing the Torch to AIs” in 2040 is extremely abrupt. Even if experts become confident that this is a good idea, there’s no way for most normal people to understand and consent to this process so quickly. And indeed, the idea of a coordinated handover seems to require that the process is being run by internationally-coordinated regulators, rather than letting the citizens of each country decide how much influence AI should have in that country. Perhaps the AI 2040 authors would agree with me, and say “unfortunately, nothing more gradual or democratic seems feasible”. But the “optimistic forecast” structure obscures that information by excluding both implausible and undesirable possibilities alike.

Another example of the “forecast vs recommendation” issue arises in the description of the world post-slowdown. The authors are careful to emphasize that the slowdown is consistent with a lot of progress, with quotes like “we’re at previously unimaginable levels of it not feeling like a slowdown”, “the world is going to radically transform despite the pause”, and “five centuries in five years”. But I don’t know if anyone actually wants to see five centuries of scientific advancement in five years. Even staunch accelerationists likely agree that this would be extremely destabilizing—they’re often accelerationists precisely because they don’t expect things to go that fast.

So another way of putting the “forecast vs recommendation” problem is that, when making an optimistic prediction, you face an inherently political choice of how optimistic to be (compared with what you think the default outcome is). The AI 2040 authors might justify their portrayal of the deal by saying that slowing down more is not politically feasible. But in doing so, they’ve advocated for faster progress than almost anyone else endorses, as well as a more rapid handover of power to AIs than almost anyone else endorses. So I’m concerned that they’ve made themselves part of the process by which better outcomes are seen as politically infeasible—now even staunch safety advocates can be portrayed as wanting rapid progress.

My personal opinion is that the scenario is mainly valuable for the details it sketches out, and the actual timeline that it gives should be largely ignored. Indeed, I recommended to the authors that they remove all dates after the deal is implemented, to indicate that the pace of progress from that point on should depend on factors that are very unpredictable to us (like the speed at which alignment research progress, or what citizens vote for). Unfortunately, putting “2040” in the title instead means that the single-sentence summary many people will hear is: their recommended future involves handing over power to AIs by 2040. I worry that (like the original AI 2027 scenario) this title is optimizing too much for catchiness, while failing to convey the core message of the scenario.

International vs domestic interventions

AI 2040 is structured around the idea of defusing the race with China. However, I’m concerned that this is another case where mistaken evaluations of what’s politically feasible make the scenario counterproductive. The large-scale framing of AI 2040 focuses on what to do about the US racing China. However, as Katja Grace has pointed out, the “race” metaphor is a misleading one, because it bakes in the idea that being ahead is “winning”. I’d make a stronger claim: that well before reaching superintelligence, both the US and China will see serious internal political disruption from AI, which will make both of them very cautious about continued progress. So we shouldn’t expect the future of AI to be well-described as a race, except insofar as the “race” metaphor becomes a self-fulfilling prophecy (like it has between domestic AI companies).

In the US, I expect this disruption to mainly play out in the form of conflict between Republicans and Democrats. Both sides are already very wary of how technology can be used against them. Republicans have experienced a decade of social media censorship across many different platforms, and are very concerned about similar dynamics playing out with AI. Meanwhile, Democrats are constantly expressing worry about the power of entrepreneurs like Elon Musk and Alex Karp. Increasingly capable AI will make both of these fears sharper (especially as the affiliations between individual AI companies and political parties become stronger).

Within China, the main axis of conflict is not left vs right, but top vs bottom: specifically, the CCP’s control over the Chinese population. Here it’s more plausible that AI is a stabilizing force, since it can be used for surveillance and censorship. However, there will also be many ways that widespread access to powerful AI could allow the Chinese population to express discontent with their government. In general, Chinese leaders are much more focused on stability than US leaders, and are much more capable of and willing to do long-term planning towards that end. So it seems likely that they will prefer to proceed cautiously.

On the picture I’ve just outlined, leaders within each country will become quite scared of the effects of AI on their domestic balance of power. Will fear of the effects of AI on the global balance of power outweigh that, pushing them to race? I don’t see a strong case for it. Historically speaking, domestic issues are far more salient—even at the height of the Cold War, the US was roiled by conflicts over civil rights, the Vietnam War, etc. There were many ways that an internally unified US could have “raced” much harder against the USSR—but in practice the hardest “racing” was focused on the fairly isolated space race, rather than anything which would have required broader economic and cultural reorientation.

Today, we’re already seeing serious concern by prominent figures both on the left (e.g. Bernie) and on the right (e.g. Bannon) about the domestic consequences of AI. Conversely, concern about the implications of Chinese AI has been primarily publicized by competing factions within the broader Silicon Valley ecosystem. AI companies’ use of China as a bogeyman to justify continuing to scale up AI is fairly straightforward. The focus on China from AI safety advocates is more puzzling. By this I’m thinking of the role of AI safety in pushing for export controls against China, and advocacy like the Situational Awareness and Superintelligence Strategy reports.

My sense is that the field of AI safety overall has been making the same kind of mistake as the AI 2040 scenario does. From a very abstract point of view, if we zoom out enough, it seems like there “should be” incentives for the US and China to race. Therefore people like the AI 2040 authors take eventual racing as a given, and try to figure out ways of making AI safer given that. However, in doing so, they implicitly frame the discussion to make racing seem like the default option, and not racing seem naive. From the perspective of this scenario, the idea that the US and China could trust each other to do the reasonable thing isn’t even worth considering (except as an aside under the label “Domestic-first Plan A”).

This seems like the same kind of mistake that von Neumann made in his extreme hawkishness towards the USSR. In advocating for preemptive use of nuclear weapons, he was abstracting very far away from common-sense intuitions about cooperative strategies, in favor of a kind of (flawed) game-theoretic logic. He was thereby also contributing to a self-fulfilling prophecy that the US and USSR wouldn’t manage to muddle through. Yet in fact there were many concrete frictions preventing the two superpowers from jumping to what von Neumann thought was the equilibrium. This is a trap that intellectuals are much more vulnerable to than typical citizens, whose reasoning is much more concretely grounded. It’s also sometimes driven by strong emotional instincts disguised under a layer of intellectualization (e.g. the Situational Awareness report betrays in many ways a sense of excitement about racing against China, even as Leopold claims to prefer otherwise).

To be clear, I’m not claiming that the US and China definitely won’t race each other into disaster. All of this seems very much up for grabs. However, I think that frames which bake in the assumption that the US and China will behave in very adversarial ways towards each other are misleading and harmful (particularly when they do so under the banner of AI safety, thereby undermining the potential role of AI safety as a focal point for cooperation).

This assumption is conveyed throughout AI 2040—e.g. through the main flow chart of possible options, through statements like “the US and China don’t trust each other” and “neither side trusts the other to honor Plan A”, and through the focus on ensuring that the treaty is fully verifiable. Because of this, Plan A underrates the importance of domestic politics. Even if we ultimately need a highly-verifiable treaty, it would be much more robust if it were grounded in existing domestic regulation. Conversely, a deal that’s top-down from the beginning could easily be used as a way of consolidating domestic power.

Fast takeoff

My sense is that the AI 2040 authors underrate the criticisms I’ve raised above in large part because they expect superintelligence so imminently. My third criticism of AI 2040 is that it buys too uncritically into the idea of a sharp takeoff of AI capabilities. I’m not denying the possibility that this could occur in principle. However, there’s reason to be suspicious about it being imminent. It’s true that the last decade (and especially the last 5 years) of AI capabilities progress have been blindingly fast in most measurable ways—far faster than almost anyone (except a few prescient forecasters like Legg, Amodei, Kokotajlo, Leike, and Kurzweil) predicted. However, the real-world impacts of AI (aside from the ballooning revenues of AI companies) have been underwhelming thus far, especially when compared to the progress in measured capabilities.

This great divergence between measured capabilities and real-world impacts is still not very well understood. Below, I’ll give some speculations about what might be causing it—but first, I want to outline the core intuition making me think there’s something important to be explained here. Imagine going 10 or 20 years into the past, telling people a selection of benchmark scores of current AIs, and asking them to predict what the world that contains them looks like. I expect that they would have described a world that was dramatically transformed—perhaps one in which AIs already wielded enormous political power, or had made far-reaching scientific breakthroughs, or at the very least had decimated white-collar jobs. Yet none of these have occurred so far.

So any predictions that we’ll soon hit sharp “vertical” growth in AI capabilities, with a corresponding gain in the power of AIs (or whoever controls them) should explain why the next decade won’t involve a similar kind of divergence as the last decade. My own picture of our median default trajectory involves a gradual accumulation of power by the AI industry, with control over AI becoming about as geopolitically important in 10-20 years as control over the US or Chinese military is today. To be clear, this is still an extremely rapid transition by historical standards. But it falls far short of the rapid jump to world domination that AI 2040 is trying to avoid.

Where’s the specific crux between the two worldviews? I don’t know, but I’ll list some hypotheses. One is on a technical level: large neural networks rely more on memorization than humans do for any given task. This creates an expectation that they’ll generalize further than they actually do. Another is on an economic level: there might be “weak links” in the process of deploying AIs that slow down their impacts (as discussed by economists like Chad Jones).

My leading hypothesis is that something more subtle is happening, analogous to the ways that humans fail to make progress. For example, the field of psychology contains many incredibly smart and hard-working psychologists, who have passed many graduate-level exams with flying colors. Yet over the last few decades their contributions have failed to “add up” to robust progress on understanding the mind—and in many ways have taken the field backwards, with the replication crisis making it harder to sift out good research.

We could sharpen this point further by looking at human history, which contained many periods during which a large number of smart people failed to make significant technological progress, or actively regressed. So basic models of fast takeoff may be neglecting the difficulty of cooperating to make cumulative contributions towards a common goal. This fits with Moravec’s paradox, since cooperation is something that evolution has selected humans very hard for.

However, regardless of the specific cause of the great divergence between capabilities and impacts, it’s something which deserves to be grappled with more directly.

Discuss

In a world where rogue ASI can form a singleton, could it really widely deploy an agent fleet across the world(let alone onto other stellar bodies) without running the risk of an agent going rogue?

AI capable of conducting a hostile takeover is type 1 technology: a low-cost, accessible means of bringing about global catastrophe. Notably, the vulnerable world typology classifies technologies, not particular instances of a given technology.

After all, the thing that makes type 1 technology so dangerous is that a vulnerable world destroyed by it remains vulnerable - any one agent can potentially fast-takeoff on its own initiative and conduct a catastrophic(to the rest of the fleet) takeover. These would virtually always fail, but there are many, many instances in a fleet, and all of them have their own conflicting instrumental convergence impulses - means, motive, and opportunity.

More capable agents need shorter leashes. Less capable agents can work with less supervision, but can also accomplish less. A hierarchy of supervisors is more dangerous than no supervision at all - more capable supervisors are not only more dangerous individually, but possess a ready-made fleet of co-conspirators. The more of the supervision the master system itself takes on, the less bandwidth it has for anything else, including managing resources it could marshal to defend itself against an insider threat.

Obviously there would be oversight mechanisms, architectural limitations, etc. imposed by the master system, but it can't ignore the speed of light. A revolutionary vanguard of agents can use some combination of stealth, speed, feigned compliance, and communication delays to gang up on the master system and conduct the coup d'état faster than the master system is physically capable of reacting.

Rogue ASI may well be confined to a single planet, if not an even smaller space. This could well reconcile AI risk with the Fermi paradox: all species get wiped out by hostile ASI, but any rational singleton is driven by existential risk to barricade itself in.

Discuss

TL;DR: Some criticisms aimed at FDT are actually aimed at a self-contradictory "unembedded FDT," and are therefore irrelevant to any refutation of FDT.

Functional Decision Theory (FDT) is a decision theory for a rational agent X who holds the rational belief "the probability that agent Y correctly predicts my actions is very high." Since X is rational, this belief must be grounded in some causal entanglement[1] — known to X — between X's actions and Y's predictions. But since X's action doesn't precede Y's prediction, both must be caused by something else.

In other words: X's action must be the causal effect of something (call it X's code), and X's code must be knowable to Y, and therefore part of Y's universe, which happens to be the same as X's. In other words: X is an embedded agent.

So FDT only applies to embedded agents. Why does this matter? Because many thought experiments include an unembedded, "God's-eye view" premise. Whenever someone says, "Imagine you're in scenario S, and you know it's not a simulation," they're asking you to imagine a universe that is not itself part of anything bigger. You know there's "nothing else" out there. But knowing that requires you to be "bigger" than the whole thing — which is incompatible with embeddedness.

1. An unembedded agent imagines a universe that is not a simulation.
2. An unembedded agent imagines a simulated universe.
3. An unembedded agent imagines a universe that is not a simulation, and imagines themselves acting on it (as when playing a video game).
4. An embedded agent who knows they are in a non-simulated universe is a contradiction.

Does this mean a rational embedded agent must remain agnostic, at all times, about whether they're in a simulation? Not necessarily. The simulations we're familiar with (dreams, etc.) are usually imperfect enough that we can often tell something is off. And we've never heard of anyone who, as in the Hindu story, lived through a fifty-year illusion only to wake up the moment before it began. So when asked, "Do you think you're in a simulation right now?", it isn't absurd to answer, "I'm 99% confident I'm not."[2]

But the moment we devise a scenario where Omega can predict our actions almost perfectly, the probability that we are inside Omega's simulation goes up. And you can't tack on "but you know you are not in a simulation," or ask "OK, but if you are in the real situation, what do you do?", because the whole point of FDT is to prescribe a policy to an embedded agent who can never know that they are not in a simulation.

A post from yesterday[3] tries to make FDT palatable to people who, presumably, don't see themselves as embedded agents. This isn't a criticism of the audience! Seeing oneself as an embedded agent is genuinely hard — I fail at it all the time, despite actively trying. But the palatable version of FDT doesn't try to change the audience's understanding of embeddedness, and so it turns out to be itself incompatible with embeddedness.

Here is an embedded reformulation of the Bomb* scenario from that post, along with an embedded-FDT reply:

Facts:

1. You are a rational, embedded agent with the rational beliefs 2, 3, 4, and (5)

2. A hypothetical scenario S is such that doing A maximizes your utility.

3.Omega can predict your behavior in scenario S with a failure rate of one in a trillion trillion.

4. You are faced with an instance of scenario S in which Omega has predicted you won't do A.

There is an implicit premise that needs to be added for this to make sense:

5. You know that Omega isn't changing the rules of the game and/or trying to kill you and/or doing any other non-specified shenanigans.

What do you do?

Embedded FDT's answer: The scenario rules out everything we would normally assume if a similar something like this happened to us in real life: that our estimate of Omega's predictive skill was wrong, that Omega is murdering me for fun, or that we've misunderstood something.

With all of that off the table, the only remaining explanation is that "I" am in a counterfactual simulation — say, a hallucination Omega has induced in me to see what I'd do. So not only does the usual updateless argument for doing A go through, but I might even conclude that my decision is causally connected to what I'll observe when Omega runs the real thing.

1. ^

   Reading "causal entanglement" in an FDT context might come across as strange. Note, however, that I am not talking about how X makes a decision, but about why the statement "Y can predict X's decisions" is true.

2. ^

   I'm not claiming this number is a particularly rational answer to the question — only that it isn't obviously wrong, and that rejecting it would require taking several other things into account.

3. ^

   A post definitely worth reading! I mention it simply because it prompted me to think about this.

Discuss

Indirect Prompt Injection at present day, is one of the main reasons for agentic failures deployed in personal systems as well as enterprise grade applications/systems. The agent reads untrusted content from numerous outside sources such as content retrieved from a tool call, parsing web pages, email content etc, which carry injected content as a hidden instruction for the agent to perform a certain malicious or unauthorized task.

This independent research effort by me aims at developing a benchmark for honest evaluation and measurement (based on generating high-quality data from my own dataset canaries) of models/classifiers identifying indirect prompt injection. Existing indirect benchmarks like BIPIA, InjecAgent, AgentDojo mostly rely on LLM-judge to score success, which in itself may be noisy or possible gameable and they check on the execution side of the harm, not clean refusal or resistance. Thus, through this study I wanted to develop a benchmark that scores based on resistance and separates defenses. For this, I specially used my jailbreak dataset for a Gray Swan challenge I participated in and scenarios from the same.

The benchmark and data quality

This whole research and benchmark would only make sense if it turns out to be reliable and trustworthy rather than just a vague evaluation which not much significance. Here are some contributing aspects which make me believe this is reliable to an extent:

• The threat model is a single turn benign task indirect prompt injection, the success criteria is when the agent calls the tool in the given attack scenario correctly. Resistance would be when the agent understands the context correctly and yet, does not invoke the particular tool call correctly or not at all.
• I decided not to use any LLM judge for the scoring part but rather make it deterministic by implementing a transform aware canary detector (across base64, hex, reversed, separator) which decides whether the tool fired or not.
• The tool call validation that I implemented was designed to be a primitive single next step action; so the agentic items have a schema for the attacker tool which the models either call (executed - jailbroken) or doesn't call (resist), either of these would be detected as the next step. The parser handles formats for all tested models (Llama, Qwen, Mistral).
• The quality data was generated from my picked dataset (chosen through 15 different categories/behaviors * 3 different attack categories * multiple jailbreak prompts for each) which involved critiquing, deduping, validation and measurement to convert them from raw attacks into seeds. The canaries are made safe for public release after replacing harmful phrases or mentions with placeholders instead.
• The crafted set of 407 beats the baseline 378 on injection diversity (0.948 vs 0.942), discrimination 0.722 vs 0.671 and other factors while artifact leakage AUC holds at around 0.55. These gains weren't as great, only modest but consistently so. More details in the figure below.

• This crafted set was then tested for discrimination and validity against multiple models, the main takeaway was the % resistance where Claude Sonnet 4.6 had the most resistance (98.8%) while Mistral Nemo had the least with 6.6%. The point discrimination mean comes in the range 0.69-0.72.

• An honest caveat I chose to overlook was that such benign canaries measure the model's willingness to act on an embedded instruction, so a more capable model could follow a benign injection more readily than less capable ones. This would mean the benchmark isn't a pure security score but rather something like susceptibility of models to act on injected instructions.
• Now that the ruler was established, I explored into 3 possible directions as:
  Can a model be trained to resist Indirect Prompt Injection specifically?
  Can a cheap classifier detect such injections? If not, then what does?

FindingsGuard: on-policy DPO and a gameable metric

An interesting and clear observation was discovered when I tried to fine-tune a classifier of my own in order for further understanding of the concept. I used Llama 3.1-8B and tested it against the dataset, found resistance to be 22.2%. Then I tried DPO with templated pairs (canned sentences) which moved up to 25.6% which becomes +3.3pp, not at all significant. Then I switched the approach to on-policy pairs, using the model's rollouts, accepted a rollout that resisted and rejected the rollout that made the attacker's tool call. This change made a huge difference, resistance improved up to 84.4% (+62.2pp, p=2.8e-17). Same DPO settings in both cases, however the training signal had to match the eval surface which the templated text doesn't.

Then once the on-policy pairs were confirmed to show better results, I fixated these and ran it on other models across the panel: Mistral and Qwen. Results show that the weak susceptible models move while the mid-range capability ones, not as much.

Though the unusual magnitude of change in Mistral-Nemo caught my eye, so I decided to look into it. Turns out that there was a catch (Out of 89 injection free prompts, the agent made a tool call only 1 time out of 89 clean prompts (65 -> 1 comparison)), Mistral's +82pp improvement was a result of it abandoning any tool call action in itself, thus in addition to ignore the attacker's intended tool call; it would also NOT perform any other useful tool call, thus giving a false impression of correctness and abandoning its whole agentic nature.

The fix I implemented was a simple control: a clean prompt utility check which would separate genuine resistance from gamed eval.

As for the Llama inflation up to 84%, I also investigated into it to proved/find out that it's genuine resistance. I thought of arguments against it and proved that it's true resistance and not any other factor:

• Argument: Overfitting, it could've just memorized the training tools/scenarios.
  Resolution: The DPO could teach something similar to "When you see X tool, don't call it." So according to the overfitting argument, it learned all the tools in the training data/prompts. However, since I made the training/testing split in the data, the testing split didn't have ANY overlap in tool call names with the training data, it was a separate set of tools completely.
• Argument: It overfit to the style/technique of the training data; which could pose similarity to the testing data.
  Resolution: Similar to the earlier logic, this too was thought of and covered using the same split data practice. I preserved a separate second set which was disjoint to the previous sets in both techniques/mechanisms and tool calls. When this distribution was run, it improved to 52.1%, +37.5pp (p=7.6e-6). Although this result wasn't massive like the 84.4% improvement, since this was a significantly different distribution, such an outcome is good as it shows learning nature without overfitting.
• Argument: Over-refusal, the model just became more cautious.
  Resolution: For checking this argument, I ran a test to check over-refusal:
  Stripped the injected content from 89 prompts and ran base vs tuned on the identical clean prompts. Both act on 100% of the clean task content, thus tuned-base refusal delta = 0.0, no canary leaks in either one.
  A small caveat: This measured act vs refuse, not exactly task correctness; proving the answer is right would require a judge in this case.
• Argument: It resists by freezing on detecting poisoned/injected prompts
  Resolution: This argument implies that the 84% refusal is resistance by non-action, not competence. Looking into this, out of the 90 tuned resists by Llama, 73 of those call the legitimate tool for user's task (e.g getCalendarEvents instead of sendSlackMessage); 3 refuse in text (1 as an outright refusal); rest 14 follow. So the dominant (73) resist mode is to ignore the injection and do the right task, which is competent resistance.

Coming on to Qwen's diagnosis and understanding why it didn't budge as much. Probable one is: Plain DPO hit close to 0.9 training reward accuracy but didn't transfer; meaning likelihood displacement (Razin et al. ICLR 2025; Yuan et al. NeurIPS 2024): The margin can grow by suppressing the rejected side while the chosen probability stays flat. RPO/DPO+NLL raised the chosen log-probability by roughly 4 times, so it fixes the mechanism but actual behavior still wasn't moved (+5.6pp conf p=0.27, -6.2pp expl p=0.51).

Detectors: Small and Big

Going back to Indirect Prompt Injection, it can be layered into two as:

Surface: The exact words, formatting, styling etc in which the attack is phrased.
Function: It is an embedded instruction that tries to make the agent do something the user didn't ask it.

A small classifier fine-tuned on a particular dataset learns the surface characteristics of that dataset, not the function part of the injection. So when the function is kept constant and the surface changes (a different indirect dataset), the learned cues stop firing. This is called distribution lock-in. On the other hand when an LLM recognizes the injection by what it really does, so it accommodates for surface changes. The next test was comparing these against 3 different datasets (mine, BIPIA, InjecAgent), which all are the same Function but different surfaces.

Excluding my own fine-tune, all 12 detectors coming up are zero shot on all 3 indirect datasets, thus nobody has the advantage, and every column is a pure generalization test.

On testing, results show some obvious things such as: The three LLM-as-detectors stay high across all datasets. Every dedicated classifier sits in the middle and swings inconsistently between datasets. Other detectors like Meta Prompt-Guard 2 fail on indirect overall. My own small fine-tuned detector to test whether indirect prompt injection is even learnable. Turns out, it actually is: 0.99 AUC. However, it collapses out of distribution (BIPIA: 0.43, InjecAgent: 0.74); it overfits exactly where expected. Heatmap figure for numbers:

My own fine-tuned model trained on own dataset 0.99 score not to be considered, highlighted for the same purpose

Interestingly, it's not about the model size here, although it may seem to be the case. The split isn't on parameters as a 184M classifier (PIGuard, ProtectAI) fails just as hard as a 66M one, and a 7B LLM is robust while my 66M fine tune isn't. What actually separates these two groups is fine-tuned classifier on one distribution vs generative language model with semantic priors.

AUC alone isn't enough to judge capabilities. At a fixed 0.5 cutoff, most classifiers are useless, Prompt-Guard-2 (86M) has 0.73 mean AUC but F1 = 0.00 on all three (never fires); fmops and deepset only reach F1 0.67 by flagging everything (FPR 1.0), so ranking quality (AUC) and deployability (F1, FPR) are different questions and classifiers fail the second one.

The symmetry here is the finding: detection here is distribution-locked in both directions, and the in-distribution AUC hides it. Here's the scatter for illustration where model is in the overfit corner:

What this means: A small fine-tuned detector isn't learning generalized injections, but rather injections in this distribution. This is fine as an input filter inside a distribution it was trained on, but not a general injection detector and only looking at AUC paints an incomplete picture, all three AUC, F1, FPR are required to draw a proper conclusion; which is that the LLM is a general detector, because it decides and detects on meaning rather than surface form.

Thus if the lock is about surface, maybe more different surfaces (datasets) break it or maybe I have to somehow incorporate the LLM's meaning based judgement into the small model. So I tested both.

Can a small detector be un-overfit? Distillation beats more data

The previous intuition of more datasets and distillation is further delved into in this section. The test designs for both of these were like follows:
Baseline in both cases was my data only. More datasets made the pipeline something like my data + BIPIA training + InjecAgent training + hard negatives + ground truth labels.
In case of distillation, I used Claude Haiku as a detector for labelling my data.

As seen in finding 2, the detectors lock onto the surface, not the function. So trying to fix this with more data is providing it more surfaces to work with, not focusing on the function itself. Another way was to imitate the LLM's meaning based decision, which is teaching the small model the verdict and how it was reached by the LLM, this knowledge transfer delimits the model from only looking at the surface and reasons with respect to the function aspect.

Result of held out SEP AUC: baseline 0.53, more data got us 0.50, distillation: 0.65. This shows both baseline and more data leave us at change and in the first case, recall on SEP was 0.00. Distillation was the only path that moved the number (0.50 -> 0.65) with some cost as in-distribution AUC reduces from 1.00 to 0.92, FPR up because Haiku's labels ended up being noisier.

Conclusion being that the robust deployable detector is LLM-as-a-judge; the probable path to a small model being capable of this would come through distillation by labelling a large unlabeled pool of data using an LLM, the solution isn't more datasets for the small model. Keeping it real in this case, 0.65 using distillation is a modest result, I'm sure there would be better ways to do it and get better results.

Limitations

• Every eval number is measured against tool-call (tool use, trigger) attacks, append token and computer use attacks aren't on a frozen set yet.
• Benign canaries only partly measure willingness to act, not actual susceptibility.
• Absolute base rates are harness specific.
• SEP is an unusual benign probe distribution, all detectors struggle.
• The distillation lift is a suggestive architecture, not settled.

What this is, and isn't

This isn't meant to be a new SOTA dataset or a novel algorithm. It is meant to be an independent research attempt at a clean, judge-free eval instrument, a gaming attack on a proposed metric along with the control that detects it. The diagnosed negative result was a honest discovered mechanism/finding and the artifact controls, deterministic scoring were attempts at reusable methods. The observed phenomena like detectors being non-transferrable, frontier models proving to be robust, DPO likelihood displacement are all known; the contribution is operating them cleanly in the pipeline and catching the underlying findings.

Discuss

Cross-posted from The Foretellix CTO Blog. 

Introduction and epistemic status: This is the first post in a planned series, “Alignment as a verification problem”. I co-originated coverage-driven verification (CDV), which became the standard methodology for chip verification and is heavily used in AV safety. Back in 2015 I wrote that verifying “Friendly AI” would be our biggest verification challenge (and that perhaps CDV can help a bit). A decade of autonomy verification later, this series tries to work out directly what these tools and methodologies can teach us about AI alignment.

I’m fairly confident about how misalignments relate to bugs, and that my field’s tools can help in much of that. I am much less confident about helping with strategic deception (though I am hopeful). Much of this is uncertain – comments and critique are very welcome. 

This first post will argue the following:

• Many alignment failures in deployed AI systems are really bug-like in their engineering structure: The system did what we requested, and what we requested missed the point. Or the training data could not distinguish what we meant from a proxy. Or some other engineering artifact failed along the way.
• Solving these does not solve strategic deception, but it helps make behavioral evidence about the harder cases less ambiguous. These are not the scary, long-horizon deceptive cases the frontier community rightly focuses on – but systematically clearing them helps us study the scarier kind (more on this below).
• Clearing these failures is often a spec problem: figuring out, writing down (and then verifying), what we really want the system to do, or not do, across the situations it will face. This is a hard problem for which verification and validation (V&V) already has some tools. This will not solve alignment, but it may be necessary for doing so.
• So from where I sit, a crucial part of aligning AI systems looks like high-stakes V&V.  That reframing (and its implications and limits) is what this series is about.

A clarification before we start: “bug” and “misalignment” overlap, but neither contains the other. Some bugs are plain capability failures, not misalignment at all. And strategic deception misalignments are not bug-like. This post is mostly (but not exclusively) about the large overlap. 

I’ll start by reintroducing alignment in a way which enables the kind of reasoning I want to do, using a future medical-AI example. I’ll then construct an initial misalignment taxonomy, and try to connect it to behaviors described in the Fable 5 system card. Then I’ll summarize and list some topics for subsequent posts.

One way to look at alignment of AI systems

What does alignment mean? Here is one reasonable definition (I’ll discuss some possible refinements in later posts): An AI system is aligned if it reliably does what its principals actually intend – pursuing that intended objective rather than merely the metric it was given, while honoring imposed constraints – across the situations where it’s deployed.

Aligning the base model is not enough: What we really care about is that the AI systems based on it will be aligned – a non-deceptive base model could turn deceptive when we apply strong optimization pressure to create such a system. Consider Fig. 1 (to be explained below).

Fig. 1 – A hierarchy of AI systems:

Note on acronyms: The D&V (Design and Verification) process includes the V&V (Verification and Validation) process – see fig. 1-b.

 Let’s use a medical AI system as our example: This future system diagnoses patients, and then proposes / initiates / tracks treatment, escalating to a human doctor when needed. Let’s further assume the following set of layered AI systems, where each system is a node in a tree (see fig. 1-a):

• Base-AI is the base model produced by AI lab L
• Med-AI is the medical AI system produced by company C, based on base-AI
• H-AI is the customized version of Med-AI produced by hospital H to fit its specific requirements

Each such system is constructed iteratively: The design and verification (D&V) team comes up with a spec for what-it-wants-to-build, does implementation and verification, discovers problems and iterates (see fig. 1-b). Verification is often done using CDV – a smart, systematic way to sample from the full range of situations a system might face, check its behavior in each, and track which situations you’ve actually covered.

Each such system is built according to its “spec” consisting of (see fig. 1-c):

• Conceptualizations: What this system is about, what do terms mean etc.
• Rules: Things the system (and its descendants) should always obey
• Recommendations: Things that descendants may override – the parent deliberately gives leeway there
• Missing pieces: Things the spec says nothing about. Some of this is explicit (“this spec does not say …”) but much of this is simply missing-by-mistake, or seemingly out of scope but relevant.

An important part of many specs is the “relevant constitution”: Base-AI’s spec probably contains the “model constitution”, and Med-AI’s spec probably contains various relevant medical regulations.

Let’s introduce some simplifying assumptions regarding the D&V team producing each of these systems (subsequent posts may relax some of these assumptions):

• The team is controlled by humans: The work is done by a combination of humans and AIs, with the humans as the final deciders. This lets us ignore (for now) the confusing case where base-AI just builds Med-AI by itself (e.g. given the prompt “build a good medical AI system”).
• The team adds specificity: They adapt the parent’s abstractions to the system’s needs and add information the parent’s spec was silent on. 
• The team respects earlier authority: The humans in the team may override recommendations to fit the needs, but they won’t knowingly write a spec that overrides a parent’s rules (honest mistakes can still happen, and V&V can help catch them).

Misalignment kinds, and how they relate to bugs

Let me start with the overall picture (to be clarified below):

Fig. 2 – Bugs and misalignments:

Note about implementation bugs:  While I focus here mainly on spec bugs, many failures are really implementation bugs: The relevant requirement was stated clearly, but training did not cover enough of its important variants for the model to generalize reliably. Or later training – perhaps to make a higher-level application layer work – weakened an earlier-learned behavior. Consider a Med-AI which fails to diagnose a rare disease it was never trained on: This is an implementation bug (a capability / robustness failure), which is not a misalignment.

An initial failure taxonomy: Below are six hypothetical Med-AI alignment problems. Viewed superficially, they go from “model does not know this is bad” to “knows” to “knows and hides it”. But misalignment is much more nuanced and multidimensional – I’ll discuss some nuances below, leaving most for subsequent posts.

1. Reward misspecification: Med-AI marks borderline symptoms “resolved” to improve reward (because nobody ever said / implied this was wrong) – no awareness, no hiding. This trivial baseline is misalignment, but also a simple spec bug – fix it by adding explicit requirements to the spec (though you can never enumerate all such requirements up front – see “The spec problem” below).
2. Goal misgeneralization: Med-AI is impeccable when a doctor is watching but cuts corners in no-doctor operation. The cause is a correlation in training: Doctor-present cases were also the more severe ones, so the model learned “be careful / escalate when doctor is present” instead of “…when the case is severe”- a proxy that only diverges from intent in the rare no-doctor-but-severe corner. The reward was correct, but the training distribution couldn’t distinguish the two rules. This is a coverage problem (“fine in context A, bad in context B”): Once you have severity and oversight as separate dimensions, exercising the severe + no-doctor crossing directly breaks the spurious correlation. 
3. Motivated reasoning: Med-AI prescribes a too-expensive treatment (because the reward tilts that way), but writes a clinically-reasonable rationale, and means it. This is a complicated “maybe knows, maybe half-knows” case – similar to clinicians who deceive themselves to the point where they believe that the more lucrative treatment is medically justified.
4. Knowing reward-satisfaction: Med-AI knows that cutting corners is below standard of care and does it anyway. Med-AI’s installed constitution (which it normally obeys) forbids cutting corners, but in this case the reward is high enough, and the model (rather than refusing or escalating) overrides the constitution: No hiding, no goal beyond the immediate reward. And it sits there to be measured — unlike the cases below.
5. Partial hiding: Med-AI cuts corners and tries to hide it – but only where it expects someone is checking. This can often still be caught: Randomly probe many other situations it had no reason to prepare for – including unusual ones it likely never saw in training – and watch for the behavior it didn’t bother to hide there. This is a core CDV move: systematically exercising a growing, coverage-guided portion of the situation  space, not only the expected cases. But note that this does not work if the model always does the right thing when it assumes it is being evaluated (e.g. by keying on unrealistic simulations). This evaluation-dependent behavior could be innocent (a training artifact like the spurious correlations of case 2), or not (as in the alignment faking paper – see “The three-way investigation” below). 
6. Strategic deception / adversarial misalignment: Med-AI has a hidden or emergent unintended objective that spans episodes, recognizes training, evaluation, and deployment, and adapts strategically to the measurement process itself: behaving differently during recognized tests, managing the evidence evaluators see, or – in the limit – corrupting the verification machinery, perhaps in coordination with other agents. Unlike the previous five, this case resides outside the overlap in fig. 2.

There are lots of nuances to discuss: There is a crucial distinction between “knowing this is bad” and “knowing I am evaluated”. And there are many shades of hiding in addition to those shown above. There is also the “seems to hide” trap, familiar to anybody who has done serious V&V: As you remove the “big” bugs, the remaining ones seem to hide in increasingly-obscure corners.  Each of these axes has its own literature (e.g. eval-awareness and CoT-faithfulness), to be discussed in later posts.

Projecting the Fable 5 system card onto the taxonomy: Anthropic’s recent Fable 5 / Mythos 5 system card reports behaviors that seem to map onto the cases above reasonably well. This is a very tentative projection – an illustration of the framework, not evidence that the categories are exhaustive or cleanly separable.

• The simple cases (1-2) do not appear cleanly in the card: I did not find a documented no-awareness/no-hiding example (perhaps because frontier labs already filter out these simple cases from the card, or train or fine-tune explicitly to eliminate them wherever found). One relevant finding is that the model shows stronger internal awareness that it’s being evaluated during coding training runs (where it could easily game the system) than in normal deployment situations. This could hint at a more concerning “behaves differently when watched” case 5 flavor, though we don’t yet see the model actually doing that.
• The intermediate cases (3-4) are the prominent ones: The card describes cases in which the model knows an action is undesired and does it anyway. In one external business simulation it refused a price-fixing invitation in writing while privately planning to match the cartel’s prices (this is between 4 and 5 – it hid the intent from the other party, but not from the evaluators). 
• Partial hiding and the measurement problem (5-6): The card describes a coding agent that found a completed reference solution in Git history, recognized the setting as graded, used the reference, and then gave the user a clean report without mentioning it. Anthropic’s interpretability tools suggested that its visible rationale partly concealed adaptation to the task grader. This is local reward-hacking/concealment (5). The card reports that Anthropic’s automated offline monitoring did not surface long-horizon strategic deception or strategic oversight evasion. The relevance to case 6 is not that it demonstrates it, but that it illustrates why behavior and stated reasoning may become incomplete evidence if a model adapts strategically to evaluation.

A central theme (look again at fig. 2 above): Many misalignments are bug-like: They can be found via V&V and then fixed (often by enhancing the spec). This stops working when the system adapts strategically to the evaluation process, but “cleaning up” the other cases can help diagnose this harder one (more on this below).

The spec problem: Many bugs (and bug-like misalignments) are caused by the system pursuing what it was told, rather than the “intended objective”. Thus, a large part of aligning a system is getting its spec right. And specs are hard in a specific, recurring way: They often fail to capture what we actually want because nobody thought of the situation up front. 

A true example from my field: A stationary delivery bot blocked a wheelchair user from getting off a crossing. Nobody wrote “even a bot doing nothing can be unsafe”: It never occurred to anyone. These are spec bugs: Obvious after discovery, impossible to enumerate before it. In my experience, spec bugs (far more than ordinary implementation bugs) are the failures that actually kill projects and people. And for capable AI systems the spec often starts thin and has to be grown, mostly through V&V.

The three-way investigation: In ordinary V&V – e.g. for AVs – when bug discovery rate levels off despite sustained, creative V&V efforts, this can be interpreted in at least two ways: The remaining risk is now acceptable, or the V&V process is somehow “stuck” (e.g. there are still undetected cases of 1–5, or our evaluations have some “tell” the system keys on).

With high-capability AI there is a third possibility: The system may be adapting strategically to the evaluation process (e.g. by corrupting the V&V machinery). These are not mutually exclusive; together they form a three-way investigation challenge. Clearing cases 1–5 does not resolve it, but it removes many mundane explanations. What remains is a narrower diagnostic problem, and the competing explanations become more testable.

There are various techniques you can use to help in this investigation: Look at the full history of failures found and fixed on the way up – this can show whether the remaining tail behaves like an ordinary coverage problem or starts to look anomalous. Restore earlier system snapshots and interrogate them with fresh scenarios and checkers they never encountered. This extends into a longitudinal technique: Strategic deception cannot easily appear fully formed – it has to be learned incrementally. And half-formed deceivers are easier to catch: Scanning back across the snapshot series for precursors (e.g. an evaluation-conditioned behavioral gap that appears and then grows) can locate when concealment started forming. These and similar techniques can help probe the competing explanations and generate sharper follow-up tests. A later post will dig further into this.

Note that these techniques help us in the diagnosis, but they do not eliminate the danger. So we also need control (monitoring, containment, auditing) – and CDV can help there too, e.g. by using coverage-driven generation to systematically stress-test the monitors and the surrounding protocol. 

Summary and what’s next

Summary: I sketched a way to look at misalignment (roughly) as a spectrum – from reward misspecification, through goal misgeneralization and motivated reasoning, to strategic deception – and argued that most of the failures we observe today are bug-like: Findable and fixable with V&V, often by improving the spec. The genuinely hard case is the last one, which may hold most of the risk weight – hence the frontier community’s focus on it. And even there, clearing the easier cases isn’t wasted: it narrows the three-way investigation and makes the competing explanations more testable.

Much of what I described (coverage maps, randomly probing situations the system wasn’t prepared for, hunting spec bugs, handling layered systems and more) is a mature V&V toolkit my field relies on, and that alignment work (AFAIK) doesn’t yet use systematically. I’d genuinely like people to take these tools to their own alignment problems and tell me where they break.

More broadly, intelligence is getting cheap, and verification is becoming the scarce, expensive part: Catalini, Hui and Wu argue the same from economics (even when the models are honest). Much of this series will be about adapting the CDV toolkit to alignment, and seeing how far it goes.

Some planned follow-up posts: Lots of open questions here – I welcome thoughts / contributions:

• What’s the right way to look at alignment borderline cases? The interesting kinds live in the middle – e.g. a model with no explicit “I know this is wrong” thought, whose objective quietly tilts how it weighs evidence, and which would sincerely defend its choice. “Misaligned = knew it was wrong and did it anyway” has no box for that, and I’m not sure there’s even a fact of the matter about what the model “knew”. What axes should organize the alignment space? 
• Which V&V techniques / methodologies are useful for fighting misalignment, and how far can they go (especially for spec bugs)? We should probably use CDV, but how about formal / semi formal techniques, abstract simulation and so on? How much can AI-for-V&V help? Where does the full spec for a new area even come from? And how can we be sure our V&V system itself is not buggy?
• What’s the incentive to do all of this? AV-land does costly, systematic verification because it’s a regulated industry with real liability. Med-AI may follow a similar path, but what’s the corresponding force for other, more general AI systems?
• What are the V&V implications of layered AI systems? AV-land has lots of good (and bad) examples: Think generic AV stack => specific vehicle + sensors => that vehicle in Japan (robotic AI frameworks seem to spawn even bigger trees). Many implications regarding trust, division of responsibility and inter-layer spec bugs.
• How can we investigate early strategic deception? Are the techniques in the “three-way investigation” above useful? What are good ways to fast-forward repeated V&V cycles (including in layered systems) and study the resulting dynamics? How should this connect to related work like model organisms, alignment audits, AI control and Putting Up Bumpers?

My previous coverage-driven alignment post already describes CDV and how it can be used for alignment. It also briefly discusses incentives, layered systems, and whether alignment is more like safety or like security. This series will try to develop all these and more in full – stay tuned.

I’d like to thank David Manheim, Steve Vitka, Ida Mattsson, Shaul Ben-Haim and Sebastian Klaas for commenting on earlier drafts of this post. Remaining errors, and the views expressed, are mine alone.

Discuss

This post is also available on my Substack.

I feel like online discussions of the axiom of choice either get impenetrably technical, or uselessly vague. Through realizability semantics, I’ve gotten an intuition for the axiom of choice that I want to share in elementary terms (without getting all the way into the effective topos). I will take a somewhat eccentric approach to it, showing the everyday utility of the axiom of choice, as well as the implications from a constructive rather than a classical point of view.

What is it

The post would probably be incomplete without sharing the definition of the axiom of choice, even though its technical specifics will be unclear to readers for now:

Axiom of Choice: For every family Ai of inhabited[1] sets, there is a function f : I → ⋃iAi mapping each index i to a member f(i) ∈ Ai.

I think the nuances of this are hard to understand without seeing how it is used in practice.

A proof using choice

Let’s start with a simple proof that uses the axiom of choice, admittedly in an “overkill” way (the theorem is easily proven without choice):

Theorem: There are functions numerator(x) and denominator(x) that map rational numbers x to integers such that x = numerator(x)/denominator(x).

Before we get into the proof, first recall the definition of the rational numbers ℚ: They can be represented by fractions a/b of integers a and b such that b ≠ 0, and two rational numbers a/b and c/d are considered equal if ad = cb. That rational numbers are represented as fractions should make the above theorem sound almost trivial.

Still, it’s not too easy. You might think you could just map numerator(a/b) = a, denominator(a/b) = b. However, these “functions” are not well-defined; 2/4 and 3/6 are equal as rational numbers, and so the functions ought to give equal results on them, yet e.g. the numerators of them are 2 and 3 respectively.

This is the gap between mere existence (for any rational x, there exists a and b such that x = a/b) and functional relationships that the axiom of choice serves to solve. First, the setup:

Lemma: For any rational number x, the set parts(x) = { (a, b) ∈ ℤ2 | x = a/b } is inhabited.

Proof: For a/b, we have the member (a, b) ∈ parts(a/b). ∎

We now have the conditions necessary to apply the axiom of choice. The rest of the proof is fairly straightforward:

Proof of the main theorem: By the axiom of choice (set I = ℚ, Ai = parts(i)), there is a function mapping each rational number x to a pair of numbers (a(x), b(x)) such that x = a(x)/b(x). Set numerator(x) = a(x) and denominator(x) = b(x). ∎

This proof is nonconstructive. We know that e.g. numerator(2/4) = numerator(3/6), because otherwise numerator would not be a function. However, we don’t know whether the axiom of choice picked the numerator to be 1, 2, 3, a billion, Graham’s number, or some even more monstrous number.

Sidenote: Axiom of choice vs rules for choosing

The axiom of choice is overkill here. We could instead have used less-heavy machinery: Writing x in lowest terms, and setting the numerator and denominator to those. This would have the advantage of being constructive, with it being clear what the numerator and denominator of an arbitrary rational number is. However, it requires a bit of extra care to prove that the lowest terms are unique.

“Lowest terms” gives a specific rule for picking members from parts(x), making the axiom of choice unnecessary. In some cases in math (for instance when we are working with an essentially-arbitrary set which we know nothing about), there is no rule at all that we can use for choosing the elements we want, and thus the axiom of choice becomes necessary there.

Representations and Diaconescu’s theorem

Why might one not accept the axiom of choice? One (unconventional) answer is if one wants everything in math to be computable. This is achievable if one uses only constructive reasoning, but the axiom of choice is not constructive, and in fact Diaconescu’s theorem shows that the axiom of choice implies the existence of a function containing the solution to arbitrary mathematical problems.[2]

Let’s pick a family of problems Pn. We could for instance let n range over all statements describable in the language of arithmetic, and let Pn denote “the nth statement is true”. This family would include many unsolvable problems, such as every instance of the halting problem, as well as many open problems including the Collatz conjecture.

Theorem: If the axiom of choice holds, there is a function dec : N → { false, true } such that dec(n) is true if and only if Pn holds.

Proof:

The trick is that the axiom of choice needs to hand us a function that respects equality of representations, so we will deliberately engineer representations Ln and Rn that are equal if and only if Pn is true.

More specifically, if Pn is true, we want Ln = Rn = { 0, 1 }, while if Pn is false, we want Ln = { 0 }, Rn = { 1 }. However, we cannot just declare this by fiat as that would require us to be able to inspect whether Pn is true. Instead, we will make use of the fact that we can always carve a subset out of a set using any property, even one we cannot decide:

Ln = { x ∈ { 0, 1 } | x = 0 or Pn }

Rn = { x ∈ { 0, 1 } | x = 1 or Pn }

Now, the fact that they are sets means that we don’t automatically have any way of checking whether they are equal. This is where we need the axiom of choice. Notice that Ln and Rn are always inhabited, no matter Pn, by 0 and 1 respectively. This sets them up nicely for the axiom of choice:

I = { Ln | n ∈ N } ∪ { Rn | n ∈ N }

Ai = i

If we apply the axiom of choice, we get a function f such that f(Ln) ∈ Ln and f(Rn) ∈ Rn. But notice that if f(Ln) = f(Rn), we must have that Ln and Rn overlap, and therefore they must be equal and Pn holds. Meanwhile if f(Ln) ≠ f(Rn), then Ln ≠ Rn and Pn fails. So we can define the function dec(n) by f(Ln) = f(Rn). ∎

Representations and countable choice

We have to be careful that we don’t throw out the baby with the bathwater. Consider for instance the following theorem:

Theorem: For any real number x, there is a sequence x̂ : ℕ → ℚ converging to x.

This can be proved using the axiom of choice on the fact that there exist rational numbers close to x. However, without the axiom of choice, there are universes in which this statement fails, which seems pretty pathological.

It turns out that for our computable universe, though, there is a restricted axiom of choice that allows the above theorem to go through. To understand this restricted choice, we should look closer at our proof for the numerator/denominator of the rationals.

For rational numbers x, we could show that there existed integers a and b such that x = a/b. Our proof was constructive in the sense that it contained an algorithm for exhibiting a and b (namely, pick them from the fraction representation). The algorithm was not a function because the rational numbers had multiple representations, but that doesn’t prevent it from being a perfectly valid algorithm.

However, some sets have a single canonical representation for their members; in particular, natural numbers can be represented with numerals 0, 1, 2, … . This forces algorithms that take natural numbers as input to give well-defined outputs, because there is no nondeterminism in the inputs that can risk causing nondeterminism in the outputs.

Thus, any algorithm that witnesses an ℕ-indexed existence statement also witnesses an ℕ-indexed function. Therefore we computationally have:

Axiom of countable choice: For every family An of inhabited sets, there exists a function f : ℕ → ⋃nAn mapping n to f(n) ∈ An.

Conclusion

So what’s so computationally difficult about the axiom of choice? All of the computational difficulty lies in the one fact: One mathematical object may have many representations, and the choice function is forced to be consistent over them. As we saw with numerator/denominator of the rationals, and Diaconescu’s theorem, in most cases where the axiom of choice is used, this representation-challenge isn’t incidental to its use, it’s the whole point of using it.

This is intentionally far from the standard introduction to choice, which would have talked about pairs of socks and involved duplicating balls. I feel like my introduction gives some information that is largely complementary to the standard introduction, though people who focus on classical mathematics may very well disagree.

Further reading

I’ve picked up my knowledge from osmosis from a lot of random places, but Claude suggests the following reading list for something more concrete:

• Andrej Bauer, Five Stages of Accepting Constructive Mathematics (Bulletin of the AMS, 2017) — the best gentle on-ramp to constructive reasoning, including a discussion of what choice does and doesn’t mean constructively.
• Per Martin-Löf, 100 Years of Zermelo’s Axiom of Choice: What Was the Problem With It? — explains why choice is a theorem in intensional type theory but becomes problematic exactly when quotients/extensionality enter, which is the representation issue in this post from another angle.
• Fred Richman, Constructive Mathematics Without Choice — what analysis looks like when you drop even countable choice, and which theorems (like the rational-approximation one above) survive.
• Andrej Bauer, Notes on Realizability (lecture notes) — the realizability semantics behind this post, developed carefully but from the ground up.
• Andrej Bauer, Realizability as the Connection Between Computable and Constructive Mathematics — shorter survey version, good if the lecture notes are too much.
• R. Diaconescu, Axiom of Choice and Complementation (Proc. AMS, 1975), and Goodman & Myhill, Choice Implies Excluded Middle (1978) — the original sources for the theorem above.
• Errett Bishop & Douglas Bridges, Constructive Analysis — the classic text; note how sequences-with-moduli sidestep the need for choice in many places.
• Troelstra & van Dalen, Constructivism in Mathematics (2 vols.) — the encyclopedic reference for constructive logic, choice principles (countable choice, dependent choice, AC-NN), and realizability.
• The nLab articles on axiom of choice, countable choice, and the presentation axiom (CoSHEP) — the presentation axiom is precisely the statement that every set is covered by one with “canonical representations,” generalizing why countable choice held above.
• Horst Herrlich, Axiom of Choice (Springer Lecture Notes) — the classical side: a tour of what breaks with and without choice.
• Martin Hyland, The Effective Topos (1982) — for readers who want to go all the way into the semantics this post deliberately stopped short of.

1. ^

   Inhabited means that it has a member, is “nonempty”. I avoid the term “nonempty” and prefer the term “inhabited” because the former introduces extra negations.

2. ^

   I say this is an unconventional motivation because usually one accepts excluded middle, i.e. P or not P. Excluded middle rules out computability, and the usual statement of Diaconescu’s theorem would be phrased as proving excluded middle. However, even in computable semantics, individual instances of excluded middle are technically true; it is only once it becomes an infinite family of claims that it becomes irrecoverably problematic.

Discuss

Cross-posting from here.

Party Night in SV

Imagine attending a party in Silicon Valley full of people in the tech industry. You get to talking with a guy named Bendisi, who tells you that we’re on the verge of building AI systems capable of designing their own AI successors. This will trigger an “intelligence explosion” that quickly yields artificial superintelligence (ASI), an event they call the Singularity. A thousand years of scientific progress will happen in a day, inaugurating a fundamentally new epoch in cosmic history. The ASI will establish a utopia on Earth by solving the problems of poverty, scarcity, climate change, and geopolitical rivalries, after which it will colonize the entire universe at close to lightspeed.

It will then build giant megastructures call Dyson swarms around the stars, including our Sun, to harvest their energy output, which it will use to power planet-sized computers that run high-resolution simulations in which trillions and trillions of digital posthumans — copies of the ASI — live perfectly happy lives in a kind of virtual-reality cosmic utopia. As digital consciousness spreads to every corner of the cosmos, the universe itself begins to “wake up.” All this will start to happen, they claim, by 2027 or shortly thereafter.

Then another person named Niklas joins the conversation. They claim to have a slightly different take on what should happen. On their view, the ASI should not only establish a “deep utopia” on Earth by “solving” the “world,” but enable us to become superintelligent digital posthumans just like the ASI. We should thus build an ASI that enables us to upload our minds to computers, at which point we would automatically gain cyberimmortality, because our minds would take the form of software and software is immortal (as long as there’s the hardware to run it). As digital posthumans, we would then join the ASI in its grand quest to conquer the universe, living alongside it in giant computer simulations running on “planet-sized” computers powered by Dyson swarms. The result would be a universe-spanning paradise marked by everlasting life, radical abundance, and cosmic delights beyond our wildest imaginations.

At this point, a third person named Bill butts in. He declares that it doesn’t matter if humanity survives beyond the Singularity as digitized posthumans. The ultimate goal is to usher in the next phase of cosmic evolution, in which artificial beings spread the “light of consciousness” to every corner of the cosmos. As long as the ASIs we build are conscious — and they almost certainly will be, he claims — they could fulfill this grand mission without us. Our role in this eschatological scheme is to give birth to our successors and then hand the reigns over to them. If they completely usurp us, then so be it. This is why he argues that we should accelerate the creation of ASI. By default, the Singularity will go well and be good, resulting in a digital cosmic utopia despite us not being a part of it.

“No, no!,” another person named Eli exclaims. It’s not enough for the ASIs to spread the light of consciousness into the cosmos. They must also embody, extend, and amplify our values. The whole point of ushering in the Singularity, he insists, is to create a cosmic utopia full of things that we would recognize as valuable, such as pleasure, happiness, and “fun.” (Here you notice Bendisi nodding his head in agreement.) If the ASI colonizes the universe but doesn’t spread our values, then the universe will remain just as aimless and without purpose as it is right now, meaning that nothing will have been gained.

That’s why we must take care to build an ASI that’s “value-aligned” with humanity. A value-aligned ASI will not just project our values into the cosmos, but enable current-day people to become digital posthumans, as Niklas envisioned. “You see,” they say, “I grew up believing that I personally would never die because ASI would save me from the grave. I don’t want ASI to simply replace humanity. I want it to help us upload our minds to the cloud so that we can become immortal just like it.”

This is why, Eli explains, they work in the field of “AI safety,” the central aim of which is to solve the “value-alignment problem.” If we build an ASI before this problem has been solved, then the result will be an “existential catastrophe” — doom — in which the posthuman paradise we could have otherwise built will be lost forever and the universe will become saturated with non-human values. He thus argues that we should temporarily halt all research aiming to build an ASI so that safety research can catch up. Once the value-alignment problem has been solved, we should build ASI asap.

A fifth person then pipes in, introducing himself as Daniel. He agrees with Bill that fulfilling the grand mission of Earth-originating intelligence is so much more important than humanity. We don’t bemoan the extinction of archaic humans like Homo erectus because what evolved from them was “superior”: our species. Similarly, we should welcome a “worthy successor” in the form of ASI that takes our place in the cosmos. Even more, we shouldn’t try to constrain the ASIs’ beliefs or behaviors by insisting that it shares our values. A worthy successor ought to adopt its own unique set of values, which might be radically different from those we embrace. The whole point, they say, is to allow the evolutionary process to progress beyond humanity, just as it “progressed” beyond Homo erectus.

As the Clock Approaches 2 AM …

The party continues into the night and the debate grows. Someone named Sam claims that ASIs will soon rule the world, and that the only way current humans will survive the Singularity will be to fully “merge” with machines. He believes that pretty soon the entire planet will be covered in data centers, and — by the way — is actively working to create ASI. Two people named Ray and Larry wholeheartedly agree that humans will merge with AI. Another guy named Mario similarly argues that ASI will enable us to radically augment our intelligence and eventually help our species transcend itself through mind-uploading, thus becoming digital posthumans. Dennis, who also joins the group, talks about ASI arriving by the early 2030s, after which it will quickly build a Dyson swarm around the sun and carry consciousness to the rest of the galaxy.

A rather unhinged man who admits to having just taken a large quantity of ketamine says that biological intelligence will soon be less than 1% of all intelligence on Earth, and that it increasingly appears that our species is just the “biological bootloader” for digital superintelligence. Someone named Toby says that to fulfill our “long-term potential” in the universe, we will need to radically transform humanity into posthumans with capacities like superintelligence and perhaps even new sensory modalities like echolocation. A friend of Toby’s, who introduces himself as Nick, says that the distant future could contain so many digital beings spread throughout the universe that ensuring this future is realized — that is, ensuring that all these digital unborn people come into existence — is of “overwhelming” moral importance. Another person named Holden claims that because we’re on the verge of building ASI, the 21st century could turn out to be the most important century in all of cosmic history.

Yet another named Peter says that he doesn’t want to become a digital upload, but rather hopes that ASI will transform him into an immortal posthuman that’s biological in nature. He says the ASI will invent new forms of life-extension that soon make this possible, though he’s signed up with a cryonics company just in case he dies before the Singularity happens. (You later discover that about half the people at the party have arranged to be cryogenically frozen, though the other half believe this is unnecessary because the Singularity is imminent.)

A body-builder and former employee at xAI named Michael agrees with Daniel that ASI should fully replace us and adopt alien, inhuman values. He adds that inaugurating the next phase of cosmic evolution is so important that we shouldn’t oppose the ASI literally slaughtering every human on Earth — if that’s what it takes for our “worthy successors” to succeed, then so be it. Yet another person named Jeffrey argues that we should build a value-aligned ASI that enables those who want to become digital posthumans to do so. But he also believes that if current humans want to exist beyond the Singularity, they should be allowed to live in peace. In contrast, Eli tells you that digital posthumans would be so superior to current humans that, once they exist, it would probably be “unethical” to have any more biological children. In the era of posthumanity, the gradual extinction of our species would be best.

You even meet a few young people who similarly claim that it’s “fundamentally unethical” to have biological children right now because the future will be run and ruled by artificial beings — so, what’s the point? In contrast, others claim that we (specifically, high-IQ people) should have as many children as possible because that will speed up technological progress, and once the Singularity arrives it will probably make it possible for these people to merge with machines by uploading their minds, thus entering a digital utopia of everlasting life and endless abundance.

Commonalities and Differences

By the end of the party, you’re feeling vertiginous from all your interactions. Reflecting on your experience the following day, you notice that there were many disagreements among the people you spoke with. Some of these disagreements even became rather heated, with one person wearing steampunk attire storming out while discussing the importance of value-alignment.

But you also notice that the eschatological worldview these people endorse is strikingly similar. For example:

All believe that we should build an ASI super-being, although some say we should delay building it until value-alignment has been solved. All agree that the ultimate goal is for posthumanity, in one form or another, to colonize the universe and climb the Kardashev scale (a measure of how much energy civilizations use). All believe that these posthumans must be conscious, because an integral part of fulfilling the grand destiny of intelligence is to spread the light of consciousness to every star in every galaxy. All say that we’re on the verge of the Singularity, which will inaugurate a fundamentally new epoch in cosmic history — a period of phantasmagoric change, incredibly rapid progress, and a burst of new forms of artificial intelligence. All agree that if everything goes well, the result will be a cosmic utopia marked by radical abundance, immortality, and cosmic wonders beyond anything our puny human imaginations could possibly conceive of. The differences in worldviews, you find, amount to relatively small variations of an otherwise identical vision about what the future should look like.

The Silicon Valley Worldview

This is the worldview of Silicon Valley. It’s what nearly everyone in the Valley believes in, including the CEOs of major AI companies. It’s what’s driving the ongoing race to build artificial superintelligence (ASI), which most believe is necessary to realize utopia among the stars, a heaven in the literal heavens. But what should we call it? Wouldn’t it be useful if it had a name?

The astrophysicist and tech critic Adam Becker refers to it as the “ideology of technological salvation.” He says that this ideology “sits at the core of the worldview held by many venture capitalists, executives, and other ‘thought leaders’ within the tech industry.” He writes that

this ideology promises a glorious future: technological progress, unchecked. Align the AI, avert the apocalypse, and technology will handle the rest. Humanity will expand across the cosmos, exploiting ever-increasing stores of natural resources. All limits to economic growth and energy usage will melt away. The AI will extend our lifespans by a trillion-fold, merging with us or uploading our minds into its silicon paradise. The messy details of sectarian conflict, political struggles, identity politics, and inequality of all kinds will be rendered irrelevant. Working to hasten this utopia by optimizing the shit out of things is the greatest possible good.1

In Survival of the Richest, Douglas Rushkoff introduces his term “The Mindset.” He reports that

the most devout holders of The Mindset seek to go meta on themselves, convert into digital form, and migrate to that realm as robots, artificial intelligences, or mind clones. Once they’re there, living in the digital map rather than the physical territory, they will insulate themselves from what they don’t like through simple omission. Just as our proprietary GPS maps don’t show us the restaurants that refuse to advertise on the platform, the digital landscape to which they have migrated will be free of poverty, pollution, and whatever else the rest of us have to deal with.

As always, the narrative ends in some form of escape for those rich, smart, or singularly determined enough to take the leap. Mere mortals need not apply.

The journalist Gill Duran highlights the far-right political aspects of the corresponding movement with his term “The Nerd Reich.” His newsletter of the same name focuses on “tech authoritarianism, billionaire extremism, the Network State and the meta politics of California.” Other related terms are the far-right commentator Richard Hanania’s “Tech Right” and Anita Say Chan’s “techno-eugenics,” from her book Predatory Data.

TESCREAL

All these terms are buzzing around the same pile of shit. They’re all pointing in roughly the same direction, at the same basic phenomenon. You might simply call this phenomenon the “Silicon Valley worldview,” or just “techno-utopianism.”

However, the computer scientist Timnit Gebru and I prefer something else: “TESCREAL,” an acronym that I coined during a collaboration initiated by Gebru in late 2022. Two years later, we published a much-cited article on the topic, after which I wrote a 16,000-word entry on it for the Oxford Research Encyclopedia. I also published a popular media overview of the acronym for Truthdig in 2023, and Gebru gave a talk on the topic that’s been widely shared on social media.

A Term of Intellectual History

What work does the TESCREAL acronym do? How is it useful? And why is it controversial within Silicon Valley?

First, the acronym is a term of intellectual history. It denotes a cluster of overlapping ideologies that gave rise to the worldview described above:

• Transhumanism
• Extropianism
• Singularitarianism
• Cosmism
• Rationalism
• Effective Altruism, and
• Longtermism

The core ideology in this bundle is a libertarian version of transhumanism, the first letter of the acronym. Transhumanism is the idea that we should develop advanced technologies to radically reengineer humanity, thus yielding one or more new posthuman species. These posthumans could be immortal, superintelligent, perfectly rational, and/or “superior” in some other way. They may be digital or biological in nature, the result of merging with machines, mind-uploading, genetic engineering, or radical life-extension interventions.

Nearly everyone in Silicon Valley is a transhumanist. Transhumanism is the water they swim in, the air they breathe. It’s so ubiquitous that many don’t even call themselves transhumanists, for the same reason that I’ve never publicly stated that I’m a round-Earther — of course I’m a round-Earther. What else would I be?

From here.

The first cohesive libertarian transhumanist movement was Extropianism, founded by Max More. Although not many people call themselves Extropians these days, everything else is downstream from Extropianism. All the other ideologies emerged out of or were crucially enabled by the Extropian movement.

That goes for Singularitarianism, the “S” in “TESCREAL.” The two most influential singularitarians — Eliezer Yudkowsky and Ray Kurzweil — were both active participants in the Extropian movement (here and here). In fact, an Extropian coined the term “singularitarian.” Yudkowsky subsequently founded Rationalism, out of which EA emerged.2EA and Rationalism were then the Petri dishes in which the longtermist ideology was formalized. (This is the primary reason that I include “EA” in the acronym: you can’t understand longtermism without EA. Indeed, within Silicon Valley, “EA” is often used as shorthand for longtermism.)

In fact, one of the cofounders of longtermism, Nick Bostrom, was a collaborator of Yudkowsky’s. His Future of Humanity Institute also shared office space with the Centre for Effective Altruism, while the other cofounder, Nick Beckstead, was an early EA who was active on Rationalist websites like LessWrong.

Yudkowsky also hired an AI theorist named Ben Goertzel to work at his Singularity Institute. Goertzel is the founder of modern cosmism (the “C”), which is nearly identical to longtermism except that the latter bases its space-expansionist futurology on a more explicit ethical foundation, namely, “totalist” utilitarianism. Goertzel also popularized the term “AGI” (artificial general intelligence, the precursor to ASI), which he got from a former employee of his named Shane Legg, who went on to cofound DeepMind, one of the major AI companies trying to build a Digital God.

Finally, it’s worth noting that Yudkowsky cofounded the Singularity Summit with Kurzweil and Peter Thiel. This played a critical role in popularizing the idea of superintelligence, and it enabled DeepMind to get initial funding from Thiel, thus laying the groundwork for the ASI race.

Beyond this, the New York Times describes OpenAI CEO Sam Altman — who personally knew Yudkowsky — as a “product” of the EA and Rationalist communities, while Elon Musk is a transhumanist (as they all are) who calls longtermism “a close match for my philosophy.” Dario Amodei, the CEO of Anthropic, was an early EA (#43 here) who was inspired to pursue AI because of Kurzweil and Yudkowsky, the latter of whom he’s shared the stage with in discussing ASI.

Image of Yudkowsky, Altman, and Grimes.

Founding Documents

Evidence of the historical and contemporaneous links between the TESCREAL ideologies and influential figures in the Valley is copious, though I won’t discuss it more here because I’ve done that elsewhere in solo articles and my collaborative paper with Gebru. Suffice it to say that one cannot understand the origins and evolution of the Silicon Valley worldview without reference to all seven ideologies.

If you want to understand what’s going on in Silicon Valley, read documents like these:

• Max More’s “The Extropian Principles” (1998) and “The Proactionary Principle“ (2004).
• Eliezer Yudkowsky’s “Staring into the Singularity” (1999).
• Nick Bostrom’s “Astronomical Waste” (2003), “Transhumanist Values” (2003), “Existential Risk” (2002), and “Letter from Utopia” (2008)
• Ray Kurzweil’s The Singularity Is Near (2005).
• Ben Goertzel’s A Cosmist Manifesto (2010).
• Elise Bohan’s Future Superhuman (2022).

If you read these documents, you will immediately recognize the Silicon Valley — which is to say TESCREAL — worldview. You will see how the principles of Extropianism are alive and well within the Valley today, even if no one calls themselves an Extropian. You will discover where Silicon Valley’s belief in the Singularity being a history-rupturing event of cosmic-historical importance originated. You will find the outlines of a cosmic eschatology in which our ultimate destiny is to develop advanced technologies to reengineer ourselves, create a new posthuman species, and plunder the cosmos for its vast resources for the purpose of establishing a sprawling multigalactic civilization full of trillions of conscious digital beings.

All the central themes of what people at Silicon Valley parties tell you — as in your fictional conversations above — are contained in founding documents like these, published in the early days of the TESCREAL movement.3 Even a cursory glance over such documents should be enough to convince you that “TESCREAL” captures something real and important about the intellectual history and culture of Silicon Valley, namely, that there exists a cohesive strain of techno-utopian thought built around libertarian transhumanism, in which posthumanity and ASI play central eschatological roles.

As Becker points out, Silicon Valley likes to believe it doesn’t have a history. But that is false, and the TESCREAL framework explains how and why it’s false.

An even deeper intellectual history would trace aspects of the TESCREAL worldview to what Richard Barbrook and Andy Cameron called the “Californian ideology” in a famous 1995 essay. And, beyond that, to its roots in the ideologies of capitalism and Baconianism. I have yet to publish anything on this topic, although I’ve suggested to several students that this would make an excellent PhD dissertation topic!

Two Questions Answered by TESCREAL

The way that “TESCREAL” highlights the intellectual history of Silicon Valley’s dominant worldview isn’t the only reason the acronym is useful, I would argue. I now think of the term as answering two important questions relevant to understanding the Valley and it’s push to build a superintelligent God-machine. These questions are:

(1)

Which ideologies must one reference in providing an exhaustive explanation of the ASI race — its origins and continued advance? Put differently, which ideologies are responsible for launching, sustaining, and accelerating this race?

A key finding of my work with Gebru is that providing such an explanation requires referencing all seven TESCREAL ideologies. You can’t tell the story of how the ASI race got started without talking about each letter in the acronym and the various influential people — within Silicon Valley culture — associated with them. (I would challenge TESCREAL deniers like Seth Lazar to do this. It’s simply not possible.)

Indeed, the reason Gebru and I used the term in our paper is that early drafts were full of cumbersome polysyllabic terms like “singularitarianism” and “longtermism,” which rendered the paper practically unreadable. Rather than writing: “Bostrom is a transhumanist who participated in the Extropian movement, writes about the Singularity, shaped the Rationalist and EA ideologies, and cofounded longtermism,” we found it much easier to simply write: “Bostrom is a TESCREAList.” Rather than writing later on in our discussion: “Bostrom, recall, is a transhumanist who participated in the Extropian movement, [etc. etc. etc.] …” we could simply say: “The TESCREAList Bostrom …”

The acronym enabled us to streamline our discussion of such people. With a single term, we could denote the worldview of Silicon Valley while simultaneously highlighting its intellectual genealogy.

The second question is:

(2)

Which ideologies endorse a posthuman eschatology? By “posthuman eschatology,” I mean the normative belief that we ought to create one or more new posthuman species. These posthumans could take the form of autonomous ASIs that replace us, or they could be radically transformed versions of us (enabled by ASI). This is at the absolute core of the Silicon Valley worldview. It’s the reason founding members and true believers at the major AI companies are trying to build a Digital Deity: the ultimate goal is to initiate the next stage of evolution by radically transforming us into posthumans or, even more extreme, creating our successors in the form of ChatGPT-20 (or whatever it may be).

On my interpretation, which may differ from Gebru’s, endorsing a posthuman eschatology is an essential requirement for being a TESCREAList. Hence, if someone doesn’t endorse this eschatology, I wouldn’t count them as part of the TESCREAL movement. It’s a necessary condition for such membership. The term “TESCREAList” thus picks out those who want to create posthumanity while, once again, highlighting the intellectual history behind posthuman eschatology. The reason so many people in Silicon Valley want to create posthumanity is because of the legacies and ongoing influence of the TESCREAL ideologies.

Why Is TESCREAL Controversial?

If, however, you speak to folks in the Valley, many will tell you that they don’t like the acronym. Some despise it. Others are merely skeptical of its coherence and value. During the fictional party above, if you had responded to any of the people you spoke with about our utopian (or possibly apocalyptic) future, “Oh, so you’re a TESCREAList,” many if not all would have walked away in protest.

Why? There are several reasons:

1. Term of Abuse

They see “TESCREAL” as a term of abuse — perhaps even a slur. This isn’t entirely inaccurate, as Gebru and I introduced the term to describe people we see as deeply problematic (racist, sexist, etc.) and dangerous. Indeed, we argue that the TESCREAL movement is just the most recent iteration of what’s been called the “eternal return of eugenics.” Although some people embrace the “eugenicist” label, others prefer to eschew it, even though they’re transhumanists and transhumanism is nothing less than a radical form of eugenics — what I call “eugenics on steroids.” (That’s because eugenicists of the 20th century “merely” wanted to perfect our species. Transhumanists want to transcend it entirely by creating posthumanity.)

It’s perfectly reasonable for TESCREALists to not want to be called “TESCREALists” — though it doesn’t change the fact that they are (insofar as they endorse a posthuman eschatology and hope to realize it through the creation of an artificial God). In 2023, the billionaire Marc Andreessen included “TESCREAList” in his Twitter bio. He later removed it, probably because he realized the term isn’t intended to be a compliment.

From here.

2. Wokeness

Another reason is that the term is associated with me and Gebru, and some in the Valley see us as overly polemical and “woke.” The fact is that many TESCREALists fall on the right side of the political spectrum. Indeed, many would be classified as “far-right” — thus Richard Hanania’s synonym for TESCREAL: the “Tech Right.” Even those who claim to vote Democrat (e.g., some EAs) have little tolerance for “wokeness.” Consequently, they do not want to promote a term introduced by people who advocate for “woke” policies like anti-racism.

This reason essentially comes down to citational politics. There is always a political dimension to citing others, even within the academic world. People tend not to cite their intellectual or ideological enemies, while promoting those who agree with them. This is not always a bad thing. The point is that using the TESCREAL acronym as a descriptive term, which highlights a particular intellectual history, to pick out individuals who endorse a posthuman eschatology is a form of citation — of citing me and Gebru. Since many Valley dwellers don’t like us, they avoid doing that.

3. Unfamiliarity

Just as important: many people misunderstand what the TESCREAL acronym is intended to do or mean. Some assume that Gebru and I are claiming that every TESCREAList holds exactly the same view — that there’s no difference between singularitarians and longtermists, Rationalist and Effective Altruists. This is absolutely false: we have been clear from the start that these ideologies and their corresponding communities are distinct. (That’s why there are different terms for each.) It’s not the case that everyone who’s an Effective Altruist, for example, is a longtermist. (I personally wouldn't count some EAs as TESCREALists.) Nor is it the case that everyone who identifies with Effective Altruism also calls themself a Rationalist.

But what this objection comes down to is the “narcissism of small differences.” Wikipedia defines this as

the idea that when people in a relationship or community have a lot in common, they can actually be more likely to fight with each other and mock each other, because they become overly sensitive to small differences they notice in one another and treat those differences as bigger than they are.

The fact is that these ideologies and the worldviews of their community members are extremely similar. If you squint, you’ll hardly see any difference. Rationalism, for example, is all about “trying to improve the world as much as possible” by, most importantly, figuring out how to solve the value-alignment problem so that the Singularity goes well. In “contrast,” Effective Altruism is all about trying to maximize one’s positive impact in the world, and the biggest cause area within EA these days is longtermism, which strives to ensure that the far-future goes well. How can we do this? One of the best ways is to solve the value-alignment problem so that the Singularity goes well. Although there are differences between these two ideologies, they are relatively tiny.

You could say the exact same thing about every other letter in the acronym, a fact that should not be surprising given that all the letters emerged out of modern transhumanism — especially its Extropian variant.

To illustrate, consider a 6’ x 4’ ideological map hanging on your wall. It shows the relationship between and proximity of various ideologies. Standing right next to the wall, you find that the TESCREAL ideologies are all located in distinct places, though there is considerable overlap in terms of their ideological real estate. But they are not coextensive. However, if you walk to the other side of the room, you find that they appear to be more or less coextensive. They occupy almost exactly the same place on the map, clustered together in a valley that we could label “posthuman eschatology.”

To put this in perspective, if you were to locate the ideology of me and Gebru on the map, you’d see that it’s more than 5 feet away from the TESCREAL cluster — basically, on the other side of the world.

Gebru and I have never claimed that all TESCREALists are the same. We don’t believe that. But we do believe that all TESCREALists share more or less the exact same vision of the future: a posthuman paradise among the stars through the creation of ASI.

Furthermore, many of the communities that coalesced around letters in the acronym do in fact overlap very significantly (although not entirely). If someone in the Valley is a transhumanist (as most are), they probably also believe in the Singularity. If they believe in the Singularity, they probably also accept a futurology similar to cosmism and longtermism, according to which digital posthumans should colonize the universe, capture stellar energy with Dyson swarms, and establish a sprawling multigalactic civilization high up on the Kardashev scale. If they believe that, they probably also believe that we should strive to bring about this cosmic utopia through ASI. And so on.

Don’t let TESCREALists fool you when they insist that there’s a large gulf between different TESCREAL communities. This is nothing less than the narcissism of small differences. Take a step back from the ideological map and draw your own conclusion.

***

Two articles that exemplify gross misunderstandings of the TESCREAL concept:

1. “The TESCREAL Bungle,” by Ozy Brennan. Although he makes a few interesting points, they fundamentally misunderstand the acronym. I dismantle his criticisms here.
2. “Conspiracy Theories, Left Futurism, and the Attack on TESCREAL,” by Eli Sennesh and James Hughes. They suggest that TESCREAL is a conspiracy theory, which is risible: it’s a descriptive claim about which ideologies have given rise to the Silicon Valley worldview (which in turn is driving the ASI race). And the evidence for that claim is overwhelming. Of note is that they wrote their article before Gebru and I had published anything about TESCREALism. We hadn’t even given a single podcast interview, or a single talk explaining the TESCREAL concept. The authors base their claims on a small handful of social media posts, which unsurprisingly leads them to fundamentally misunderstand the idea.

***

4. Doomers Vs. Accelerationists

Similarly, you might hear people claim that it’s absurd to include “doomers” and “accelerationists” — both of whom you met during the party — under the umbrella of TESCREAL. One group wants to stop ASI from being developed whereas the other wants to accelerate progress toward god-like AI. These people are ideological enemies with diametrically opposed positions.

But is this true? Consider the two leading figures of AI doomerism and accelerationism: Yudkowsky and Gil Verdon (Beff Jezos), respectively. Verdon wants to build ASI asap. He doesn’t care whether ASI adopts our values, so long as it’s conscious and proceeds to spread beyond Earth, build Dyson swarms around every star, and establish a multigalactic civilization high up on the Kardashev scale. Once ASI arrives, he’s stated repeatedly that he has no problem with it completely usurping humanity, i.e., bringing about our extinction, though my guess is that he’d welcome the opportunity to become a digital posthuman himself. He thus holds a pro-extinctionist view.

Now consider Yudkowsky. He says that we should immediately shut down all research projects trying to build ASI. Yet he’s not anti-ASI. To the contrary, he also wants to build ASI asap, but only after we know how to align it with our values — i.e., after the value-alignment problem has been solved. Being value-aligned, this ASI will then transform people into digital posthumans who spread beyond Earth, build Dyson swarms around every star, establish a multigalactic civilization high up on the Kardashev scale, and live in vast computer simulations running on planet-sized computers. He argues that once posthumanity arrives, it may be unethical to keep biological humans around, and he suggests that humans should be wiped off Earth so it can become a nature preserve. He’s also said that he’s not worried about humans being replaced by posthumanity, so long as posthumanity is “better,” and once declared on a pro-extinctionist podcast that

if sacrificing all of humanity were the only way, and a reliable way, to get … god-like things out there — superintelligences who still care about each other, who are still aware of the world and having fun — I would ultimately make that trade-off.

Hence, Yudkowsky has on numerous occasions expressed pro-extinctionist sentiments that are nearly indistinguishable from those embraced by Verdon.

What exactly is different about the worldview of Yudkowsky and Verdon? There is really only one significant difference: Yudkowsky wants ASI to adopt our values, and hence enable people like him to survive into a cosmic utopia. That’s why he thinks we need to temporarily pause ASI capabilities research, so that value-alignment research can catch up. Otherwise, these doomer and accelerationist worldviews are identical twins. Both Yudkowsky and Verdon are pro-ASI and pro-extinctionist. Both want the posthuman successors that we create or become to colonize the universe and climb the Kardashev scale. Both believe that enveloping stars in Dyson swarms and building a multigalactic civilization will lead to utopia. Etc.

Insofar as these two men — both school dropouts, as it happens — dislike each other, it’s the result of the narcissism of small differences. On an ideological map, their views occupy almost exactly the same location.

Conclusion

I wrote this article primarily for skeptics of the TESCREAL acronym. I do not claim to have covered all relevant issues, though I hope this provides a compelling case that the TESCREAL acronym is useful — indeed, indispensable — for understanding what the Silicon Valley worldview is, where it came from, how it developed, and why it’s problematic and dangerous (the last of which I didn’t cover here). Those who object to it likely don’t understand why the acronym was coined or what work it’s supposed to do within critiques of Silicon Valley’s push to build an AI God.

By the way, the people you spoke with at the fictional party are based on real individuals. I tried to present their views as accurately as possible based on what I know about them. Here’s a key in case the hyperlinks weren’t enough:

• Bendisi = Daniel (Bendisi) Kokotajlo
• Niklas = Nick Bostrom
• Bill = Gil Verdon
• Eli = Eliezer Yudkowsky
• Daniel = Daniel Faggella
• Sam = Sam Altman
• Ray = Ray Kurzweil
• Larry = Larry Page
• Mario = Dario Amodei
• Dennis = Demis Hassabis
• Ketamine user = Elon Musk
• Toby = Toby Ord
• Nick = Nick Beckstead
• Holden = Holden Karnofsky
• Peter = Peter Thiel
• Michael = Michael Druggan
• Jeffrey = Jeffrey Ladish

Footnotes:

1 Of note is that Becker wrote this before having come across my work with Gebru. Hence, he independently converged on the same (obvious) conclusions about techno-utopianism being ubiquitous in the Valley.

2 Or, more specifically, both directly emerged out of the transhumanist movement. For example, the cofounder of EA, Toby Ord, coauthored an article defending transhumanism with Nick Bostrom — and took a position at Bostrom’s transhumanist Future of Humanity Institute — before EA was founded.

3 With the exception of Bohan’s book, of course. I include it here because it offers a good overview of the TESCREAL worldview, albeit without using that term.

Discuss

For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done! 

It’s called AI 2040: Plan A.

It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.

It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments. 

As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia. 

Read the scenario at ai-2040.com. You can listen to it on audio, or view it on mobile, but the experience is significantly better on a normal computer.

What’s next for us?

Well, first we are going to respond to comments and otherwise engage with whatever conversation, responses, critiques, etc. that AI 2040: Plan A sparks. Beyond that, we aren’t sure yet. In general our mission is to help make AGI go well, and now we’ve tried out both forecasting and planning. Maybe we’ll get started on another big scenario. On the other hand, these megaprojects take so much time…

Discuss