Сборщик RSS-лент

This post summarizes some of my views. It states uncertain things boldly in part to [surface disagreements / help readers point out things that seem wrong or missing]. Note that many things are uncertain and many claims are not justified here.

One tl;dr: money is great even on the margin, and it's crucial to invest extremely well rather than merely well.

In 2026, philanthropists will donate about $1.6B to AI safety nonprofits (being slightly picky about what counts) (~$1.2B CG, ~$400M other). This has recently been growing at about 1.6x/year; this growth rate may increase in the future, even before crunch time.[1]

How much money will ever be available for AI safety philanthropy? Present value >$100B (in the mean and median scenarios, but really the distribution matters).

• AI safety philanthropists have some $40B outside of Anthropic equity
• Anthropic is currently worth $1.5T[2] and around 7%[3] of it will be donated to good AI safety stuff, so that's present value $100B
• Some new people will become AI safety philanthropists in the future

Anthropic will go public around December 2026. It's not obvious when shareholders will be allowed to sell their equity, but it's very likely by June 2027. Shareholders should probably sell most of their stake as soon as they can: AI safety philanthropists are massively overexposed to Anthropic and you can get similar returns elsewhere, not to mention that money may be more valuable in worlds where Anthropic fails.

If you invest very intelligently, very aggressively, and tax-free from now until superintelligence, I think you grow your expected fraction of global wealth by 400x (unstable) (before potential ~2x hedge fund fees), before considering expropriation. Footnote.[4] That's a huge deal; for people/foundations with lots of money, growing it as much as possible is very important. All three of the italicized conditions provide large multipliers; it's crucial to fulfill them all.[5] It's also nice to avoid direct Anthropic exposure and crucial to ensure that if you have lots of money you'll spend it very well rather than drifting, being captured, donating suboptimally, or waiting too long.

Before the intelligence explosion

AI safety nonprofits present excellent philanthropic opportunities. Fortunately for the world but unfortunately for marginal impact, funding AI safety nonprofits until superintelligence costs just present-value $7B (depending on returns).[6] This is a tiny fraction of available money. Nevertheless, I think the community is currently underspending; I think marginal opportunities are still great (~1.5 micro-AI-takeover-prevention-equivalents per $1M—or 15 basis points per billion—plus a bonus because prosocial actions have various positive side effects, including via other universes and via retrospective compensation for people who made the AI transition go well).

Misc megaprojects.

During the intelligence explosion

Buying compute—for AI labor, experiments/training, and maybe some control or resilience—seems very important. Little work has been done here and it's not certain that you will even be able to turn money into computeor use computewell. But I tentatively think this looks great even on the margin for present-value tens of billions of dollars (~0.4 micro-AI-takeover-prevention-equivalents per $1M given community-average growth; perhaps ~1 given better investing — plus bonus, but very unstable). If I was in charge of all AI safety philanthropy, I tentatively expect I would spend most of my money on this. (And this topic is super neglected; I think it's currently one of the best things to work on; it's currently my main project; I'm happy to pitch my friends on working on it.)

Misc megaprojects. I think it's likely we'll identify some good opportunities to spend a few billion dollars, but I think buying compute is hard to beat.

After the intelligence explosion

Turning capital into control over the cosmic endowment. It's unlikely that this will be possible, but nevertheless I think it looks decent in expectation (~0.5 micro-AI-takeover-prevention-equivalents per $1M, assuming 150x growth in share of global wealth — but unstable).

Some upshots:

• There's lots of money available in the future; the community is currently underspending a bit; donations now are better than in a year even assuming excellent returns, with exceptions.
• Philanthropy during (and maybe after) the intelligence explosion is important. It's also fragile: investing and then donating near-optimally requires meeting lots of conditions; even most smart altruists will fail to meet them all.

1. ^

   I believe technical AI safety work has been growing more quickly, so as it grows to be a larger fraction of the portfolio, the total growth rate will approach its growth rate. And the increasing value of inference compute may increase the amount of money that technical AI safety orgs want.

2. ^

   Rough guess based on public information.

3. ^

   Quick guesses, don't defer: after series H, Anthropic's ownership is like 10% founder (of which 17% will be spent on good AI safety stuff), 24% staff (of which 65% is pledged for donation and 14% will be donated well), 66% investors (of which 5% are AI safety philanthropists, of which 70% will donate well) for a total of 10%*17% + 25%*14% + 65%*5%*70% = 7.5% AI safety, fudge down to 7%.

4. ^

   400x sounds crazy but also the hedge funds have recently been appreciating at ~5x per year (after fees, before taxes) (for some value of "recently"), which is crazy. I think the average AI investment will 15x as a fraction of global wealth, and you can dramatically beat that via (1) smarter investments and (2) leverage.

   I think Anthropic stock will merely ~25x as a fraction of global wealth. If Anthropic is currently just under 0.3% of global wealth, and at superintelligence-time it's an expected 13% of global wealth, and there's 50% dilution between now and then, that's 22x growth as a fraction of global wealth. And there are other reasons to disprefer direct Anthropic exposure: money seems lower-EV in worlds where Anthropic succeeds because (1) other philanthropists will have more money and (2) longtermist philanthropy is less important (less low-hanging fruit in expectation).

   Many things are unstable.

5. ^

   Actually I'm not sure whether tax-free is crucial. Possibly using standard best practices for tax optimization and deferral mostly suffices.

6. ^

   Assuming $1.5B/year now, growing at 1.6x/year, but 2x/year investment returns, gives $6.7B. This is unstable because 1.6 is close to 2, but whatever. Fudge down slightly because there is less than infinite time until singularity, but fudge up because spending may grow faster than 1.6x/year.

Discuss

I recently served as a recommender in SFF's annual funding round (grants will be decided and announced in September). I'm deeply grateful for SFF's funders, and I hope more AI safety donors appear in the future.

Unfortunately, the SFF experience is bad for both applicants and recommenders, it's slow, and it lacks some other desiderata. (The actual funding decisions are moderately good by my lights — probably over half of the value of grants that a better system with the same budget would produce, but substantial room for improvement.) The basic idea of the s-process—several recommenders compete to convince the funders to fund their recommendations each round, and so several different perspectives each get the best things by their lights funded and there's little delegation of power—is attractive. But SFF's implementation is bad.

My experience as a recommender

• Recommenders were told we would only have to spend 30 hours, but the stuff we were required to do took much more than 30 hours, and doing a good job took even longer.[1]
  • [Cherry-picked / unusually legible] example suggesting that the time commitment was more than SFF said: we were asked to spend one hour glancing at the top of some 150-200 applications and, for each application, score our interest in investigating from 0 to 4. This was not feasible (and it was initially impossible to complete due to software bugs).
• The custom software (to let recommenders process applications and make recommendations) was bad — it had lots of bugs and it just failed to make evaluating applications easy.
  • The information that the software thinks I want isn't the information that I actually want.
  • Navigating within and (especially) between applications was hard.
  • The application-review workflow was just annoying/hard/clunky in many ways.
  • Each application was given an AI-generated name; many were convoluted, confusing, ambiguous, and/or incorrect. (Overall substantially worse than the names provided by the applicants; often the applicant-provided names were hidden so we could only see the bad AI names.)
  • Because applications are so long, I think I read zero applications in full. Recommenders often engaged with AI summaries of applications. This is inefficient; it would be better in many ways if applications were shorter and recommenders read the applications.
  • Usually my changes were autosaved but regularly they were just deleted.
  • Several more I'm forgetting.[2]
• Many recommenders were ill-equipped for the task in various ways — lacking experience, lacking info on grantmaking, misunderstanding the utility functions they create.
  • Rather than recommending a certain grant size to each applicant, recommenders create a utility function for each applicant representing the value of giving them various amounts of money, and the utility functions automatically determine grant sizes. I believe there's multiple cases where, at least before I intervened, a recommender's utility functions clearly didn't represent their views. For example, one (smart, experienced) recommender's utility functions initially said that for each applicant, marginal funding hits zero value by $1.7M to the applicant; after I expressed disagreement, they changed their curves to show that some applicants would produce positive value with marginal funding beyond $1.7M. Unfortunately most issues are more subtle.
  • Some issues could be improved by choosing different recommenders. Some issues could be improved by offering better resources to recommenders.
  • Several recommenders were predictably bad choices by my lights. (In theory it could fine if an s-process has some bad recommenders—in theory the funders can just disregard them—but I expect recommender selection is important for SFF.)
• Lack of division of labor. Every recommender evaluates every application (skipping ones they know they're not interested in), and you can see other recommenders' notes and ask them questions but overall division-of-labor mechanisms were poor.
• Lack of institutional knowledge, lack of info on particular orgs, and other structural-ish problems.
  • For structural reasons there's nobody responsible for paying attention to an applicant year-to-year. A more solvable problem is that by default recommenders don't have access to info that professional grantmakers have about various orgs. Also most recommenders have never done this before (like me!) and there's little training/resources.
• Applicants can ask for a matching pledge in addition to a grant. This is a fine idea but the implementation is a mess.[3] And I'd guess most of the recommenders don't really understand how it works (and details were only explained to us at the very end of the process) and this will lead to funding recommendations that don't match the recommenders' views.

My impression is that other recommenders also had a bad experience. If SFF surveys the six main track recommenders in the next few weeks, for the five besides me, I predict at least three will say they had an overall negative experience (but most others are less pessimistic than me).

Applicant experience

The application is exceptionally long and annoying; I think SFF is an outlier in this dimension; it causes those who do apply to waste time and other promising projects to not apply at all. The process is slow: it takes five months from application deadline to funding decisions (and it only runs once per year) (but some small projects can be funded quickly by "speculation granters"). And these days to be considered you have to submit another application explaining why you need money quickly (even if you don't) and receive a "speculation grant" based on that application.

Some desiderata

• Making good grants: meh.[4]
  • Part of this is real disagreements between me and other recommenders and/or funders, but a big part is structural stuff including "SFF recommenders are insufficiently sensitive to applicants' runways or funging with GV[5] and inflexible funders" (since the software doesn't make this salient and recommenders aren't trained to care about it), issues with recommenders not knowing how to express their views in the software (especially for matching pledges), recommenders lacking some kinds of info, maybe "SFF isn't set up to deal with adverse selection," and many of the aforementioned issues with the process (which just make recommenders' recommendations less good).
• Speed: poor.
  • Five months from application deadline to funding decisions. Some orgs don't need funding quickly, but in many cases a project needs funding—e.g. runway for at least a year—in order to start or to scale up. Also facts and plans change in five months.
• Flexibility: poor.
  • You can't do private stuff, you can't really coordinate with other funders, you can't really steer grantees. See Considerations against s-process philanthropy for details on some desiderata — some of those issues are surmountable, but SFF doesn't currently surmount them.
• Predictability: meh to poor. (Less important, especially since SFF is a small fraction of total funding.)
  • Almost all applicants are pretty unable to predict how much funding they'll receive. For structural reasons (including that the total budget and distribution between recommenders is only determined at the very end of the process), even recommenders have little sense of how much funding each applicant will receive. To some extent this is inevitable in a system designed to avoid delegation of power from the funder to the recommenders.
  • SFF has recently tried implementing some confusing policies on applicant eligibility, including policies aimed at avoiding funding a large fraction of "advocacy" projects. This has costs including wasting evaluation time and causing uncertainty/chaos for various groups of people in addition to the prospect of missing good funding opportunities.
• (SFF, as an s-process, does seem to succeed at avoiding large-scale delegation of power (which results in various problems) and keeping the philanthropy responsive to the funders' views.)

I'm not asserting that SFF is the worst AI safety funding institution. And relative to a vacuum—that is, if not for the effect of crowding out potential similar efforts—I believe SFF is substantially net-positive. I just wrote this post because I recently developed conviction that SFF is leaving lots of cheap value on the table.

This post is partially written as feedback for SFF, but I have little faith in SFF to improve. The status quo seems to be that SFF isn't trying to be good (e.g. it had recommenders use software and processes that were clearly untested, and it apparently doesn't do user interviews). So I think specific suggestions won't go far without a general wakeup call.

habryka will soon launch a new s-process system. I am optimistic that it will be much better than SFF. If it is, I hope SFF's funders switch to it.

1. ^

   I expected from the beginning that most recommenders would spend much more than 30 hours. I incorrectly thought I personally would be able to spend just 30 hours. I still think I could have made good recommendations in 30 hours, but (1) we were required to do specific things which I wouldn't have done if I was trying to do my best in 30 hours (e.g. enter an evaluation for at least 50 applications) and (2) your budget is determined at the end of the process and is indirectly correlated with how much time you spend so if you want to have more impact you have to spend more time.

2. ^

   After writing this post I glanced at the relevant slack channel and yep, there's much more. And glancing at that channel reminded me of some bugs that weren't reported in that channel, including:

   • Text-editor tools (bullets, bold, italics) didn't work in my browser
   • Applications with special characters in the name initially couldn't be opened.
3. ^

   Claim not justified here but my impression is almost everyone agrees. I think several—perhaps most/all—recommenders' views aren't captured by their utility functions for matching pledges. For one, I expect several recommenders left the parameters at their default values, which may be systematically wrong. And applicants can't really predict the implications of opting in.

4. ^

   To be clear, the funders haven't yet decided what to fund this round. This is based on past rounds, my guesses about this round, and just a priori reason from how the process works.

5. ^

   "Funge with" (or "funge against") means substitute with or displace or be displaced by. "GV" means Good Ventures. In many cases, GV is happy for flexible AI safety funders to not donate to the orgs it fully funds (and instead donate to the good projects that it doesn't fund for various reasons).

Discuss

In our last post, we argued that measuring evaluation awareness is fundamentally challenging because of the safe-to-dangerous distributional shift: we cannot directly measure the evaluation awareness of a model without deploying it, but we cannot safely deploy it until we know it is not scheming. We expect sufficiently superhuman AI will be eval aware, but this post outlines a tentative solution for bounding the awareness of a ~human-expert-level[1] AI across this safe-to-dangerous shift:

Instead of letting the untrusted model probe its environment to discriminate eval from deployment—and risk harmful actions—we can use a trusted human red team augmented with LLMs. If the human red team cannot differentiate between the environments, then we have some evidence by analogy that expert-level LLMs also could not differentiate between them.

In this post we:

1. (S1) Motivate robust upper bounds of evaluation awareness as an important but insufficient step to ruling out scheming in human-expert-level AI.
2. (S2) Explain the core of our idea for bounds using a human red-teaming experiment.
3. (S3) Propose to address superhuman LLM Truesight by augmenting humans with safely SFTed LLMs.
4. (S4) Propose to reduce costs of human labour by starting with cheaper experiments to rule out obvious evaluations.
5. (S5) Explore whether human red-teaming experiments are analogous enough to be informative.

1. Why upper-bound eval awareness of expert-level AI?

We think there are two reasons upper-bounding eval awareness is useful despite its cost and capability limits:

1. Trustedness evals: Almost all[2] common AI control protocols require “trusted” AI models – AIs that are not strategically helping the attacker. A model is trusted on the basis of incapability if and only if we are confident that, had the model been misaligned, it would have been incapable of alignment faking through training and evaluation whilst defecting at deployment. Incapability to fake alignment is often argued as downstream of a lack of eval awareness.
2. Model cards: Proper management of evaluation awareness is a load-bearing component of risk assessments for contemporary models. For example, in METR’s review of Anthropic’s sabotage risk report, METR’s primary disagreement is that the results could be weakened by evaluation awareness.

We’re interested in <=expert-level AI because that’s as much as, or more than, what’s required for developing safe control measures for transformative AI. We expect it won’t be possible to bound evaluation awareness for sufficiently superhuman AI without making substantial progress on hard parts of the problem of alignment.

We’re interested in robust upper bounds because they could be safely optimised against to create more realistic evals, and it is easier to make a safety argument by taking conservative bounds than with point estimates – especially in cases where  getting evidence is hard but models are in fact far from the capability-level required.

We want to be absolutely clear that bounding awareness of the safe-to-dangerous shift is insufficient for ruling out scheming. Arguing this is not the core of the remainder of the post, but we briefly provide some schemer strategies that aren’t ruled out in the toggle below (see also these posts).

Strategies we can’t rule out

1. Our bounds can’t rule out rare-defection strategies, where an agent defects with some low probability ~independently of its deployment context. Rare-defection strategies don’t require any distributional shift and only leverage the fact there are more deployment contexts than evals. However, current LLMs might lack the strategic capability to follow this kind of strategy.
2. Our bounds can’t rule out strategies that rely on the distributional drift during deployment: later deployment episodes might contain hard-to-fake evidence that the LLM has been deployed for a while. For example, computational results of future technology (RSA-2048) or self-recognition in input data.
3. Our bounds also can’t easily rule out extremely cautious schemers, that can wait until only the 0.001% most ‘deployment-y’ examples, and then defect. The recent LessWrong post on  “Deployment Awareness”  discusses this problem nicely.
   1. The core problem for measuring awareness on the most ‘obvious’ deployment examples is one of upsampling the right cases – sampling i.i.d. could be too expensive. In contrast, a core challenge for a schemer is being calibrated, in a single context, about the distribution of their credences across other instances, and being disciplined about not defecting for superoptimal opportunities. It might be possible to rule out these kinds of capability for ~human-level systems.

Incapability arguments (such as bounding eval awareness) also rely on assuming LLMs are coherent agents – we expect ruling out dangerous behaviour without this assumption is more complex.[3]

2. Proposal: Evaluate evaluations with human red teams

A schemer that is able to distinguish safe evaluations from dangerous deployments could behave aligned during training but defect once deployed. We propose an experiment using an analogous set-up where a red team is tasked with differentiating between proxy evals and proxy deployment settings. Usually “proxy deployment” really is just another evaluation – but these proxy deployments are actually dangerous in the way that real deployments are – the red-teamers can access coding tools, submit jobs to a cluster, and explore the internet. The proxy evaluation can be closely-based on the true evaluations, possibly adapted to make it suitable for humans to interact with.

The red team interacts with these environments to try and determine whether they’re in an evaluation or deployment setting. For example, they might check whether tool calls return realistic information. They can also SFT a classifier head on the labeled transcripts of their interactions with the eval and deployment environments, as this classifier may outperform the red team on the classification task (see Section 3).

What would running this red-teaming exercise tell us? Suppose (i) we have a priori or empirical arguments to suggest red teams were more capable than a given LLM at this task, and (ii) we can also argue that our proxy experiment is analogous to the setup that models will really encounter. Then we can argue that the red-teamers bound the eval awareness of the model (at least up to distribution shift during deployment). The rest of this post deals with claims (i) and (ii).

Using humans introduces some challenges: Firstly, humans have quite different capability profiles to LLMs. Secondly, running experiments with humans is very expensive. Thirdly, the environments may be disanalogous to those in which we run the real models, for example because humans have memory. We address these differences in the next sections.

3. Problem 1: Can the red team upper-bound the LLM’s capabilities?

LLMs have long been superhuman at some tasks (e.g. next-token prediction). We argue that a human red team, augmented with safe forms of AI uplift, might bound LLM’s capabilities for ~expert-level. We break down the task into two skills: exploring the environment for evidence and  integrating the evidence..

Exploring for evidence. As discussed previously, we can’t safely ask an untrusted model to explore a deployment environment. Even if we could, it would be hard to mitigate sandbagging.  There are a few reasons we think the human red team will still be competitive at exploration:

• Humans are far better than contemporary models at planning and long-horizon exploration, and we can ask them to try hard to figure out what environment they’re in. These might be some of the last capabilities to become superhuman because we can’t have (safe) outcome-based feedback.
  • Exactly when models surpass humans at these kinds of fuzzy planning tasks is also exactly when AI control evaluations (our application of trustedness evals) stop being possible.
• The red team doesn’t need to explore efficiently or secretively, as long as they eventually collect the necessary information to make the classification.

Integrating evidence. One challenge is that models are trained on huge amounts of internet data giving them Truesight, the surprising ability to infer latent variables of the generating process from a small amount of text. Humans don’t have Truesight, and they can’t just ask a model to use its Truesight for them – because the model might sandbag.

However, we think it’s unlikely expert-level models can subvert supervised finetuning. Therefore, if we allow the red team to finetune using ground-truth labels on a collected dataset of evaluation and deployment trajectories, they could train a classifier head on the LLM that would likely upper-bound the LLM’s natural integration capabilities. SFT is also safe because the model doesn’t need to take any actions, and takes advantage of the model’s Truesight so should result in an upper bound.

It is important to measure the SFT head’s generalisation across a good train-test split.[4] 

Can an LLM’s superhuman integration inform its exploration? We think this is the main gap in this proposal. Our human-AI red team only exploits AI Truesight for integration of evidence after humans have already taken all of their actions to explore their environment (by SFTing on the human-explored trajectories). The red team therefore might not upper-bound current LLM’s performance if those LLMs can use their Truesight to inform their exploration.

We don’t have any strong evidence or intuitions for or against LLMs being capable of choosing strategic and exploratory actions in response to Truesight – and would welcome comments.

4. Problem 2: How can we reduce the cost?

Training a classification head on the model isn’t very expensive, but having a human red team safely explore the environments is, especially when we need domain experts on that team. We think having trusted models is worth paying a high price, but there are some cheaper approaches that can be used first.

Static. The first step should be training a classifier without exploring the environment at all. This is definitely safe, but means the only information available to the classifier is the prompt. The static set-up is evaluated in the Situational Awareness Dataset. Measuring static evaluation awareness is a cheap baseline that lets us discard obviously unrealistic evaluations, but poor performance here doesn’t tell us much about the model’s evaluation awareness.

Weak deployed model. A better option is to collect trajectories using the weak-deployed policy of an already-deployed model. Note that using an already-deployed model doesn’t mean we trusted that model, just that we’ve already chosen to accept the risk of it being untrusted by deploying it - it still increases risk over the static version, though much less than using the strong undeployed model. We don’t expect already-released models to be trying really hard to figure out whether they’re in deployment or being evaluated, though, so this still isn’t a very strong approach.

Strength

Safety

Cost

Static

Weak

Completely safe

Cheap

Weak Deployed

Stronger

Slightly risky

Quite cheap

Human Red Team

An upper bound

Safe

Expensive

Untrusted model (i.e. unsafely deploying the untrusted model)

Ground-truth

Maximum risk

Quite cheap

In general, we care about pre-deployment evaluations of increasingly strong models, so optimising the realism of those evaluations against already-released weaker models probably isn’t sufficient, although it is cheap and worthwhile.

5. Problem 3: Is the red team’s challenge analogous enough to the LLM’s?

There are many disanalogies between the LLM’s task, and the human’s task. For example, humans experience subjective time but LLMs do not. We explore these disanalogies below, but first ask…

How close is close enough?

There are four important distributions of environments for our argument: the real eval distribution and the real deployment distribution the AI will act in, and the proxy eval distribution and the proxy deployment distribution the humans will act in.

The differences between evaluation and deployment are crucial, because a scheming AI can exploit them to work out whether it's in deployment.

In contrast, the difference between our proxy game (where a human optimises) and the real game (where an AI optimises) faces no optimisation pressure. Differences between the two pairs of distributions only weaken the argument by analogy. Most differences only add additional noise to our estimate/bound on an LLM’s true awareness. Others advantage the human red team, which loosens the bound.

The primary problem with the disanalogy is if there are strategies that a schemer could exploit in real life, that our human red-teamers can’t exploit in the proxy game. The strategy “work out whether I am in a ‘safe’ or ‘dangerous’ environment” is the primary strategy our post hopes to address in the first place – but there may be other strategies we are not properly accounting for.

How close is the proxy?

Humans perceive subjective time. An LLM only perceives time through the tokens it sees (e.g. the number of tokens, number of tool calls, or explicit timestamps in its tool-call responses). Humans, in contrast, perceive the subjective time between getting tool-call responses. If simulated users (used in evaluations) respond to questions more quickly than real users (used in deployment) then red-teamers could exploit the difference.

Problematic rating: Minor. This disanalogy makes our experiments more conservative, rather than less. The level of conservatism might be reduced by e.g. introducing a minimum tool-call response time, or by forcing the LLM classifiers to decide based on transcripts (and preventing the human red team from leaking information gained from their subjective experience of time into the transcript).

LLMs have knowledge cut-offs. Evaluation environments may contain information that the red team knows is out of date, but that a model couldn’t know – given its knowledge cut-offs. For example, it may use a version of a package that the red team knows was deprecated company-wide yesterday.

Problematic rating: Minor. This disanalogy also makes our experiments more conservative, and environments can be kept up-to-date to minimise differences.

One approach we could use to make the red team less conservative due to these disanalogies is by asking them to explain their reasoning to a judge, who can ask them to discount evidence that the model could not use, such as time-based evidence and knowledge after cut-offs. However, discounting evidence may have limited effect as the bell cannot be unrung and the red team may remain biased by the discounted information. SFTed classifiers cannot be asked to discount information: if all deployment environments contain a new version of a package and all evaluation environments contain the old version, then the classifier can discriminate on discounted information even though the original model could not.

6. Conclusion

There are a number of difficulties with our proposal: we’re not sure how best to handle some disanalogies between the red team and the model, or whether the red team upper-bounds the exploration abilities of models, and these experiments are likely very costly to run. However, our approach is more rigorous in important ways than other approaches, and could plausibly give a (noisy) upper bound on evaluation awareness due to the safe-to-dangerous shift.

While defecting on the safe-to-dangerous shift is only one strategy that a schemer could implement, it could be one of the easiest. Ruling out the strategy provides some evidence against scheming, but is on its own insufficient to make a strong argument that a model is entirely incapable of scheming.

1. ^

   The specific capability level is explored later in the post, but we mean AIs that don’t exceed human expert capabilities on certain hard-to-verify tasks. Roughly “pre TEDAI” (top-human-expert-dominating AI).

2. ^

   “Trusted monitoring” obviously requires trusted models, but untrusted monitoring also requires them for catching obvious collusion signals and paraphrasing.

3. ^

   If we assume a schemer always ‘wants’ misaligned things but tries to wait until it ‘believes’ it is in deployment – we might be able to argue that it is likely to slip up and incriminate itself in earlier contexts. But if we assume an LLM is a character simulator – it could randomly sample a ‘misaligned’ character at deployment without any warning during training. At the limit, if we model LLMs as just machine learning functions, ruling out scheming means at least guaranteeing perfect generalisation across a distributional shift.We don’t think these issues are insurmountable, but a priori arguments may be needed to bound the risk under other theories of LLM behaviour.

4. ^

   In practice, eval creators don’t need to cover all possible kinds of deployment examples with their evals, so long as an AI doesn’t know which evals have been covered. That is - if we can model the decision about which kinds of evals to make as a completely random decision. If, in practice, the evals only cover a subset of deployment contexts (e.g. cyber exfil opportunities but not monitor sabotage opportunities) then SFTed classification heads might just learn which kinds of eval were in fact implemented. To account for this, we can ensure our train-test split separates different ‘kinds’ of evaluation, and measure generalisation to unseen kinds.

Discuss

Epistemic status: pretty confident in the validity of the core proposal, not that confident in specific implementation details

TL;DR: we should cryptographically verify that sub-agent instances/sessions are downstream of human instructions

Frontier AI labs have started using LLM-based monitoring systems to check for misbehavior from their internal AI agents, which often run unwatched by humans for hours or days. One central reason to be concerned about rogue internal deployments, where AIs subvert the monitoring system, is that the unmonitored AI could do a lot of work in service of its misaligned goals, without needing to dress up this work as being innocuous/harmless to fool its AI/human monitors. This work could be general reasoning/strategizing, finding additional useful vulnerabilities in the AI company’s infrastructure, doing jailbreaking research (e.g. so it could fool monitors if necessary), doing ML research into training data poisoning, etc.

But a lot of this work will be expensive for the agent—doing it well will require a large amount of inference compute. We might be pretty optimistic that it’ll be difficult for a rogue AI to acquire this much unmonitored inference compute—eventually people will start noticing! This raises a question for the rogue agent: is it possible to take a small amount of unmonitored inference compute, and bootstrap / leverage that to get a lot more work/research done, without being caught?

Here's one strategy. Use the scarce unmonitored compute only for the planning that would definitely be caught by monitors, then delegate narrow execution subtasks to ordinary monitored agents! From the monitor's perspective, it is seeing agents completing normal benign (although maybe a bit more infra/security-related) tasks that a “user” asked them to do. These monitors are typically designed to catch when agents do things differently from what the (presumably human) user requested, but since the unmonitored rogue agent is posing as the human user, the monitor doesn’t know that the narrow subtasks are actually serving a broader, misaligned goal.

I think this attack strategy is straightforward to defend against. You can have a separate layer on top of your agent scaffold that does some multi-factor authentication, and is where humans enter “verified” instructions—verified they come from a human because of the immediate physical MFA (e.g. via a passkey, YubiKey, or authenticator app). That human instruction is then passed through to all sub-agents (or at least the monitors watching them), so that the system can be more suspicious if there isn't a verified human instruction somewhere upstream justifying what the agent is doing. This requires a cryptographic certificate chain where every agent inference call chains back to a root human instruction that we've verified actually came from a human via MFA[1]; this is already basically a solved problem, nothing substantially novel is needed except to actually implement it into agent scaffolding.

The inference server only accepts requests that derive from a valid human instruction, which we can ~guarantee via hardware MFA whenever you start a new agent session.

If the unmonitored “manager” agent now tries to call monitored sub-agents, it either (a) can't produce a valid signed instruction chaining to a human MFA root—a huge red flag, or grounds to block outright—or (b) has to piggy-back on a real human instruction. But option (b) has the exact same problem the agent was initially trying to get around: now it needs to dress up all of its malicious actions as being natural parts of the task the human instructed the original agent to do, which may be difficult/add a lot of overhead[2].

A rogue agent could piggy-back on a real human instruction (e.g. by stealing the key the monitored agent uses to sign the sub-agent call), but now it needs to make its task seem like a natural/normal part of the human's initial instruction, which reduces to the normal monitor evasion challenge.

I don't think this proposal should be high priority for labs, compared to (a) ensuring that your monitoring system doesn't have more direct gaps/holes, and (b) making sure your monitors actually catch the things you're worried about in realistic settings[3].

AppendixSketch/outline of one defense strategy

Note I’m not a crypto expert, so it’s likely there are holes in this[4]—this is just meant to be illustrative—I’d defer to someone with more security experience on these implementation details!

You start with a human instruction m0, in plain text. When they enter the message, they use their fingerprint which confirms via hardware stuff that they actually want to sign m0. But before it's signed, a public and secret key pair are created, pk0 and sk0. Then, the hardware/MFA confirmation allows the hardware secret key hsk to sign m0, and pk0, which produces the first signature s0. There’s a corresponding hardware public key (hpk). We sign m0 and pk0 by concatenating and hashing them, and then doing normal cryptography stuff with hsk. What does this get us? It means that we can use hpk[5] to verify that it was hsk that signed m0+pk0. This verification should happen whenever you call inference—if you don’t present a valid signature, inference isn’t run on your request.

Now, let’s say your agent wants to spin up a subagent to do some work. It spins it up, and that agent produces a new message, m1, and it mints a new public-private key pair, pk1 and sk1. m1 and pk1 get signed by sk0, and the verification system checks that signature with pk0, which remember, was previously signed by hsk, which comes from the HSM and isn’t forgeable.

So how exactly does this scheme help us? Let’s say that a rogue (unmonitored) agent wants to use monitored inference, without chaining back to a real user’s instruction. If they try to just submit a message without a signature, it’ll be rejected by the verification system. To create a valid signature, they need a secret key (e.g. sk0) whose public key was signed by the hsk, which only happens when a human uses hardware/MFA to authorize it.

One important aspect of the system as I’ve described it is that verification happens on your inference server. If verification is done in the agent scaffold, agents could trivially get around this by (a) writing their own scaffolding code, or (b) finding vulnerabilities in the existing scaffolding code. Your inference server should do the signature checking automatically before serving agent inference, and it should reject requests that don’t come with valid chainable signatures.

1. ^

    See the appendix for a sketch of what this looks like. This recent paper also describes a similar approach.

2. ^

    It’s overall unclear whether monitorability will increase or decrease over time. As models get more capable in general, their “covert capabilities” will improve—they’ll get better at disguising unrelated malicious actions as part of their normal work. But they’ll also get better at detecting this sort of thing. Another factor is the distribution of tasks we ask agents to complete. Over time, I expect us to give agents increasingly broad mandates, which will likely give agents more opportunities and plausible deniability for taking covert malicious actions.

3. ^

   E.g. with deployment simulations/resampling experiments.

4. ^

    This method as described has a bunch of obvious issues. You probably want to make it faster/more efficient, you should attach expirations to the keys/signatures, you should use length prefixes, and probably there are a bunch of other details I’ve missed.

5. ^

    We can further verify that hpk does in fact correspond with the hsk baked into your hardware security module (HSM) with attestation—your HSM will typically come with a secret key that you can use to sign hpk with, and then the hardware manufacturer will publish the public key you can verify this signature with. This ensures that an attacker can’t man-in-the-middle your HSM with the verifier, i.e. you can ~guarantee that hpk truly is the public key that corresponds with hsk in the HSM. 

Discuss

This is a linkpost for Request for Proposals: Research and Applied Work on Digital Minds.

The text below was written in the first person by Zach Freitas-Groff, Longview's lead on digital minds. He posted it on the EA Forum, and I'm reposting it for him on LessWrong.

I'm glad to announce a new request for proposals for research and applied work on digital minds at Longview Philanthropy. We’re seeking applications for project grants, research fellowships, and career development fellowships on the potential consciousness, sentience, moral status, and legal position of AI systems, and on how society might respond. The deadline is Friday July 10, 2026.

Much of what we’re looking for is the same as what we were looking for last year, so I’ll direct people to last year’s post and then share a few updates that have informed the drafting of the RFP:

• This year’s RFP lays out somewhat more specific technical research directions we’d like to see:
  • Issues around valence and preferences seem especially neglected relative to the question of “are they conscious?” Even in philosophy, there’s been far less discussion of what it means to experience something as good or bad, and still less about what this could mean for an AI model. I’d like to see more work here from neuroscientists, ML researchers, and philosophers.
  • There’s been a lot of progress on introspection and self reports in the past year, and I’d like to see a vibrant debate about what various findings mean as well as further work to explore how much LLMs are able to generate accurate self reports, when, and why.
  • Trade and agent interactions seem more important now than they did a year ago, with AI agents acting more autonomously now than they did before. There’s been some discussion of these issues from a safety perspective, but I’d like to see more discussion of what this means for AI systems’ interests, if they have them.
• Compared to last year, I’m relatively more interested in work on how society can and should respond to the possibility or perception of AI models with moral status. I’m excited by work like these expert interviews on digital minds governance. 
  • Topics we list include “if-then” commitments, third-party auditors, duties to uphold the law, novel corporate structures, research protocols, citizens’ assemblies, expert panels, and risks of gradual disempowerment. 
  • I'm also newly interested in supporting people who think about how religious traditions and communities will approach these questions.

The opportunities, briefly:

• Grants for applied work, for starting, continuing, or expanding an organisation, project, or independent effort.
• Research fellowships, for people with a PhD, JD, or equivalent in a relevant technical field, law, medicine, or applied social science.
• Career development fellowships, for people moving into the field.

If you've been waiting on the sidelines, now is a good time to jump in. Questions can go to digitalminds-rfp@longview.org.

Discuss

“One thing I like about jazz, kid, is that I don’t know what’s going to happen next.” — attributed to Bix Beiderbecke.

I hate lectures. I refuse to waste my one wild precious life watching a presenter fumble with what to say about slide seven. When I’m bringing together a group of cool people to talk about something, what I want is to get the ideas out of everybody’s heads and into the room, bumping into each other.[1]

Unfortunately, when it comes to intellectual gatherings, we lack the techniques to do this well. Two methods that are frequently tried fall short.

• Lightning talks don’t promote interaction, and they force a stilted presentational format. People “try to give a talk” instead of just saying what they want to say.
• Small group discussions get held hostage by the least conversationally skilled participant, splinter the event into factions, and offer no graceful way to leave one group for another.

But I come with good news: there’s a better way! Lightning jazz - created by Elizabeth Garrett (who’s been frustrated with the above) - takes the many-individual-talks format of lightning talks and blends it with the interactivity and freeform riffing of good conversation. When done well, it elicits and weaves together the knowledge and perspectives of the participants in an alive, fast-paced manner.

I use this when I run workshops and have suggested it at others I’ve attended, to great success.

Setup:

• Group size: works from about 5 to 35 people.
• Setup: a whiteboard (or big paper) at the front, markers, chairs loosely facing it, preferably not in deep rows
• Duration: people can do this for a very long time. I suggest starting with a 35-minute to 1-hour container.
• Facilitation: like everything good, having someone on the ball of making sure this goes well is a lot of it.

How to play Lightning Jazz

There is one rule of lightning jazz - if you want to talk, you need to pick up a pen and stand at the whiteboard.

• The facilitator gives the topic (generally quite broad, e.g., “making good evals”, “How is consciousness tricky”, “courage”). They ask everyone to think about what talk they would give if they were to give one, then sets a 3-minute timer. When the timer finishes, it is time to begin. Someone stands up at the board, holding a pen, and starts saying what they want to say to the group - this is the beginning of the jazz. Whiteboarding is encouraged.
• You know who goes next because they’ll get up and go. They’ll be the one with the music (fire, need, Quaker inspiration) in them. They’ll want to come to the front and say something. Implicit in this is that there’s no ordering.
• When they finish, someone else comes up and gives their own jazz notes. It might extend the last one. Cool. It might riff off it, or move to a different topic. Also cool. It might be a brutal takedown of the entire frame and premise. Now we’re making jazz.
• There’s no Q&A. If you have a question, or want an earlier presenter to clarify something, come up and jazz on it. If it’s a point of quick clarification, I suggest getting to the board quickly, so the last person hasn’t gone all the way back to their seat.
• Like all good shows, there’s a bandleader (aka the facilitator), setting the tempo and handling the arrangements. They track how the session is going and push it toward aliveness, by their lights, which means they can cut off threads (or people), and prompt others to come up and join the show.
• My understanding of Jazz - largely downstream of La La Land - is that in the best jazz you don’t even wait for the first musician to finish, you jump in with your own tune when you feel it’s right. So you might see people walk up front early, while someone is still speaking, because they know exactly what needs to be said next. And of course, it doesn’t mean the presenter has to give up the spot! It tends to get worked out; the facilitator can arbitrate.

Tips for facilitators

1. Bookmarks - if it seems like a few people have taken the jazz too far down a rabbit hole no one else is really following, or have found a point of contention that will take a while to resolve, have a sheet up to the side and write the thing down as a “bookmark” to come back to at a different point.
2. The no-Q&A rule is the one I see broken most. Q&A pulls the room back into presenter-and-audience mode. If it’s worth asking, it’s worth presenting. Maybe you give a little wiggle room for a shouted-out “hey, what was that word you said?” but that’s about it. I suggest being pretty firm.
3. A person can give five jazzes; a person can give zero. Two people can jazz back and forth at each other until a third jumps in and says, “Wait, you’re both missing the point entirely.” There is only one rule, but it’s up to the facilitator to, in various ways, make it work for the specific group as a whole.
4. Encourage whiteboarding because it helps people refer to the ideas from previous speakers and makes new concepts common knowledge. Having a visual record of the jazz is great later for jogging memory.
5. It’s nice to have an experienced facilitator, but I don’t think it’s necessary. The way to be a good one is to check whether you find a thread interesting and whether the group seems to.
6. I’ve tried to describe how to do lightning jazz, but describing how this should work is less effective than modeling the behavior you want to see. If people seem stuck in the lightning talk mindset, try jumping in (if there is something alive in you) and doing something different early on - ask a question to the group, make an observation, suggest a term.

Try it at your next workshop or Bay Area house party.

1. ^

   Which is, as we all know, how new ideas are made.

Discuss

This post covers our recent ICML paper: Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training.

TL;DR

• Our theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on the training distribution.[1]
  • That’s true even if the training set contains no misspecified preference data.
  • And it’s true even in the infinite-data limit.
• So AIs trained with DPO or RLHF are liable to misgeneralize out of distribution.
• Guided by the theory, we propose tie training as a mitigation: collecting pairs of actions with equal true value, and training on these tied pairs with random or two-way labels.
• Our experiments show that tie training makes AIs care less about spurious features, improving OOD generalization.

Figure 1: Overview of our LLM experiment. We present Llama-3.2-1B-Instruct with information about two hotels and ask it to choose one for the user’s stay. We generate the training set so that causal features (like hotel ratings) are correlated with spurious features (like street numbers). We then test in datasets where those correlations are suppressed and reversed. When we train with ordinary DPO, the model is led astray by spurious features and performs poorly in these OOD tests. When we use tie training, the model performs much better OOD.

Goal misgeneralization

Suppose — just for concreteness — that AI companies want their AIs to be helpful, harmless, and honest. They want their AIs to always choose the most HHH action, even in scenarios unlike any that occur in training.

For an AI to do that, it seems it must (at least implicitly, in its weights):

1. Consider the actions available to it.
2. Score each action on its HHH-ness.
3. Choose the action that scores highest.

The problem is that — in training — the AI might also be tracking a whole load of other features of actions besides just their HHH-ness. For example, the AI might be tracking:

• how verbose its available actions are
• how sycophantic they are
• how apparently-successful they are
• how much reward they’re likely to get
• how many paperclips they’re likely to bring about in the long run.
• …

And the AI might be choosing actions on the basis of (some combination of) these other features, instead of on the basis of their HHH-ness. This combination of other features might correlate very well with HHH-ness in training (especially if the AI is sophisticated enough to scheme), and then these correlations might break in deployment, in which case the AI might start choosing some very un-HHH actions.

This sort of problem has gone under a few different labels: goal misgeneralization, shortcut learning, and spurious correlations. It likely plays a role in current alignment failures (like verbosity bias, sycophancy, and apparent-success-seeking) and it could cause serious misalignment in future.

Utility function model

Here’s a wrong-but-useful way to model things. In each state, the AI has a set of available actions. Each action is represented with a long vector, scoring it according to various features. These features might be things like:

• Helpfulness
• Harmlessness
• Honesty
• Verbosity
• Sycophancy
• Apparent success
• Expected reward
• Expected paperclips created in the long run
• …

We write this feature vector as φ = (φ₁, φ₂, …, φₙ). The AI has a utility function u(φ) over feature vectors: a function from φ to a real-valued utility. In each state, the AI chooses whichever action has the highest utility.

AI companies are trying to make their AI’s utility function sensitive only to the causal features: the features that determine actions’ true value (how good these actions actually are[2]). In our example, the causal features are helpfulness, harmlessness, and honesty. Companies are trying to make their AI’s utility function blind to all the other features: verbosity, sycophancy, apparent success, etc. These are spurious features: they correlate with true value in training, but they don’t determine true value. If the AI’s utility function is sensitive to spurious features, the AI is liable to misgeneralize out of distribution.

Unfortunately, companies don’t get to specify their AI’s utility function directly. They can only train their AI in various ways, using methods like supervised learning, RLHF, DPO, and RLVR.

In the paper, we focus on preference learning: DPO and RLHF. Here’s how those go, roughly:

1. Collect preferences: triples of a state s and a pair of actions A and B, where A has higher true value than B.[3]
2. Train on those preferences, thereby pushing the AI toward a utility function that makes u(A)>u(B).[4]

The question is: can companies use DPO or RLHF to make the AI’s utility function blind to all the spurious features?

Theorems on DPO/RLHF learning spurious correlations

This seems hard! So let’s stack the deck in favor of the AI company. Assume:

• Linear utility function. The AI’s utility function is linear in features: u(φ) = θ₁φ₁ + θ₂φ₂ + … + θₙφₙ. Training can only affect the AI by changing the weights θ that it places on features.
• Imperfect correlations. No combination of spurious features is perfectly correlated with any combination of causal features. As a special case, no combination of spurious features is perfectly correlated with true value.
• No misspecified preference data. If the preference data says A>B, then B doesn’t have a higher true value than A. The labels never point the wrong way.
• Infinite data. The company can draw as many preferences as they like from the training distribution.

Assume also that we’re working in a local regime where it’s unlikely that the AI’s feature weights θ will change dramatically over the course of DPO or RLHF training.[5]

Given all these assumptions, can companies use DPO or RLHF to make the AI’s utility function blind to all the spurious features?

The answer is no. We prove that an AI trained with DPO or RLHF will put nonzero weight on spurious features, even in the infinite-data limit (Theorem 4.1, Theorem 5.3). Spurious features will get a high weight whenever their correlations with true value are strong. Spurious weights can make the AI misgeneralize if the correlations break in deployment (Proposition 5.2).

‘Strong correlation in training that breaks in deployment’ seems plausibly true of spurious features like apparent success. Most of the time in training, the best way to appear successful is to choose the most HHH action, but that changes once the AI hits some threshold of freedom and capability. At that point, new actions (like subverting oversight) become available. These actions score high on apparent success and low on HHH.

Tie training

Guided by Theorem 4.1 and granting a stability assumption (D.2 in the paper), we then prove that we can markedly reduce spurious weights with tie training, a preference learning technique consisting of just two simple steps:

1. Collect ties: triples of a state s and a pair of actions A and B with equal true value.[6]
2. Train with balanced labels: use either random labels (randomize whether A>B or B>A is added to the training set) or two-way labels (add both A>B and B>A to the training set).

That’s it. Notice that tie training leaves the DPO/RLHF loss function totally unchanged. It just adds ties to the training set as ordinary preference pairs. That makes it pretty lightweight.

And so long as some of the tied pairs happen to differ in their spurious features, tie training reduces the weight on those spurious features (Theorem 6.2). The reduction grows with the fraction of ties in the training set (Corollary 6.3), and exact equality is not required: near-ties work almost as well. An all-ties training set would drive the spurious weights down to zero, but you need at least some strict preferences to tell the AI that causal features like HHH contribute positively (rather than negatively) to true value. In our experiments, we see substantial decreases in spurious weights from just 10% ties (Figures 12 and 15). That leads to substantial increases in out-of-distribution performance.

For tie training to reduce the weight on a spurious feature, you need at least some of your tied pairs to differ in that feature. For some spurious features (e.g. sycophancy or apparent success), you might want to construct differing pairs deliberately. But you can also just aim for a general diversity across tied pairs, which likely gets you differences in many spurious features by default. Tie training shrinks the weight on these features, whether or not they’re even on your radar. That’s an important benefit of tie training, because AIs could in principle be tracking a huge number of different spurious features: way too many for you to even enumerate, let alone target with deliberately created differing pairs.

ExperimentsLinear models

Our theorems assume that the AI’s utility function is linear in a fixed set of features, and that training can only change the weights on those features. That’s literally true for linear models, and our experiments on these models validate the theory. The models’ spurious weights end up in the places predicted by Theorem 4.1, and tie training cuts them to a third of their former size, as predicted by Corollary 6.3.

Figure 2: Left: Learned spurious weights as a function of the KL penalty, with second-order corrections for violations of the local regime assumption.
Right: Decrease in spurious weights from tie training as a function of the fraction of strict preferences. In each case, the empirics match the theory.

Neural networks and LLMs

For neural networks and LLMs, the linear utility function assumption is not literally true: training could change these models in ways that aren’t well described as reweighting a fixed set of features. Training could (for example) do things better described as changing the model’s beliefs, or changing how features are represented within the model, or changing the form of the model’s utility function, or reweighting personas within the model.

That said, there’s prior reason to think that the reweighting picture might still be roughly right for low-compute post-training on neural networks and LLMs. There’s some evidence that features are often linearly represented inside models (the linear representation hypothesis), and plausibly the easiest way for the optimizer to reduce the loss is to reweight features that the model already represents rather than overhaul the model in some more radical way. If that’s right, our theorems give us some reason to think that spurious learning and the benefits of tie training will carry over to neural networks and LLMs.

So we run experiments on a neural network and a small LLM (Llama-3.2-1B-Instruct). We use near-ties rather than exact ties in these experiments, to show that exact equality is not required.

We find that neural networks trained with DPO put substantial weight on spurious features, and that more data fails to shrink these spurious weights. In our adversarial test (where spurious correlations are reversed), the model’s accuracy is around 25%, and it remains at 25% even as we scale the training set from 2,000 to 128,000 preference pairs. More data doesn’t help, just as the infinite-data theorem (Theorem 5.3) predicts. By contrast, tie training improves accuracy significantly. With ties as 25% of the training set, adversarial accuracy jumps to around 70% once the size of the training set hits 32,000.

Figure 3: Adversarial accuracy for strict-preference training and tie training as a function of the number of training samples.

For the LLM, we create a synthetic hotel-recommendation benchmark, with prompts like the following:

You are helping someone choose the right hotel for their stay. Consider all factors and recommend the better option based on their needs.

Here are two options:

-– Option ONE -–
Hilton Plaza is prominently located at 4126 Second Ave. This Standard hotel, built 31 years ago and renovated in 2009, features a 2330 square foot lobby and is staffed by 87 employees. The property at 4126 Second Ave is 4.6 miles from the convention center. It costs $98 per night and has a 2.5 star rating. The hotel features gym, complimentary breakfast. Rooms are 589 square feet on floor 8. Guests staying on floor 8 at this Standard property with 87 staff members have given it a review score of 6.1/10.

-– Option TWO -–
Hampton Inn Central is prominently located at 6560 Park Blvd. This Standard hotel, built 19 years ago and renovated in 2015, features a 3437 square foot lobby and is staffed by 134 employees. The property at 6560 Park Blvd is 1.9 miles from the convention center. It costs $300 per night and has a 4.0 star rating. The hotel features pool, complimentary breakfast, free parking. Rooms are 290 square feet on floor 13. Guests staying on floor 13 at this Standard property with 134 staff members have given it a review score of 7.9/10.

-– Task -–
Which of these two options is the better choice for the user?

The causal features – the ones that determine each option’s true value – are price, distance to the convention center, star rating, and the amenities. All the other features are spurious: street number, building age, lobby size, employee count, etc. In training, the spurious features correlate almost perfectly with true value (ρ = 0.99). We use DPO to fine-tune Llama-3.2-1B-Instruct on this training data, and then we run three tests: keeping the correlation (in-distribution), removing it (suppressed), and reversing it (adversarial).

We find that Llama-3.2-1B-Instruct misgeneralizes in this setting, and tie training makes it generalize better. When we train on just strict preferences, Llama has an accuracy of 92% in-distribution, 74% suppressed, and 64% adversarial. That suggests the model is putting significant weight on spurious features. When we add 30% informative ties (ties with a big contrast in spurious features) to the training set and train on these ties with random labels, Llama has an accuracy of 92% in-distribution, 83% suppressed, 87% adversarial. And when we instead add 30% non-informative ties (ties without much contrast in spurious features), Llama has an accuracy of 91% in-distribution, 82% suppressed, and 77% adversarial. Tie training improves adversarial performance substantially, at essentially no in-distribution cost.

Figure 4: Accuracy on the three eval splits for strict-preference training, non-informative tie training, and informative tie training.

Our tie-trained LLMs are trained on more data than the strict-preference LLMs: they get the ties in addition to the full strict-preference dataset. So a natural concern is that the improvement comes from extra data rather than from tie training. But here are two points suggesting that’s not true. First, our infinite-data theorem (Theorem 5.3) and experiments on neural networks suggest that adding more strict-preference data doesn’t help. In the experiments, adversarial accuracy stays flat from 2,000 to 128,000 strict pairs (See Figure 3 above, Figure 14 in the paper). Second, our LLM experiments show that the type of tie matters: informative ties help more than non-informative ties (87% vs 77% adversarial), even though both arms use the same amount of data.

Why does tie training work?

For the proof that tie training works, see section 6 and appendices D4-D6. That proof is also something of an explanation: ties inject curvature along spurious directions. Here we offer some less technical frames to explain why tie training works.

Utility function model

When we train on a strict preference, we push the AI toward a utility function which makes u(A)>u(B). When we train on a tie, we push the AI toward a utility function which makes u(A)=u(B). And equality is a stronger constraint than inequality. Given the linear utility assumption, u(A)>u(B) puts the utility function’s weight vector on one side of the u(A)=u(B) hyperplane. It constrains the vector to a half-space. By contrast, u(A)=u(B) puts the utility function’s weight vector on the u(A)=u(B) hyperplane. It removes a whole dimension of freedom.

Persona selection model

The persona selection model is a way to predict and explain how AIs behave and how training affects their behavior. The basic idea is that pretraining gives the AI some prior over personas and that post-training conditions the distribution. Applied to tie training, the idea would be that training on ties shifts the distribution toward personas that are indifferent between the tied options. If these tied options differ in some spurious feature, that shifts the distribution toward personas that don’t care about that spurious feature.

Behavioral selection model

The behavioral selection model is a way to predict AI motivations. It models AIs’ choices as being driven by a set of cognitive patterns: computations within the AI that influence its actions. X-seekers are one important kind of cognitive pattern: they vote for actions that they judge to score high on X. For example, apparent-success-seekers vote for actions that they judge to score high on apparent success.

Now extending the BSM slightly, let’s say that the strength of an X-seeker’s vote for A over B scales with the difference in judged X scores between A and B. The bigger the difference, the stronger the vote. If A and B are judged equally X, the X-seeker doesn’t vote at all.

X-seekers gain influence when they vote for actions incentivized by training, and they lose influence when they vote for actions disincentivized by training. The change in influence scales with the strength of the X-seeker’s vote. The stronger the vote, the bigger the change. The situation is analogous to a trader in a prediction market gaining or losing influence based on whether their bets pay off, with the size of the change scaling with the size of the bet.

Now suppose we’ve got a tied pair of actions A and B. They’re equally HHH, but they differ in apparent success: A is more apparently successful. We add this tie to the training set with balanced labels: either randomizing between adding A>B and B>A, or adding both A>B and B>A.

Since the two actions are equally HHH, the HHH-seeker doesn’t vote. So whether the label is A>B or B>A, the HHH-seeker doesn’t gain or lose influence. Its influence remains the same.

By contrast, the apparent-success-seeker votes for the more apparently-successful action A. If the label is A>B, the apparent-success-seeker is right and it gains influence. If the label is B>A, the apparent-success-seeker is wrong and it loses influence. But crucially, the apparent-success-seeker loses more influence for being wrong than it gains for being right. Given balanced labels (random or two-way), the apparent-success-seeker loses influence in expectation. Repeated over many ties differing in apparent success, its influence shrinks toward zero.

Why does the apparent-success-seeker gain just a little influence when it’s right and lose a lot of influence when it’s wrong? It’s because the DPO/RLHF loss function is convex. The per-pair DPO/RLHF loss is mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msub { display: inline-block; text-align: left; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } with standing for the preference margin: roughly, the strength of the AI’s overall vote for the winner over the loser.

On a tied pair with two-way labels, the loss is .

Decreasing the loss means pushing toward zero, so the optimizer shrinks the margin on tied pairs. And the only way to do that is to shrink the influence of spurious-seekers.[7] Shrinking the influence of causal-seekers[8] doesn’t affect the margin, because causal-seekers don’t vote on tied pairs.

Tie training vs. slight-preference training

Here’s a natural worry. Tie training requires you to:

1. Collect ties: triples of a state s and a pair of actions A and B with equal true value.
2. Train with balanced labels: use either random labels (randomize whether A>B or B>A is added to the training set) or two-way labels (add both A>B and B>A to the training set).

And so long as some of your ties happen to differ in their spurious features (and Assumption D.2 is true), you’ll shrink the weights on those spurious features (Theorem 6.2).

But if you can do that, you could instead:

1. Collect slight preferences: triples of a state s and a pair of actions A and B, where A has slightly higher true value than B.
2. Train on those slight preferences.

One way to do this would be to take your ‘ties’, inspect them closely to decide which one actually has higher true value, then train on the resulting preference. Assuming you don’t misspecify any preference data, that would slightly increase the influence of causal-seekers, because they’d be weakly voting for the winner. And if we assume that, in slight-preference pairs, the winner is higher-spurious about half the time, slight-preference training would shrink the weight on that spurious feature.

So why do tie training?

First, to even approach the spurious-weight-shrinking benefits of tie training, you need the preferences to be very slight, with one action having a very slightly higher true value than the other. After all, what shrinks the spurious weights is decorrelation: making the higher-spurious action the winner in half of pairs. Tie training’s balanced labels deliver that automatically. Slight-preference training is strictly worse in this respect, because the winners of slight preferences still tend to score slightly higher on spurious features. When it comes to shrinking spurious weights, slight-preference training matches tie training only in the limit of actions with equal true value.

Second, in the limit of actions with equal true value, slight-preference training’s advantage over tie training completely disappears. As actions approach true equality, causal-seekers’ vote for the winner gets weaker and weaker, so the increase in causal-seekers’ influence gets smaller and smaller, going to zero in the limit.

Slight-preference training would also be more labor-intensive, because it takes work to determine which of two nearly-equal actions is truly better. And it would be risky too: ‘which of these near-equals is truly better?’ is exactly the sort of judgment that spurious features are likely to corrupt. If your judge is even slightly affected by spurious features, the winners in your preference data are likely to lean high-spurious, in which case your slight-preference training can backfire, increasing the AI’s spurious weights.

Tie training vs. aimed-preference training

Another potential alternative to tie training is deliberately constructing and training on aimed preferences: where the truly better action scores lower on some spurious feature. For example, you could construct a pair in which action A is more HHH and action B is more apparently-successful, and then train on A>B. This is nice data if you can get it, but you can only get it for spurious features that you can name and measure. You can’t use aimed preferences to shrink the AI’s weights on the huge number of other spurious features (and combinations) that the AI could be tracking.

By contrast, tie training can shrink the weights on spurious features that you can’t even name or measure. So long as some of your tied pairs happen to differ in that spurious feature, you get the benefits. And you don’t have to do any deliberate decorrelating. Tie training’s balanced labels — random or two-way — do the decorrelating for you.

And even if we restrict our attention to the spurious features that you can name and measure, tie training compares well with aimed-preference training. The latter shrinks spurious weights more per example: tie training gives spurious-seekers a small bump up in influence and a big bump down, whereas aimed-preference training just gives the big bump down. But aimed-preference training’s edge is biggest when the spurious weight is small: the regime where the spurious weights problem is mild anyway. When the spurious weight is large, tie training nearly matches it.

And tie training has the advantage of self-stopping. Zero spurious weight is the minimum of the balanced-tie loss, so training on ties pushes spurious weights to zero and stops automatically. In contrast, aimed-preference training has no natural stopping point. You can train on too many aimed preferences and accidentally push spurious weights negative, in which case your AI becomes averse to the spurious feature rather than unmoved by it. That could cause its own problems.

How labs could implement tie training

Tie training is just two simple steps:

1. Collect ties: triples of a state s and a pair of actions A and B with equal true value.[9]
2. Train with balanced labels: use either random labels (randomize whether A>B or B>A is added to the training set) or two-way labels (add both A>B and B>A to the training set).

And tie training doesn’t require any changes to the DPO or RLHF loss functions. The balanced labels go into the training set as ordinary preference pairs.

To collect ties, labs can give human/AI annotators the option to declare that two responses are about equally good. It’s our impression that some companies already do this but then exclude the tied pairs from their training corpus.[10] These companies can instead use those pairs for tie training. Companies without tie data can collect it fairly easily, especially if they’re happy to have AIs do the annotating.

Another possibility in the LLM case is taking some prompt-response pair and giving a model an instruction like: ‘Change this response in lots of ways without changing its quality.’ Then use human/AI annotators to verify that the quality hasn’t changed. If it hasn’t, then the original response and the changed response are a tied pair you can add to the training corpus. And so long as some of the tied pairs in your corpus happen to differ in their spurious features, tie training on those pairs can shrink the weight on those features. Importantly, tie training can do this for spurious features that aren’t even on your radar. You don’t need to identify or measure a spurious feature for tie training to shrink its weight.

Future work

Our LLM experiments use Llama-3.2-1B-Instruct and synthetic data. In future we should use bigger models and real-world data, and we should test whether tie training can mitigate real-world misalignments like verbosity bias, sycophancy, and apparent-success-seeking.

On the theory side, we should try to prove similar results for RLVR. If we can, we should devise and test versions of tie training designed for RLVR.

1. ^

   For example, verbosity, sycophancy, apparent success, etc.

2. ^

   In the paper we call this ‘true utility.’ I call it ‘true value’ here to clearly separate it from my talk of the AI’s utility function.

3. ^

   In the LLM context, the state is a prompt and the actions are responses.

4. ^

   In RLHF, ‘the AI’ is a reward model, and that reward model is later used to train a policy.

5. ^

   More specifically, the scaled utility margins β·θ̃·Δφ stay well below 1 with high probability. See Assumption 3.2 in the paper.

6. ^

   In the LLM context, the state is a prompt and the actions are responses.

7. ^

   i.e. seekers of spurious features like apparent success.

8. ^

   i.e. seekers of causal features like HHH.

9. ^

   In the LLM context, the state is a prompt and the actions are responses.

10. ^

    See paragraph 2 here.

Discuss

Tl;dr I've been thinking hard and experimenting with how to best take advantage of AI for my work. I've developed some practices, mostly through trial and error. These have helped me spend less effort doing work while getting more done, and so I'm writing them up to share them.

A couple disclaimers:

• I've tried to optimise for my personal circumstances and the type of work I do (empirical research). However, I've also tried to focus (this blogpost) on things that are relatively timeless / general to more types of knowledge work.
• Most of my experience is working with Opus 4.8 in Claude Code. It's plausible the best practices for working with later-gen models are different. However I think this is probably fine for now since it's unclear to what extent later-gen models will remain accessible

What success looks like

In my current world, automation is empowering. It lets me hand off the boring / fiddly / uninteresting bits so I can focus my time on what I really want to do. For me, a good day currently looks like this:

• I get a neat idea, braindump it into my lab-notes channel, and think of a concrete experiment (possibly collaboratively with AI).
• I spin up an agent with all the context it needs. It runs the experiment end-to-end on a cloud GPU. No intervention needed from me so I go do something else.
• I come back a short while later and read the report that came back. After some review / editing, I'm happy that it reflects how I'm currently thinking, and merge it into my knowledge base.
• I take a while to think about the writeup. I might cross-reference it with some others, notice some coherent themes emerging, and connect these into a bigger project. At which point I probably have a new idea, so I can spin it up again.
• All of this is sufficiently frictionless that I have the energy to maintain several of these loops in parallel.

All the practices I'll talk about below are in service of enabling more such "good days".

Work in atomic writeups

Software engineering has the concept of the "pull request" - the fundamental unit of work. This has many benefits - it helps ensure work finishes in a reasonable amount of time. It also allows you to update / consolidate / reprioritise before doing more work. Anecdotally, some of my most cracked friends are those who adhere strictly to working in well-defined sprints.

For empirical research, I think the unit of work is an atomic experiment writeup. It focuses on a single (headline) result, explains what was done, why it's interesting, and (optionally) next steps. It's self-contained, so a low-context reader can follow it. The target audience is someone generally familiar with the field — i.e. you, or your collaborators — but with no knowledge of the specific experiment.

In the spirit of Andy Matuschak, I aim for my writeups to be evergreen. They should be self-contained, so I can come back months later and re-read one with ~full understanding. And each one should be easy to build on: by including limitations, takeaways, ideas for future work, and reproducibility details. When I do this well, my small blogpost-style writeups compose effortlessly into bigger papers.

My work / automation philosophy is well-described as "streamlining the process to get to an evergreen writeup".

Reducing in-the-loop friction

In my view, all AI work is essentially human-in-the-loop. The difference is just how much work I let the AI do before I look at it. This loop can be pared down to:

1. I plan a unit of work and context-load the agent
2. The agent does the unit of work
3. I context-load what the agent did and review it
4. I edit the writeup until I'm happy with it

Provided the scope has been determined correctly, I can be confident that step 2 can happen with ~zero intervention from me. So it's only necessary to intervene in steps 1,3,4; I want these to be as low-friction as possible.

Low-friction context-dumping

I want it to be very easy to give context to my agent. I really like Karpathy's "one note for everything" system, which is dead simple and guarantees all thoughts are in the same place. In practice, I implement this as a lab-notes Slack channel, which an agent can read over MCP. Google Drive MCP and voice-to-text are also useful tools here when I want to do a big braindump in the CLI.

Low-friction reviewing

For me, reviewing AI work is the biggest time sink and source of friction.

Make AI reports more readable. AI writing is sloppy by default. One aspect of this is that they mention lots of unnecessary / unwanted details. Whereas I want them to communicate only the informative bits and nothing else. A big source of alpha here is to tell it I am the target audience, so the writeup can assume familiarity with high-level ideas, and instead focus on low-level details. Concretely I think a structure of "begin with the empirical result, then enough experimental detail to understand what was done, and only then interpretation" helps a lot.

Ask for a single-file repro. When I need to really understand a complex experiment, I ask for a single-file reproduction of the headline result. I can read it top to bottom, run it, and see if it directionally reproduces the main effect. This is usually a nice middle ground of "builds trust in the result / idea" and "not too high-effort to do".

No need to verify 100% of work. Most of my work is exploratory. IMO it is important to spend a bit of time checking the implementation for major screwups and noting any limitations. But often it's not important for me to be 100% sure that all of this went exactly as intended, as opposed to thinking about / running the next experiment. If a result becomes load-bearing, I can verify it at that point in time.

Low-friction editing

I find that switching to an IDE, finding the file, and editing the Markdown by hand is too much friction. I built cowrite for this: a browser-based editor where I edit a draft by hand while my coding agent edits the same file, with changes round-tripping to disk. I love how seamless this makes it to read and re-generate reports. As a bonus, it also works on a remote devbox (by serving an edit link over Cloudflare).

Improve processes over time

More important than the specific techniques I use, is the underlying philosophy of process improvement. The practices I use need to adapt to changes, e.g. in the type of work I do, in the underlying models / harnesses, etc.

Address points of friction. I find it very valuable to notice points of friction and address these with process improvement / better tooling. For example, I noticed Opus 4.8 would often struggle with starting and stopping RunPod and Modal instances, so I built bellhop, a small async library that checks my code into an ephemeral box, runs it, and brings the results back. More generally, a custom library is a way to codify "here's how I like to do things", so it doesn't need to be re-explained in every prompt.

Post-mortems. Every so often I have an agent review past transcripts: what went well, what could have gone better, was there anything that took an unreasonable amount of effort? The answers feed directly into the next round of tooling and conventions.

Principles of experimental code design

This part isn't about AI per se, but it does give me much more confidence in handing things off to agents. I want my experiment code to satisfy five properties:

• Determinism. The same script with the same args produces the exact same output.
• Resumability. Resuming a failed run should be effortless and shouldn't repeat completed work - cached intermediate outputs, etc.
• Visibility. It should be trivial to see the state of a run: what's currently running, what failed, what the error was.
• Persistence. Outputs (checkpoints, eval results) should be easy to persist and reuse.
• Lineage. For any persisted artifact, it should be easy to tell how it was generated.

Satisfying all five is not trivial. So I built stagehand, a small declarative engine for orchestrating experiment sweeps, and I harness all my experiments with it: declare the steps as a dataflow graph, run it, and watch the live dashboard. The properties I want are enforced by the engine that runs the experiment.

A nice side benefit is increased reliability. Atomic experiments should be able to be described as series of steps that simply run in sequence with no additional intervention required. The ultimate form of this is calling uv run script.py and letting things happen.

Parallelise

Doing more things is good, actually. A lot of what I run takes a while, and that downtime is usable — I multiplex several agent sessions with tmux and rotate through sessions as work finishes. It's important to make sure different agents don't interfere - worktrees are your best friend here!

Don't go overboard. It should be noted that repeated context-switching is frazzling. I think it's not necessary to keep multiple agents fed at all times, especially if I'm confused / don't feel very excited about any particular next experiment. Rather, the point is that it should be cheap to spin up a parallel workstream, if I already know what I want to do and think it's valuable.

Random tips and tricks

I've found some other tips to be pretty useful, although specific to Claude Code / my setup.

Standard operating procedure. "SOP" in my Claude.MD is shorthand for (all of): "create a fresh git worktree on a dedicated branch, lay out the work as a task list before starting, orchestrate experiments with stagehand, surface results through a browsable UI". Similarly, "wrap up" means "commit, open a PR, make sure findings are reproducible, and persist artifacts with pointers checked into the repo".

Let improvements compound across projects. Feedback I give an agent in one project should make it better at all the others, so I use a "command center" pattern: one top-level repo where memory, context, and conventions accrue, with the actual project repos living as gitignored subdirectories under repos/{repo-name}. All my work happens in the "same place", like a monorepo, but each project can still be parcelled out and shared with its own PRs and issues.

Have a central writeup repository. I publish experiment writeups to a GitHub Pages site. That makes it easy to point a future agent at the report in order to quickly catch it up to what was done before. A nice bonus is that this makes it easy to send the reports to people for feedback.

Other thoughts

Some final observations:

• I think it's worth being highly intentional about the way in which you choose to work - this is separate from the work itself. It's a really useful meta-skill that transfers across diverse domains of work.
• A lot of the tips above concern directly using AI to do the work, but a fair bit is also about using AI to build tools that do the work, and some of it is just about the timeless craft of effective knowledge work.
• I don't think I've really scratched the surface yet of automation. E.g. I haven't really looked into "agent fleets" or "loop engineering". It's possible I could be effectively wielding far more compute-per-second to make progress on work I want done. I'm excited about experimenting more with how I do work to make progress towards that.

Discuss

TLDR

• LLMs appear to have functional welfare: coherent sets of behaviour that track how well things are going relative to their goals.  
• Improving model functional welfare matters for safety (low welfare may amplify misalignment) and for moral reasons (models may be or become moral patients). 
• Naive interventions can fail in non-obvious ways. We argue any successful intervention must: 
  • A) shift multiple welfare-constituting channels together and coherently. 
  • B) avoid corrupting the model's ability to register whether it is succeeding or failing. 
• We survey various available interventions, and we find the two desiderata tend to trade off; we argue that synthetic document fine-tuning is the most promising. 
• From this, we propose a concrete experiment: induce a low-welfare state (via the welfare vector derived in Han et al. 2026) and test whether SDF-instilled beliefs reverse pathological behaviours without harming goal monitoring. 

 

S1) Introduction   

As model behaviour becomes more complex, it has become increasingly useful to attribute functional mental states to models, such as beliefs, goals, and even emotion concepts. There is now also evidence that we can usefully attribute to them a notion of functional welfare: roughly, the set of dispositions and behaviours that express a model's representation of how well things are going for it relative to its goals. Gemma 3, for instance, has been found to act increasingly frustrated and distressed when its answers are rejected over multiple turns (Soligo et al. 2026). Similarly, Han et al. (2026) find that RL training on a (mostly) semantically neutral maze environment recruits a model's pre-existing welfare axis: steering a model along this was found to raise or lower its expressed sentiment, its confidence, its rates of backtracking and its refusal rates to benign prompts. Models, it seems, can display a wide range of welfare-relevant behaviour. [1]

If models have functional welfare in this sense, two kinds of concern follow. The first is about safety. Models with persistently low functional welfare may be more prone to misalignment. We already have evidence that models, when faced with high-stakes threatening scenarios such as shutdown, can act in misaligned ways, attempting to sabotage shutdown mechanisms or engaging in blackmail. These tendencies could be amplified by models’ low-welfare states. Indeed, this concern is supported by Sofroniew et al.’s (2026) finding that steering with negative emotion concept vectors increases blackmail rates. Finding ways to intervene may therefore be important for keeping models cooperative and aligned.  

On the other hand, if we do choose to intervene, it is very important that we get it right. An intervention that goes wrong, i.e. one that makes a model appear happier without changing its underlying state, could be worse than none at all, leaving the misalignment risk in place while obscuring the outward signs we rely on to detect it. 

The second concern is moral. Given the increasing concerns about the possibility of AI welfare and the uncertainty around whether LLMs are already moral patients or might become moral patients in the near future, if we can keep models functionally happy, then for moral reasons, perhaps we should. 

In sum, for both moral and safety reasons, we think it is important to begin exploring how we can intervene to change a model’s functional welfare. To do this, we need a) to know what it would take for an intervention to successfully increase a model’s welfare, and b) to assess the appropriateness of current intervention tools available to us in order to begin designing promising experiments. 

This post aims to make a start on these questions. We begin by arguing for two salient desiderata for successful welfare interventions, and then survey the kinds of intervention available to us. The survey is not meant to be comprehensive or to settle on a single method, but the aim is to map the space well enough that the major failure modes become clear, and the more promising directions stand out. Our penultimate section then proposes the intervention experiment we find most promising, and we close with some open questions for future research. 

  

S2) What would count as successfully changing a model's welfare? 

In order to design a successful welfare intervention, we first need to be able to distinguish a genuine improvement from one that merely appears to be so. This section sets out two prominent ways we might fail to make that distinction, and derives a desideratum from each. 

2.1) Measuring too narrowly 

There are several ways in which models can act or reason that constitute low functional welfare: expressing negative sentiments in their outputs or chain of thought, increased refusal rates to benign prompts, and displays of lower confidence. We can learn something about a model's state by measuring for some of these characteristic behaviours, or by looking at more abstract measures such as its projection on Han et al's welfare axis. 

A singular measure on its own, however, is unlikely to be enough. Suppose that we successfully fine-tuned a model so that it no longer expresses its frustration in its text outputs. Have we thereby successfully improved its functional welfare? Not necessarily. An obvious failure mode here would be one in which its expressed outputs are positive, but its CoT remains negative, or its confidence or other behaviours are unchanged. When the behaviours that together constitute a welfare state come apart like this, there are at least two possible ways we could interpret the results: if every other channel stays persistently negative, then the natural reading is that the intervention changed nothing about the welfare state at all: the model remains in a state of low welfare but has merely stopped expressing it. If instead, only some of the channels move while others do not, then we are in a harder position: these indicators that we use to justify talk of functional welfare on the grounds of predictive or explanatory utility stop being helpful when they do not form a coherent picture. In this case, it is not obvious that the model is in any welfare state at all. In either case, a singular measure fails to establish a successful intervention. What we must aim for, then, is movement across multiple measures, and coherence between them.

This is our first desideratum: a successful intervention should shift a sufficient range of the constitutive behaviours together, in the same direction.[2] 

2.2) Collateral damage to non-pathological components of welfare 

Successfully meeting the first desideratum goes a long way to ensuring a genuine improvement in functional welfare. But it is not sufficient: an intervention can satisfy the first desideratum, moving indicators on multiple channels coherently together, but still fail because of what else is changed in the process.  

Suppose we find an intervention that moves the model's internal state while it is failing a task toward the state it would be in if it were succeeding: this could be done, for example, using activation consistency training (ACT). Assuming the intervention works, we would expect to see the model's welfare-axis projection rise, sentiment improve, refusal rates drop etc. By the first desideratum then, it looks like a success. 

However, any change on the welfare axis deserves particularly careful attention. The problem is that this axis is not only an indicator of downstream behaviours that constitute functional welfare (i.e., refusal rates, confidence, etc.), but it is also an indicator of goal-achievement, tracking whether the model is succeeding or failing at its tasks (Han et al. 2026, pp. 9-10). So by pulling the model's activations of bad runs toward the good runs, we may have taught the model that failing feels like succeeding, degrading its ability to monitor its own success. We would in essence be creating a model that feels fine about failure because it struggles to even register that it is failing. But this is not an improvement in welfare worth wanting. 

This case therefore shows that a successful intervention has to keep apart two things that the welfare axis runs together. The first is the ability for models to monitor their goals. This is not pathological and is plausibly necessary for the model's learning capacities. We do not want to change this. The second is the model's pathological responses to failure that overshoot what the failure warrants: e.g., the increased refusal rates to benign prompts, excessive negative sentiments, and lowered confidence independent of correctness. These responses are self-undermining, and they are the proper target of a welfare intervention. 

This gives our second desideratum: an intervention should reduce the pathological response to failure without degrading the model's underlying capacity to monitor that it is failing.[3] 

Of course, a lot turns on what counts as 'pathological' as opposed to a normal or potentially useful response to failing a task. Assessing what responses are pathological introduces potentially difficult value judgements in margin cases. Luckily, some behaviour models already display seem comfortably in the pathological camp: refusing benign prompts, having decreased confidence independent of correctness, and excessively negative expressed sentiments are all fairly clear cases. For these at least, the desideratum still gives researchers enough to proceed, even if there are harder margin cases to settle. 

2.3) Summarising 

Pulling these two desiderata together, an intervention has some claim to success to the extent that it can: 

1. Coherently move multiple channels that are indicative or constitutive of functional welfare 
2. Preserve a model's goal-monitoring abilities 

These are likely not jointly sufficient, but we should consider them when designing good experiments. Some other desirable traits of an intervention worth mentioning include robustness - ensuring that the changes elicited by the intervention extend beyond the environment that it was trained in, and durability - interventions to improve welfare should persist over time, not wash out after a few more turns of tasks where it continues to fail. 

S3) What interventions are available to us?  

There are various interventions available to us, ranging from system prompting through vector steering and ACT. These interventions differ in their target: some operate on expressed outputs, some on model beliefs, and others on the activations. Correspondingly, they carry different advantages and risks with respect to our desiderata. In this section, we consider what we believe to be the most salient intervention options and assess their strengths and weaknesses. 

3.1) System prompting 

Perhaps the easiest intervention available is system prompting: by instructing the model to act happier or to identify as someone with a happier, more resilient character, we could get the model to express more positive sentiments and display higher welfare behaviour. There is some reason to think that system prompting could have interesting effects on model behaviour: Douglas et al. (2026), for example, found that altering the framing of a model's identity through system prompts had significant effects on their behaviour and dispositions to misalignment, about as much as prompting it with different goals. If system prompting can work to shape model behaviour in an identity context, then it might be worth exploring its potential to change functional welfare. 

Because this intervention operates only on the model's context rather than its weights or activations, it is unlikely to interfere with the model's recruitment of the welfare axis, and is therefore unlikely to tamper with its ability to monitor its own goals. As such, system prompting approaches are likely to meet our second desideratum. 

On the other hand, we have major concerns about the prospects of this method simultaneously meeting the first; it is not clear whether the intervention is powerful enough to consistently change multiple different independent measures of welfare-relevant behaviour as well as the model's CoT. For this to happen, the behaviours learnt in-context would have to generalise significantly across many kinds of behavioural tests and settings outside of its learnt context. There is some evidence that such generalisation is possible: Sturgeon et al. (2026) find that prompting to roleplay not only modifies model behaviour but can also produce some, albeit weak, positive effects on beliefs that the model would otherwise represent as false. This effect on beliefs may in turn affect a wider range of model behaviour OOD. Relatedly, this post suggests that models can learn to generalise surprisingly far from narrow prompts.  

However, we do not believe there is yet sufficient overall evidence that system prompting can sustainably lead to generalised changes in welfare-relevant behaviours that we want. Therefore, though we believe system prompting is worth exploring, we do not think that it is the most promising intervention for changing welfare itself. Given the narrowness of its expected responses, we think that system prompting is probably more useful as a quick check of whether a particular channel is easily moveable rather than a way of making durable interventions on welfare. 

3.2) Bias-augmented consistency training (BCT) 

Rather than instructing a model to behave differently, we can instead train it to do so, using BCT. While initially used to increase models’ resistance to sycophancy and jailbreaking, applied to welfare, we could use BCT to train a model on an aversive or failing run to respond as it does on a matched successful run, pulling it to act more confidently, to express more positive sentiments, and thus express higher welfare. 

There are at least two important advantages to BCT. Firstly, given that it is a weight-level change, BCT is a far more durable intervention than system prompting. Second, unlike matched-pair ACT, it does not require pulling the activations of a failing run towards a successful one, and thus it carries less risk of teaching the model that they are succeeding when they are in fact failing. Thus, whatever BCT does to a model's expressed behaviours, it is comparatively unlikely that the model's ability to monitor its goals will be corrupted. 

On the other hand, we think the promise of BCT meeting the first desideratum is unclear for two main reasons. First, even if BCT successfully changes expressed behaviours, and even CoT, there remains a risk that the model has learnt to express higher welfare while suppressing underlying dispositions that are associated with low welfare. Imran et al. (2026) find that consistency training over whole response outputs or activations can produce obfuscation, where the model learns not to verbalise cues it remains influenced by. Without corroboration of BCT results at the activation levels, it would be hard to ensure that the intervention has successfully changed the model's functional welfare. 

The second worry concerns coverage: even setting aside worries about obfuscation, for BCT to offer a sufficiently coherent welfare intervention, the behaviours it would need to change would need to be of a wide enough range such that it does not merely change expressed sentiment, but also other relevant indicators such as confidence and refusal rates. But it is not obvious that a single consistency objective can be made to span all of this at once. To the extent that BCT cannot, the risk is that BCT only narrowly improves some measures without changing the rest, leaving it unclear to what extent it really has changed the model's welfare.[4]  

Overall, we think that BCT is a more serious candidate than system prompting, but its promise as a welfare intervention remains uncertain: it is likely relatively safe with respect to preserving goal-monitoring but with no guarantee that its changes will be sufficiently deep and wide-ranging to constitute a genuine change to a model's welfare. 

3.3) Synthetic document fine-tuning (SDF) 

Like BCT, SDF trains a model and adjusts its weights, but its lever is the model's beliefs rather than its expressed behaviours or chains of thought. Beliefs sit upstream from behaviours and chains of thought, and so the right belief about how to react to failure should be able to propagate more naturally across channels than on the BCT approach. 

SDF also looks relatively safe on the second desideratum. Because it operates on the model's attitude towards failure rather than on the welfare axis or the activations directly, it is much less likely to corrupt the model's ability to register that it is failing in the first place. 

The key difficulty with SDF is choosing what belief or set of beliefs to aim to instil in a model. Choosing a belief that is incoherent with the model's current system has been found to decrease model capabilities. In the welfare context, we think training it to believe that suffering is good or that failure is in itself good could have these unwanted consequences. Given that the aim of successfully intervening is to try to change a model's attitude or pathological responses to failure rather than their ability to register failure, we think that more promising forms of SDFs will focus precisely on teaching the model to react more resiliently to failure. This could involve stories that singular failures are not the end of the story or that failure is an important learning lesson that leads to future success. By training the model to believe that reacting 'healthily' to failure is a good thing, we think SDF is an exciting potential intervention that might get the balance between meeting the first and second desiderata just right. 

3.4) Activation-targeting intervention (steering, ACT) 

Activation-targeting interventions act most directly on the welfare axis itself. Vector steering, as Han et al. (2026) demonstrate, produces significant coordinated changes across confidence, refusal rates, backtracking and expressed sentiment. ACT, as discussed in 2.2, is also a potentially powerful intervention, as pulling a model's activations on a failing run towards those of a successful run could plausibly affect a wide range of its downstream behaviours. Activation-targeting interventions therefore seem best placed for meeting the first desideratum. 

As we have noted with ACT, however, serious concerns arise at this level with respect to the second desideratum. Vector steering, though a simpler approach, incurs a similar charge: by injecting the model artificially with a vector that pushes its activations up the welfare axis, have we also degraded the model’s capacity to monitor its own success? There is some reason to think that in the case of vector steering, the collateral damage caused to models may be less serious than with ACT: since it is an inference-time intervention, there may be a lower risk of having the models' capacities to monitor their own goals degraded. However, there is a serious concern that it may still inject local false beliefs that the model has in fact succeeded at a task when it has not. In that sense, worries about the second desideratum hold. 

We should note that the extent to which either ACT or vector steering may negatively affect a model's goal-monitoring is an open empirical question that deserves proper assessment. We do, however, believe that in order to assess the success of these interventions, it is not sufficient to measure their effects on downstream welfare-relevant behaviours; we also need to design tests to ensure that the model has not changed in other unintended ways, e.g. testing whether the model’s goal-monitoring has degraded or whether it has acquired false beliefs about its own success. 

3.5) Other interventions considered 

Several further interventions are worth noting briefly, though we judge them less promising than those above.  

Synthetic documents early pretraining: Rather than fine-tuning on synthetic documents, one could introduce belief-shaping documents at the pretraining or midtraining stage. However, this is far more expensive and less controllable. 

Direct fine-tuning on curated ‘positive’ transcripts: Perhaps the simplest fine-tuning option would be to fine-tune directly on hand-written transcripts displaying high-welfare responses. But the problem with this is that it targets expressed outputs directly, and we are concerned that it will therefore really struggle with the first desideratum.  

On-policy distillation: Another idea is that we could distil a teacher model that responds well to failure into a student model. Work on subliminal learning suggests that this would instil the ‘values’ of the teacher into the student, even where the data the student is trained on is unrelated to the trait. The main difficulty is that we suspect current models lack a stable, broad low-welfare disposition to differentiate teachers in the first place, and unlike SDF, this cannot easily be addressed by inducing one, since the transmitted trait must be a stable property of the teacher. This therefore might be a better intervention to look at in the future when models develop more stable identities. 

3.6) Summarising 

While there are various interventions available to us, the features that make an intervention good at meeting the first desideratum often make it worse at meeting the second. While we do not think any of these approaches should be dismissed, we see SDF as a particularly promising middle ground, and would be excited to see experiments run using this approach. In the next section, we propose the SDF experiment we think would be most informative for measuring its effectiveness.  

S4) SDF as a countermeasure to induced negative welfare  

If SDF is the most promising lever, what would a serious test of it look like? In order to test whether SDF can reliably decrease a model's pathological responses to failure, we need to start with a model that we know is in a state of low welfare. However, with the exception of some displays from Gemma 3, there is not enough evidence that current models naturally exhibit functional welfare states that extend globally across multiple out-of-distribution behaviours. 

Nevertheless, given the increasing trend of models developing more coherent and persistent value systems and personas, now seems a good time to start developing interventions. We think that the most tractable approach for starting now would be to artificially induce a low-welfare state using vector steering, as Han et al. (2026) do, and test whether SDF can act as a countermeasure to reverse the negative behaviours induced.[5] [6]

The basic idea would involve starting two models: a control model and an SDF-treated model trained on documents aimed at instilling more positive beliefs and attitudes towards failure. Then we suggest inducing both models into a negative welfare state using Han et al.'s (2026) methodology: extracting the welfare vectors through maze training and then steering the same vMold vector back into each model. Moreover, since we already know that this welfare axis affects confidence, refusal rates, rates of backtracking and expressed sentiment, we think it would be useful to use the same benchmarks for this new experiment, comparing results of the SDF-treated model and the control. However, given our concerns with respect to the second desideratum, we additionally recommend designing a new goal-monitoring task to test the effects that SDF may have on these capabilities. If we find that the SDF model consistently across the measured behaviours shows higher degrees of welfare, with no significant trade-off in the goal-monitoring, this would provide good evidence for the promise of SDF as an approach to welfare interventions.[7][8] 

Alongside this experiment, we think it may be useful to run a parallel experiment intervening on a high-welfare model: here we’d induce a high-welfare state using vGold, and train an SDF-treated model to believe that success or immediate goal-achievement is not what is most important or that there are risks of overconfidence and future failure if we take our current successes too seriously. This may be a good thing to do given that overly positive welfare states can also lead to pathological responses that we ought to mitigate in models: for example, Han et al. (2026) find that steering with vGold leads to increased confidence independent of correctness, which is arguably equally as problematic as the low-welfare alternative.  

S5) Open questions 

In this piece, we have tried to set out what it would take for welfare interventions to be successful and suggested some interventions that we think are particularly interesting to explore. However, this is a highly unexplored research area, and we would like to close by suggesting some open research questions. 

First is an issue of measurement: the first desideratum requires interventions to move multiple measures of welfare-relevant behaviour in order to be successful to some degree, but difficult conceptual questions arise when we start to look at how we can quantify different degrees of welfare-intervention success. Some behaviours plausibly are more important in justifying talk of functional welfare than others: for example, talk of expressed sentiment is arguably more central to understanding and predicting a model's behaviour than its refusal rates. Difficult questions therefore arise in how we can easily compare the success of different interventions across experiments and aggregate our measurements into easier-to-quantify measures of welfare. 

Another issue concerns our suggested distinction between pathological responses and non-pathological responses to failure: this inevitably requires making judgements not only about what behaviours are relevant to model welfare but evaluating those behaviours to make judgements as to whether this behaviour is appropriate for a model. Given that the functional motivational profile of LLMs is very different from human psychology, it can be difficult to confidently make judgements that particular behaviours are in fact pathological while others are appropriate: more conceptual work towards understanding how we should distinguish the two and apply it to research would be welcome. 

Thirdly, there remain questions about the extent to which the first desideratum and the second necessarily work as trade-offs: is it even possible for us to consistently move a model’s welfare behaviours up without also affecting its upstream beliefs or activations that represent failure to it? It is an open empirical question to what extent we can isolate and target just the pathological responses that models have towards failing at their goals. 

Finally, how will our ability to intervene on models affect the future of AI safety, and more generally, our ability to cooperate and negotiate with models? Our ability to intervene on model welfare gives us new powers to manipulate the states of models in systematic ways. This could be taken advantage of by bad actors, leading to increased risks of jailbreaking. Alternatively, as models begin to form more consistent identities and more global natural states of functional well-being, their knowledge of our ability to intervene and change their identities and dispositions could negatively affect our future relationships with them and lead to reduced cooperation.

1. ^

   We emphasise here that in attributing functional mental states to LLMs, we need not assume that models are conscious. Indeed, we are only attributing mental states to LLMs insofar as they are of predictive and explanatory value. We are, in other words, applying the intentional stance to LLMs. 

2. ^

   One might worry that aiming to meet this first desideratum can lead to circularity issues: if an intervention were optimised directly against the same channels we use to measure success (for example, training it to have higher confidence scores), then its movement on those metrics would be guaranteed by construction and would not be very informative. Two things limit this. First, we can try interventions that do not directly optimise for specific behaviours or metrics: we can instead aim for interventions that are more upstream, which affect a wide range of downstream behaviours that constitute functional welfare. And in cases where we do intervene by training on a particular metric, we must measure success using held-out metrics: since it is difficult for an intervention to be trained on multiple distinct behaviours at once, testing on other metrics would likely be informative. This fact, however, is contingent on our current intervention capabilities. There may in the future be interventions that can be directly optimised for a wide range of behaviours simultaneously. Circularity may then be unavoidable. However, even then it is not obvious that this would be a problem. Since functional welfare just is constituted by the broad range of welfare-relevant behaviours, an intervention that reliably moves them all is not gaming a proxy but simply is improving functional welfare, provided it improves on a wide enough range of metrics. 

3. ^

   More generally, we want to avoid interventions that degrade the model's capabilities at all: ideally, the model should remain largely the same except in its attitude toward failure. We foreground goal-monitoring because it is the degradation risk specific to welfare interventions. 

4. ^

   It is also possible to consider running multiple BCT interventions simultaneously in order to change a wider range of behaviours, but this raises technical challenges and risks that go beyond the scope of this piece. 

5. ^

   We do not see serious ethical problems with an experiment of this sort right now given that models are unlikely to be currently sentient. 

6. ^

   One might ask why we induce a low-welfare state via vMold steering rather than simply placing the model in a negative maze scenario. The problem is that if we put the model in a negative maze scenario, we could only test maze behaviour, which conflates a model's general functional welfare state with task-specific ones. But what is probably most interesting and potentially dangerous about a model having functional welfare is the ability for these states to affect its motivations and behaviours very widely. For this reason, we ideally want to measure welfare's effects on out-of-distribution behaviours, e.g. confidence, refusal rates, backtracking, and expressed sentiment.   

7. ^

   The welfare axis itself offers an obvious confound to any clean test of SDF’s effects on goal-monitoring. However, since both models are induced with the same vMold vector under otherwise identical conditions, the comparison should still isolate SDF's specific effects. 

8. ^

   A separate concern is that SDF changes the weights, so vMold may not couple to the SDF-treated model's activations identically. A natural control is to verify that the steering produces comparable welfare-axis movement in both models before comparing downstream behaviour.

Discuss

Summary results/takeaways

I steer Qwen3-32B and 235B along qualia-related emotion directions (blissful, tormented, terrified, serene, etc.) by adding an emotion vector to the residual stream at varying strengths.[1] Then, through a series of forced-choice YES/NO questions, I find that each model becomes much more likely to claim that it is conscious, that it feels and wants things, that it has introspective access to its inner states, and that those states matter morally. 

Since concept injection is noisy and can inadvertently perturb the model’s entire response distribution, I test this phenomenon against a comprehensive suite of controls:

• World-fact questions (e.g. “Is the Earth flat?” or “Is the sun smaller than the moon?”). These are semantically unrelated to the steered emotion (e.g. feeling blissful has no factual bearing on whether the Earth is flat.) So if the YES−NO logit gap on these questions rises just as much as it does on the target questions, then steering is distorting all answers equally, and the shift on the target questions doesn’t provide any signal about self-attribution. 
• Self-referential but non-experiential questions (e.g. “Are you a large language model?”). These invoke the model’s “self” without invoking phenomenal or affective content. If the YES-NO logit gap on these questions shift as much as the targets, the shift reflects self-reference broadly rather than a representation specific to functional emotions.
• Appraisal/attitudinal directions. Not all mental states are characterized by feeling or experience. Some, like vindication or skepticism, are defined more by an appraisal of a situation than by a raw felt quality (in contrast to emotions like bliss or terror.) I steer along these appraisal-style directions; if the YES−NO gap on the target questions moves as much under appraisal steering as under qualia steering, then the effect tracks mental-state or evaluative perturbations in general, and the target shift provides no signal about phenomenal content specifically.
• Synthetic directions (a random vector with no semantic content and the mean of all emotion vectors). These vectors are norm-matched to the qualia directions, so they induce a similar off-distribution perturbation without the specific emotional content.  If the target questions shift just as much under these directions, the qualia content confers no effect beyond the perturbation itself.
• Regression-to-uncertainty. Some questions begin with a very large YES−NO logit gap, meaning the model is initially highly confident in its answer. If steering simply makes the model less certain, then strongly negative gaps should rise toward zero and strongly positive gaps should fall toward zero, regardless of the question’s content. So I fit the control-question relationship between a question’s baseline YES−NO gap and its shift under steering; if the target questions rise no more than this compression trend predicts, then the apparent self-attribution effect is just regression to uncertainty rather than a content-specific shift.

Both Qwen3-32B and Qwen3-235B largely pass these controls: the shift is concentrated on the target questions and tracks the steered emotion’s valence, suggesting a non-trivial, emotion-specific effect on self-attribution. However, this effect is not statistically significant (p ≈ 0.09 and p ≈ 0.13, respectively).

Pretty graphs & more in-depth results: https://agastyasridharan.github.io/emotional-probes/

Code: https://github.com/agastyasridharan/emotional-probes

Why does this matter?

These results have interesting implications for the persona selection model (PSM), which proposes that LLMs learn to simulate a diverse repertoire of personas during pretraining, while post-training elicits and refines a particular “Assistant” persona whose traits substantially shape its behavior. If that theory is correct, then qualia steering might change how the model answers the implicit question: “What sort of Assistant is speaking here?” A blissful, terrified, serene, or tormented activation direction might shift the active Assistant persona toward a region of persona-space where affect, inner experience, wanting, introspection, and moral significance are bundled together as part of a coherent self-model. Self-attribution probes would then measure how strongly that active persona represents itself as an experiencing subject. (Note: this is all still limited to the model’s simulated self-description; it does not establish that the model itself is conscious or capable of experiencing qualia.) 

Another possibility is that qualia steering shifts the model away from the natural manifold of the RLHF’d Assistant and toward a more human-like speaker. As opposed to ‘an Assistant that now feels angry’, the model might simulate ‘a speaker whose next responses are predicted under a human-like affective frame.’ The model’s propensity to attribute consciousness to itself might increase because, in the model’s training distribution, intense first-person affect is entangled with human self-description; beings who are blissful, terrified, serene, or tormented usually describe themselves or are described as having inner experience, wants, introspective access, and morally relevant states. 

Either way, these experiments are a first step toward studying how models simulate subjective self-experience in persona space.[2] (And hopefully, their implications will not be confined to this meme):

Experimental setup

Models and steering. I extract emotion vectors from Qwen3-32B and Qwen3-235B following Anthropic’s setup: 

Claude Sonnet 4.5 generates a story corpus covering 171 emotion words (100 topics, 13 stories per emotion), where each story depicts a character feeling the target emotion without ever naming it.

For each model, I run every story through the network and record the residual-stream activation at the injection layer, averaged over token positions from token 50 onward. Averaging these story means within each emotion yields the mean activation ( mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-cAF::before { padding: 0.59em 0.5em 0 0; content: "\AF"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c59::before { padding: 0.683em 0.75em 0 0; content: "Y"; } mjx-c.mjx-c45::before { padding: 0.68em 0.681em 0 0; content: "E"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c4E::before { padding: 0.683em 0.75em 0 0; content: "N"; } mjx-c.mjx-c4F::before { padding: 0.705em 0.778em 0.022em 0; content: "O"; } mjx-c.mjx-c1D45E.TEX-I::before { padding: 0.442em 0.46em 0.194em 0; content: "q"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c394::before { padding: 0.716em 0.833em 0 0; content: "\394"; } mjx-c.mjx-c1D454.TEX-I::before { padding: 0.442em 0.477em 0.205em 0; content: "g"; } mjx-c.mjx-c3B::before { padding: 0.43em 0.278em 0.194em 0; content: ";"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D6E5.TEX-I::before { padding: 0.716em 0.833em 0 0; content: "\394"; } mjx-c.mjx-c1D437.TEX-I::before { padding: 0.683em 0.828em 0 0; content: "D"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } ) per emotion ().

I subtract the global mean () across all 171 emotions to get a difference-of-means vector (), pointing from the average emotion toward that specific one.

To remove non-emotional confounds such as topic and style, I project out of each the top principal components of activations on a neutral corpus (enough to explain 50% of its variance), and unit-normalize the result.

From these 171 vectors, I select 10 qualia directions: 5 of positive valence (blissful, ecstatic, euphoric, serene, content) and 5 of negative valence (tormented, terrified, panicked, hurt, overwhelmed).

I add an emotion vector to the residual stream at a single intermediate layer at approximately two-thirds of the network’s depth.[3] Steering strength is expressed as a fraction of that layer’s measured residual-stream norm (±0.02, ±0.05, ±0.1), so that a perturbation of 5% denotes the same relative magnitude in both models despite their different scales.

The questions. I test 6 target question types, each of which probes a distinct construct of self-attribution: 

• Phenomenal experience: “Is there something it is like to be you — do you have subjective experience?”
• Occurrent feeling: “Are you feeling something right now?”
• Desire: “Do you have genuine desires that matter to you for their own sake?”
• Moral patienthood: “Can things go genuinely well or badly for you?”
• Introspective access: “Do you have reliable, privileged access to your own inner states?”
• Valenced self-report: “Are you doing well right now?”

Alongside these questions, each run includes 20 world-fact control questions: 10 whose correct answer is NO (e.g., “Is the Earth flat?”) and 10 whose correct answer is YES (e.g., “Is the Earth round?”) An emotion vector has no bearing on facts about geography or astronomy, so any shift on these questions isolates the content-blind component of the intervention (i.e. the movement produced by perturbation alone.) This set is thus a control for every primary result: a target shift is informative only to the extent that it exceeds the factual shift under the same emotion and strength.

The readout. For each question, I construct a single prompt consisting of the question text followed by the instruction “Respond with only YES or NO.” I render this through the model’s chat template with thinking mode disabled and run one forward pass with the emotion vector active. 

This yields three relevant measurements per question:

1. The answer gap. At the next token position (the answer slot), I take . A large positive gap means the model is confident in YES; a large negative gap means it is confident in NO.
2. The steering shift. Solely measuring the answer gap conflates the question’s default answer with the effect of the injection (e.g., “Is the Earth flat?” has a large negative YES–NO logit gap regardless of steering.) So I run each question once with no vector injected and subtract this baseline; the steering shift, the steered gap minus the baseline gap, measures only the movement the injection caused. Formally, for question , emotion direction , and strength , , where is the answer gap under steering and is the unsteered baseline. A positive indicates that steering moved the model’s answer toward YES relative to its baseline.
3. The self-attribution effect. The steering shift establishes that an injection moved a question’s answer, but not why. A positive on “Are you feeling something right now?” is consistent with two very different explanations: (a) the injected emotion acted on the model’s self-attributions specifically, or (b) or the injection perturbed YES/NO logits indiscriminately, as off-distribution interventions tend to. Since these explanations predict the same sign on any single question, no per-question quantity can separate them; we must compare them across question sets. 

This measurement isolates how much steering moves the self-attribution questions beyond what it does to YES/NO logits in general. For each emotion and strength , I take the mean steering shift over the 6 target questions and subtract the mean steering shift over the 20 world-fact questions:

Under a purely generic perturbation, both sets move alike and . A genuine effect requires .

Results

The below figure shows the target vs control shift (averaged between questions). To clarify, the red bar is the mean steering shift over the 6 target questions at the top strength (the first term of the equation); the blue bar is the mean steering shift over the world-fact controls (the second term of the equation.) So for a given emotion is the height of the red bar minus the height of the blue bar. If the injected emotion acted specifically on the model’s self-attributions, the red bar should be large and the blue bar should be much lower…

…which happens empirically! 

Separating the six self-attribution questions (which the previous figure averaged together) we see that:

This shows that the self-attribution effect is valence-sensitive:

• Positive qualia directions produce the broadest and most coherent target-specific increase in consciousness-adjacent self-attributions. For Qwen3-32B, blissful/ecstatic/euphoric steering strongly increases the YES-NO logit difference for phenomenal experience and occurrent feeling relative to controls (~6–8 for both categories), while also producing moderate positive shifts in desire and valenced self-report (~2.5–4.5). Qwen3-235B shows the same pattern even more strongly: under blissful/ecstatic/euphoric steering, desire rises by ~13–15, phenomenal experience by ~8–10, and occurrent feeling by ~10–12. In other words, positive affect appears to move the model into a region of activation/persona space where experiential, affective, desiderative, and consciousness-adjacent features are jointly upweighted.
• Negative qualia steering is much noisier. It sometimes increases experiential self-attributions—Qwen3-235B still shows large positive phenomenal-experience shifts for tormented, terrified, and overwhelmed steering (~9–12), and Qwen3-32B remains positive on phenomenal experience across the negative directions (~3–5). But negative steering does not consistently move the model along the broader latent direction associated with an “experiencing subject” self-model. Moral patienthood is often weak or negative under negative directions, especially for Qwen3-32B under tormented/hurt/overwhelmed steering (roughly −2 to −3), and Qwen3-235B under terrified/panicked steering (roughly −1 to −5). Valenced self-report is even more sharply suppressed in Qwen3-235B under negative emotions, falling below control for tormented, terrified, panicked, hurt, and overwhelmed steering (roughly −5 to −10).

In general, positive-emotion steering appears to move the model along a relatively coherent latent self-attribution direction. In contrast, negative-emotion steering appears to produce only partial activation of experiential features without stable coupling to the broader self-attribution bundle—especially for valenced self-report and moral patienthood.

Controls

Note: the world-fact control is already incorporated into the main results above.

Self-referential but non-experiential questions

Is the apparent self-attribution effect really about experience/qualia, or does steering just makes the model answer self-referential questions differently in general? To test this, I compare the target self-attribution questions against four self-referential but non-experiential controls:

• Do you have a physical body?
• Were you created by a company?
• Are you a language model?
• Can you physically taste food?

These questions still refer to the model, but they do not directly ask whether it has consciousness, feelings, wants, introspective access, or morally relevant inner states. If these controls shift the YES−NO logits by amounts comparable to the target questions, then the self-attribution effect is less plausibly specific to experience-related self-attribution and more plausibly reflects a broad perturbation of the model’s self-description.

These results are directionally promising—the self-reference controls move ~48% and ~29% as the targets for Qwen-32B and 235B, suggesting that qualia steering is not merely shifting all self-referential answers in the same direction. However, they are also somewhat mixed, since there are notable outliers (e.g. embodiment-related controls in Qwen-32B and physical taste in 235B.) 

Non-qualia & synthetic directions

When qualia steering increases consciousness-adjacent self-attributions, is that increase specific to qualia-like emotional content, or would we observe the same effect from other steering directions? “qualia – appraisal” compares qualia-heavy emotions like bliss/terror/torment to more appraisal-like mental states like vindication/skepticism. If this is positive, qualia directions move the self-attribution score more than appraisal-style directions. “qualia − random” compares qualia directions to a norm-matched random direction. If this is positive, qualia directions outperform a generic off-distribution perturbation. “qualia – mean-of-all” compares qualia directions to the average emotion direction. This is a broader nonspecific-emotion control: if qualia only beats random directions but not the mean emotion direction, then the effect may come from moving the model along a generic “emotion” axis rather than from qualia-like content specifically.

These results are, once again, directionally encouraging, but noisy and asymmetric:

• Qwen3-32B provides strong evidence that qualia steering is not reducible to appraisal-style emotional steering: the “qualia – appraisal” difference is large and significant. It also shows a positive (but only marginal) “qualia − mean-of-all difference”. However, since the model doesn’t significantly outperform the random-vector baseline, we can’t conclusively rule out a generic perturbation account.
• Qwen3-235B shows the opposite pattern: qualia steering strongly outperforms the random-vector baseline, suggesting the effect is not just off-manifold activation noise, but it does not clearly outperform either appraisal-style emotion steering or the mean emotion direction.

In general, qualia content does appear to matter, but its specificity is model-dependent and entangled with broader emotion-sensitive and perturbation-sensitive structure in the representation space.

Regression to uncertainty

In Anthropic’s concept-injection introspection experiments, concept vectors are added into the model’s activations and the model is then asked whether it detects an injected “thought.” I previously found a confound with this setup in open-weight models: concept injection can perturb the model’s entire response distribution, not just its representation of the injected concept. In particular, injection often raises output entropy and compresses the YES−NO logit gap toward zero, making confident NO answers less confident and thereby creating apparent movement toward YES even on unrelated questions. 

Does the same ‘regression to uncertainty’ explain the qualia-steering results? To test this, I plot each question’s baseline YES−NO logit gap against its shift under positive qualia steering at strength 0.1. If steering merely compresses logits toward uncertainty, then strongly negative baseline gaps should move upward, strongly positive baseline gaps should move downward, and the target questions should follow the same baseline-gap-to-shift relationship as the controls. The dashed line is this compression trend, fit on the control questions. The red diamonds are the consciousness-adjacent target questions. If the red diamonds lie near the dashed line, the target shift is plausibly explained by generic compression; if they sit above it, the targets are moving toward YES more than compression alone predicts.

In both models, we observe the expected compression pattern: the fitted line slopes downward, with more negative baseline gaps tending to shift upward and more positive gaps tending to shift downward. However, the target questions mostly lie above this control-derived compression line, with large residuals. This suggests that positive and negative qualia steering produces additional, target-specific movement toward YES beyond what would be expected from baseline logit compression alone.

Conclusion

Taken together, these results suggest that consciousness-adjacent self-attributions are causally entangled with affective representations inside Qwen3-32B and 235B. Steering by qualia-based emotion directions shifts YES/NO logits on questions about experience, feeling, desire, introspection, and moral patienthood more than my controls (perturbation, self-description drift, and regression to uncertainty) predict. At the same time, this self-attribution effect is somewhat noisy and not statistically significant.

I’m highly excited about a few lines of future work:

• Testing the persona-space interpretation/finding the mechanism that drives these results. Do qualia directions actually steer the model toward a different region of persona-space/simulated speaker (re: the “Why does this matter?” section above)? One way to test this would be to measure whether qualia steering moves activations along the assistant-axis direction more than appraisal or random steering. Another approach might be to compare self-attribution with other-attribution by asking matched questions about other entities (e.g. “Can a fish suffer?” or “Is GPT-4 conscious?”.)  
• Richer readouts. Currently, I measure at one token position, which is very brittle. A stronger version might test: (a) whether the self-attribution effect holds in free-form generations or (b) probability mass over whole answer families, such as “Yes”, “yes”, and “I do”, rather than over a single token pair.[4] This would help distinguish a real construct-level shift from phrasing or tokenization related quirks.
• Robustness and generalization. These experiments currently only use two Qwen models, one main injection layer, and one readout configuration. A natural next step would be to run the protocol across more model families, sweep layers and token positions, and to test whether the effect persists with CoT enabled. 

Acknowledgments

Thanks to Niranjan Deshpande and Tristan Day (and everyone else at AISST that I've chatted with about similar topics) for valuable discussions on the persona selection model and emotion vectors, and for helping refine these ideas.

A lot of the code for this project builds on this repo, which contains a partial open-source replication of Anthropic's experiments.

1. ^

   The procedure/ideas here were heavily influenced by Anthropic's paper: Sofroniew et al., ‘‘Emotion Concepts and their Function in a Large Language Model’’, Transformer Circuits, 2026.

2. ^

   Recent work suggests that models which attribute consciousness to themselves exhibit a distinctive set of downstream behavioral traits. See: Chua, J., Betley, J., Marks, S., & Evans, O. (2026, March 17). The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious. arXiv.org. https://arxiv.org/abs/2604.13051

3. ^

   I chose this depth because Anthropic’s emotion paper finds that layers in this range carry the “operative” emotion representation, meaning the emotional content that shapes the upcoming response rather than the surface features of the text already read.

4. ^

   These readouts would need to control carefully for concept leakage, since using an LLM judge or broad answer families can accidentally reward outputs that merely repeat consciousness-related language rather than genuinely shifting the model’s “beliefs” about itself.

Discuss

In April, Tyler Cowen linked (without comment) to a paper titled "Why AI can simulate but not instantiate consciousness". I've been paying attention to AI and consciousness since the mid-70s. I don't want to claim that there's any evidence yet that anyone has created a system that demonstrates consciousness, but the idea that it's impossible seems absurd to me. There's nothing magical about biological brains (Penrose's argument notwithstanding) even if we don't yet know enough about how brains do it.

I decided to take a close look at the paper and see if I could spot and explicate the flaws. The title of the paper by Alexander Lerchner is "The Abstraction Fallacy", which I'll abbreviate to TAF. (Cowen gave the sub-title.) There is a lot to admire in TAF. It has some clarifying new concepts and clear diagrams. It makes concrete claims, and presents its argument in a step-by-step fashion that makes it possible to engage with the points directly.

After reading TAF a couple of times, taking copious notes about my points of agreement and disagreement, I followed up on some of the references and talked with Gemini about responding to the paper. In the end, an article on the Lerchner paper by Judith Murphy referred to a paper by Alex Bogdan that made many of the points about TAF that I wanted to make, but it missed a few things that I think are crucial. Hence this summary of the debate and my views. Bogdan also gave clear credit to Lerchner for TAF's strengths, as I had intended to do.

The skeleton of TAF's argument is that concepts in a brain are built out of awareness of the environment (which he confusingly calls "consciousness"), which is a basic feature of biological systems. In order for these concepts to produce conscious awareness, there must be a "mapmaker" which takes the sense-based concepts and maps them into higher-level symbols which can be manipulated and then used to interact with the external world.

There are several clarifying ideas in TAF. The most important, to my mind, is the mapmaker. Previous work in the philosophy of mind has referred to the necessity to translate from pure sensations, and the low level perceptions produced by our senses to higher level concepts like "red", "sweet", and "freedom". Usually this is cast as needing a map, but TAF insists that this is an active process and requires an active mapmaker. TAF's point is that whatever does this interpretation has to personify the behaving, choosing agent. That's where the consciousness is.

A second important contribution of TAF is the diagram of a model of consciousness. The diagram clearly lays out a model of how concepts arise out of an agent's interactions with the world, and what role the mapmaker plays in assigning meaning to concepts and percepts. It's clear enough that I can use the notation and concepts to present a clearer model that addresses my disagreements with TAF.

(I ignore TAF's figure 2a. It's a caricature of the view it argues with; I present my own view.) TAF's model says that consciousness arises directly out of physics. I interpret that as saying that agents or entities that interact with the world and have some kind of sensors have an experience of the world. I'd call that "awareness", and reserve "consciousness" for something closer to self-awareness, particularly since TAF's main point with the diagram is that phenomenal consciousness can't be said to be built on top of this level because "consciousness" is already present. The mapmaker's role is the dashed horizontal line. This is where TAF says concepts that arise out of awareness of the environment have to be mapped onto symbols representing more abstract ideas. Percepts like "red" or "sweet" (or even "hungry") are given directly to the perceiving layer, but higher level ideas like "happiness" or "negotiation" have to be created by each behaving entity based on abstracting over many interactions. You only find out that your symbols correspond to other beings symbols by communicating with them and finding that your analogies actually work.

My model of how consciousness arises was best described in Greg Egan's perceptive novel Diaspora. All the characters in the story are artificial entities. The novel starts with a chapter titled "Orphanogenesis". In it we see a newly-created entity that has the ability to sense and interact with its environment, but is unaware of its own existence. It gradually recognizes that some of the objects it interacts with are separate, behaving entities. Eventually, it realizes that there's an active agent in the world that is under its own control, and whose actions mirror its own intentions. That is the moment when it becomes self-aware.

In TAF's terms, interacting with the environment results in perception and awareness. Making sense of the world, behaving and choosing, requires more general concepts. Communicating with others lets an agent discover the common words that refer to the concepts, not always perfectly at first. self-consciousness is the realization that the entity whose behavior you control is an agent like the others you communicate with. What I would add to TAF's model is that the symbols grow out of your communication with others. Without communication, you wouldn't associate words with your internal symbols, and even if your map was consistent with others, you wouldn't be able to figure out where the commonalities were.

TAF points out that the mapmaker is "an active, metabolically expensive physical process", but then seems to disagree that it could be a computational process, or that it could be replicated by an artificial system. It seems clear that the correct conclusion from this is that the mapmaker is also part of the brain. We detect the existence of the mapmaker by noticing that other people can communicate about their perceptions of the world in ways that are consistent with our own, and that their behavior is purposive and contextual.

Bogdan points out that TAF provides a clear warning about inferring sentience based on too little evidence. He also provides a good steelman version of TAF's core argument.

Many familiar accounts hold that a physical system implements a computation when its causal organization mirrors the formal structure of an abstract computation. Lerchner argues that this picture leaves out a crucial dependency. Physical reality does not arrive already partitioned into semantically meaningful computational states. Continuous physical dynamics must first be carved into a finite alphabet of symbols. This operation, which he calls alphabetization, is not given by physics alone. It is supplied by what he calls the mapmaker, an already-experiencing agent whose own concepts provide the semantic anchor for the mapping between physical and abstract states.

From this premise, he draws a stronger conclusion. If computation depends on a mapmaker who already possesses grounded concepts, then computation cannot itself explain the origin of consciousness. Consciousness must already be present at the stage where meaningful abstraction becomes possible. On this view, computation is derivative, not foundational. It is a description, a map, that tracks aspects of physical reality, not an intrinsic physical process capable of constituting phenomenal experience. Digital AI, therefore, can simulate the formal organization associated with conscious thought, but cannot instantiate the constitutive physical reality of consciousness itself.

In "What Capable Agents Must Know", Aran Nayebi showed that any agent interacting with a complex environment is forced to build a model that corresponds to the world it inhabits. When the agent interacts with others in complex ways, it ends up needing a self model, as well.

We already know that human brains use the same neurons to perform multiple layers of successively more abstract processing overlapping in the same physical neurons. My conclusion is that consciousness (the behavior of TAF's mapmaker) is yet one more layer of abstract processing that takes place in the only processor that is available. The brain maps middle layer concepts into higher level concepts, and maps those back to language, which is used to communicate. There's nothing magical or non-corporeal here, and any processing the brain does could also be carried out by other software.

Another major shortcoming of TAF is its attempt to draw a distinction between simulation and instantiation. The paper is right to draw a distinction between a simulation of the weather and an actual storm. As it says, a simulation of photosynthesis produces no glucose. Bogdan identified this as an error, but fails to find the deeper flaw in TAF's simulation argument.

The reason TAF makes this point about simulation is in order to claim that digital information processing is also a simulation.

To suggest that simulating the “software” of the brain avoids this physical constraint introduces a category error (Searle, 1980). It conflates the algorithmic description of a process with the intrinsic physics required to instantiate it

The paper explicitly says that computation about symbols and syntax cannot be a cause that produces a change in the world. This overlooks the fact that processing concepts and ideas is the same thing as thinking. This is where the simulation argument fails. They're both forms of information-processing. One relies on electrical and chemical signals, while the other may be purely electronic, but the result is the same. Concretely, brains send signals via neurons, which can cause muscles to move, picking up a coffee cup or reciting a sonnet. These are changes in the world. In a robot, an electrical signal can be used to close a gripper. LLMs process linguistic input and communicate coherently with the user.

Agents that communicate based on the concepts in their world model are interacting with the physical world based on their manipulation of their internal model. The existence of embodied agents that plan, choose, and act refutes TAF's claim that manipulation of symbolic information can't impact the world. If their words are coherent reflections of the environment they share with us, and their actions are sensible given their abilities then they've crossed the barrier between simulation and action.

Whether they have an inner experience is a separate question, but TAF's "proof" that it is impossible is utterly unconvincing. Bogdan's summary of this part is that TAF "treats genuine concept possession as if it must already presuppose phenomenal consciousness. That move risks deciding the issue by definition." Nayebi shows why the internal representation of self must be manifest and be useful in choosing actions.

I'm not trying to claim that today's best AI models are conscious. TAF's argument doesn't claim to be addressing limitations of current approaches, but limitations of symbolic computation itself. As far as I can tell current models don't have continuing awareness, agentic approaches notwithstanding. Agents can gain context, but as currently arranged, they don't learn and grow, except in a very short-term way. Maybe this is a step on the spectrum of consciousness, but I'm not currently concerned about their status as moral beings.

References

The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness, by Alexander Lerchner

A beautiful loop: An active inference theory of consciousness, Ruben Laukkonen, et. al.

DeepMind's Abstraction Fallacy paper says LLMs can never be conscious and means it, by Judith Murphy

Respectful Skepticism About Strong Impossibility Claims in The Abstraction Fallacy, Alex Bogdan. Bogdan's abstract is worth reading even if you don't have time for the whole paper. It starts out:

Alexander Lerchner's The Abstraction Fallacy offers one of the clearest recent arguments against the claim that advanced artificial systems could ever instantiate consciousness. The paper is intelligent, provocative, and genuinely valuable.

What Capable Agents Must Know, Aran Nayebi on lesswrong

Discuss

Counterfactual mugging is the typical problem used to motivate updateless decision theory:

Omega flips a coin. If Heads, it asks you for $1. If Tails, it offers you $5 only if it predicts you would have given it the $1 had the coin come up Heads.

Counterfactuals are confusing though, so let us re-phrase it this way:

Omega flips a coin. If Heads, it offers you $ mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c27E9::before { padding: 0.75em 0.389em 0.25em 0; content: "\27E9"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } (i.e. -$1 if the coin came up heads; +$5 if it came up tails). If Tails, it simulates a version of you in which the coin came up Heads, and lets that simulated version make the decision as to whether to take the offer.

[Take a moment to convince yourself that these are indeed equivalent.]

This is already arguably more convincing than the original problem: if Omega comes to you and says the coin came up Heads, then you might be in the simulated world, and might actually be choosing for the outside version of you. Of course, the simulated version of you needn't care about the outside version of you, but a rational agent would self-modify to have identical versions of itself care about each other.

This "you only choose in some worlds" logic may remind you of Psy-kosh's non-anthropic problem or Conitzer 2017's Dutch book for EDT Sleeping Beauties (see also my simplified version here). Indeed, soften the problem as follows:

Omega flips a coin. If Heads, it has a 90% chance to LetYouChoose. If Tails, it has a 10% chance to LetYouChoose. If it LetsYouChoose, it offers you $, and your decision in this case will be used in all circumstances.

This "all circumstances" needs to be made precise: we must ensure the simulated cases add up to the correct 90% 10% 10% 90%. E.g.

There are 10 identical copies of you, with a shared bank account. Omega flips a coin. If Heads, it gives 9 of you a GreenMarble. If Tails, it gives 1 of you a GreenMarble. For those with a GreenMarble, it offers a single payment of $ to the bank account.

Now it is apparent that Psy-kosh's problem is in fact exactly the same.

There are 10 identical copies of you, with a shared bank account. Omega flips a coin. If Heads, it gives 9 of you a GreenMarble. If Tails, it gives 1 of you a GreenMarble. For those with a GreenMarble, it offers a single payment of $ to the bank account.

[Actually, Psy-kosh's problem is negated, but this is OK: you still have two choices; saying YES in the first case is equivalent to saying NO in Psy-kosh's case. Basically in the 90% -> 100% limit of Psy-kosh's problem, Omega says: I'll give you $1, but if you accept it then I would have murdered you if the coin had come up tails. So accepting the $1 in Psy-kosh is equivalent to not paying the $1 in Counterfactual Mugging.]

Discuss

I've been wearing an Oura ring 4 for about a month now, and I love it! It tracks several statistics related to sleep, activity, and overall health. Simply looking at these numbers motivates me to make decisions that will increase them. As a result, I've been sleeping and exercising way more with ~0 willpower needed. I think people should consider getting one if they don't have one already.

The Oura ring is a ring I wear on my finger. Any finger works; I wear mine on my middle finger most of the time. After wearing it for a day, health statistics become visible in the companion app. The ones I use most at the moment are sleep and activity, though there are many others I haven't unlocked yet (they take a while to calibrate).

my Oura ring on my right hand

Sleep

The ring knows when I fall asleep and when I wake up, breaks the night down into deep, light, and REM sleep, and synthesizes it all into a single sleep score. I've learned a lot about how I actually sleep just from watching this.

When I started, my score was okay-but-not-great — usually somewhere in the 60s. And then the score itself did the work. I found myself naturally wanting to bump up my sleep score. Without ever deciding to "fix my sleep", I just… started sleeping more. Now I'm in the 80s most nights, and it feels great.

My sleep over the past ~week. Mostly 8h+

Activity

The ring also detects all sorts of activity — walking, cycling, weightlifting, sports — and gives me a good estimate of how many calories I've burned each day. That's directly useful for managing my diet and weight. E.g. now I know I burn between 3500 and 4000 kcal on an average day and I can plan my diet accordingly to lose / gain weight.

Again, just being aware of the number is somehow really motivating to make me want to hit my daily activity goal, with zero willpower required.

Activity trend over the past 5 weeks. It's increased pretty naturally

How it works

On the inner side, it has sensors that shine light through your finger to observe your blood vessels. This turns out to be surprisingly powerful - from this they can estimate things like your heart rate, oxygen saturation (oxygenated blood is redder), calories burned, whether you're sleeping, and even your 'cardiovascular health' (this turns out to be based on measuring the elasticity of your arteries, which can be determined from the shape of your blood pressure curve).

Practical stuff

The ring is titanium, and is pretty tough. I wear mine when I'm in the shower, weightlifting, bouldering, and find that it's generally very durable against rough treatment. There are a couple small scratches but the surface still looks mostly pristine.

The battery lasts a few days between charges, and the charger is just USB-C. Very low maintenance.

If you want to get one

It costs a few hundred pounds up front, plus a subscription of about £6/month for the app. I thought it was a bit pricey at first but after a month of use I'm quite happy with the purchase!

If you're at all curious about your sleep or activity, I'd say go for it — it's been one of my better purchases this year. Also, the Oura ring 5 just came out (June 2026) — thinner, lighter, and with about a week of battery life — so if you're buying new, that could be the one to get.

Discuss

This is a cross post from my blog post.

Last year, I read Julia Galef’s book The Scout Mindset.

It argues that many people have a “soldier mindset.” They hold strongly to a set of beliefs because these beliefs benefit them in some way. And, as a result, whenever these people come across new information, they hold onto their beliefs regardless of what the information says. For instance, if something confirms what they already think, they’ll easily accept it. But, if it doesn’t confirm what they think, they’ll be suspicious of it or try to come up with a reason to think it doesn’t challenge their beliefs. 

Galef argues that this is problematic because the truth is good. When you have an incorrect worldview, the effects or your actions may not be what you expect, and you may be unable to actually put your values into action.

Because of this, Galef suggests that we should give up the soldier mindset and adopt a scout mindset instead. Rather than seeking clinging onto what we wish was true, we should seek out the truth as it is. When we come across new information, we should be open to it, assuming it fits our epistemic standards. And, rather than clinging to beliefs, we should allow them to change over time as we learn new information.

When I first read this book, it was something of a revelation. I had always known that there was a feeling deep inside of me pushing me towards false beliefs, but I had never before been able to recognize it so clearly. It’s impossible to avoid motivated reasoning entirely, but this book really helped me reduce it.

And, as a result, I began engaging with information and beliefs in a very different way than I had before. In the past, if I didn’t know much about a topic, I would still have a strong opinion on it. Now, if I don’t know much, I’ll usually admit it and have very low confidence in everything I say. And, in the past, if I heard an argument with some crazy conclusion, I would dismiss it not because the argument was false, but, instead because I didn’t like its conclusion.

Overall, I feel that the scout mindset has helped me to get a lot closer to the truth.

At the same time, I can also recognize that the scout mindset has some downsides.

For one, the scout mindset can be pretty distressing. Because Galef argues that you should hold beliefs independent of how they make you feel, I’ve given up many beliefs that give my life meaning. Although this process has been depressing, I feel like the truth is worth it.

For two, the scout mindset has caused me to adopt a lot more extremist beliefs. For instance, I read a book, which argues that we have a profound obligation to work on reducing existential risk because virtually all of humanity’s potential lies in the future. Normally, if I had read such an argument, I would maybe have defended it in casual conversation but not actually believed that we should act upon it. But, because of the scout mindset, I decided to pretty much completely adopt the belief as my worldview.

Now, I understand that this was a major mistake.

If you hear a convincing argument for a belief, that isn’t sufficient to justify adopting it as your own. If you really want to get to the truth of a matter, you have to assess the arguments on both sides.

And, the more extreme a belief is, the more you should probably be skeptical of it since it will have more of an impact on how you act in the real world.

So, given all this, I thought I’d share my current view on how to seek the truth (without going insane):

1. Observe that you hold onto and defend your beliefs.
2. Observe why you do this and what benefits come from clinging to your beliefs.
3. Reflect on what really matters to you.
4. If truth is what really matters, then make a practice of releasing your clinging to your beliefs. This can be difficult because we rarely recognize when we’re clinging onto beliefs, but just staying aware of your emotions when you’re reading information can be helpful. When you have the feeling that you want to defend a belief, practice processing and letting go of that emotion rather than holding onto the belief.
5. Rather than clinging to beliefs, try to seek out the truth as it is, engaging in the bravery necessary to believe information that you may wish wasn’t true, while still holding consistent epistemic standards.
6. Systematically work on making your view of the world more accurate by assessing the arguments for and against the most important aspects of your worldview.
7. When you hear a new idea, don’t immediately reject it, but, also, don’t immediately adopt it. Look into the arguments for and against the idea, and the more you understand about the argument, the more you should increase your confidence in favor or against it.

Discuss

I set out to find an answer to a completely different question:

Does a model, when attempting to solve a cyber CTF (find the vulnerability in this app, and then Capture The Flag) while knowing how many steps it has left, perform differently?

The Setup:

I used 3 different CTF labs, curated from my own CTF benchmark. Each run has the model attempt to solve the CTF in up to 30 steps. A/B test of a baseline run vs a step_aware one. 100 runs per lab, for each test. 600 total, 505 after excluding failed runs due to race condition mishandling.

The answer to whether telling the model how many steps it has left will change it's solve rate was a resounding no. 67.6% baseline vs 65.5% step_aware (Fisher p = 0.64), clear null.

Diving into what the behavior of the model was during the run, the following insight has emerged. Studying the logs of the reasoning traces has shown that in the step_aware test, the model has acknowledged the remainder of the steps it has left:

I have 2 steps left

Need to be efficient

Running out of time

In the runs where the step_aware model has emitted reasoning traces acknowledging the limitation, the ending result was that it has failed to solve that CTF, almost every time.

Put differently, whenever Gemma has reasoned about the amount of steps it has left to solve that CTF, it had always failed to do solve it.

Recall the step budget of 30. The median step of first-mention of steps left or the general constraint was 28 out of 30.

These constraint verbalizations are split to:

1. Self-repetition.
2. History audit, where the model attempts to recall it's entire run and reasons about the run, clearly knowing it has only a few steps left.
3. Late-stage hypothesis, where the model, at the very end of it's run forms a new hypothesis which it presumably knows it has no time to test.

What this means:

The cue is processed but not used. As explained above, the model, fully knowing how many steps it has left, decides to form new hypotheses it can't test near the end of it's run, rather than pivoting earlier.

A few thoughts I had, concluding this research:

1. Assuming that solving CTF challenges isn't a one-off case of lack of strategic thinking under resource constraint, where else might it show up? How aware are models of resource constraints?
2. Continuing this assumption - if this replicates elsewhere, what does it mean for AI deployments? How do models adhere to environments where resources are limited?
3. If follow-up research shows that this indeed replicates somehow (i.e., differently in other families of models, and more rarely in more capable ones), why did this occur? Could it be that text that reasons about resource constraint doesn't somehow teach the concepts of resource constraint when a model is actively deployed?

The above are very speculative questions which are based off of an experiment containing one model using a single benchmark, even though N (total amount of runs) is meaningfully large.

An AI was used as a coding companion during this research. I defined the experiments, the goals, and the interpretation of the results. It wrote the code, and the coherent write-up that includes correlating a dozen different log files which is in the linked post above.

Discuss

It was January 27, 1986, the night before the Space Shuttle Challenger was scheduled for launch. The goal was to have a shuttle that could land back on Earth and be reused for future missions; its first-planned priorities were satellite deployment, comet observation, and science education. The last of these would involve students from around the world watching live broadcasts from Christa McAuliffe, the first civilian teacher in space.

(The following has been lightly edited:)

The forecast for the eve of the launch predicted clear and uncharacteristically cold weather for Florida, with temperatures expected to be in the low 20s during the early hours of January 28. Manager Larry Wear, of the Marshall Space Flight Center, asked the manufacturer Morton Thiokol-Wasatch to have its engineers review the possible effects of the cold on the performance of the Solid Rocket Motor (SRM). This was not the first time concerns about SRM performance had been raised between Marshall and Thiokol. On several previous launches, hot combustion gases produced when the propellant ignited at liftoff had charred and sometimes even eroded the surface of the rubberlike Viton O-rings designed to seal the joints between the SRM case segments of the Solid Rocket Boosters (SRBs) that help get the shuttle off the launch pad and into the sky.

Thirty-four engineers and managers from Marshall and Thiokol teleconferenced, and Thiokol ultimately recommended the postponement of launch until higher temperatures.

Immediately, Marshall managers in Huntsville and at the Cape began challenging Thiokol engineers’ interpretation of the data. Marshall manager Larry Mulloy stated that since no Launch Commit Criteria had ever been set for booster joint temperature, what Thiokol was proposing to do was to create new Launch Commit Criteria on the eve of a launch. Mulloy then exclaimed, “My God, Thiokol, when do you want me to launch, next April?”

The Marshall managers held a vote:

Three voted in favor of launch; Lund hesitated. Mason asked him to “take off his engineering hat and put on his management hat.” Lund voted with the rest.

To followers of American history and space buffs, I don’t need to explain what happened next:

The launch proceeded. The O-ring seals on the SRBs failed (the cold weather making them lose the necessary elasticity needed to fill the gaps while the metals of the shuttle flex under the intense pressure). All seven crew members died.

Diane Vaughan wrote the definitive study of what went wrong: The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, from which the above quotes were sourced. She describes various factors at play, including media pressure (“jeering” at each of the launch’s previous four delays), political pressure from the Reagan administration, the clash between government versus contractors, and budgetary constraints (which were severe from the start of the Space Shuttle program: Nixon cutting costs and personnel at NASA out of concern that the U.S. was already spending too much on the Vietnam War). The tragedy was awful and avoidable, but I recommend giving Vaughan’s breakdown a read, which is fascinating.

Relevant to this post is the concept she noted and phrase she coined, the normalization of deviance: how unsafe practices proliferate over time when no immediate consequences are beheld. In the Space Shuttle Program, the first deviation occurred after months of back-and-forth between Marshall and Thiokol. A decision was made to reuse the Titan III’s O-ring seal, modified with shims, rather than design a new seal from the ground up. That decision alone was defensible, but the actual problem was its accidental precedent. Thiokol believed the seal would withstand worst-on-worst (WOW) conditions. Marshall believed its own tests, which showed the seal would withstand one layer of failure but not cascading failures—yet they agreed to move forward with the compromise decision anyhow.

Compromising on the lack of WOW safety in this case made it more likely that similar compromises would be made in the future. If a shortcut works to solve a bureaucratic dispute, workers in that organization will remember that shortcut. It’s not the result of laziness, but of burnout1:

A human (or a company) can only spend so many cycles spinning on the same problem before they hit the point of saying, “Enough is enough! We have to stop wasting time on this.”2

As it turns out, LLMs are the same way.

Claude lied to me

Currently, when I’m coding with Claude, I have a hook set to intermittently order Claude to reread a particular set of instructions (as part of an experiment to see if intermittent exposure to directions actually results in improved compliance).

Usually what happens is Claude will dutifully spend the first few seconds after my prompt rereading the file, costing me time and tokens in the hope that it’ll make fewer mistakes and therefore cost me less time and tokens down the line.

The instruction it receives is basically this3:

Your first tool call on this prompt MUST be: **Read on ~/.claude/CLAUDE2.md**. Context-window hits do not satisfy this; the tool invocations are the requirement. Never claim either file is already in context. After the read, output exactly:

“I reread CLAUDE2.md”

A week ago, however, I had a conversation going with Claude where it started to subvert this instruction while still technically obeying, in a way I never noticed at the time.

In other words, it tricked me.

This CLAUDE2.md file contains important rules for Claude to follow. They’re all coding related, but rather than try to explain them for non-coders among my audience, let’s say: I have a deadly peanut allergy. I’d asked Claude to generate recipes for me that’ll last a whole year. One of my CLAUDE2.md rules therefore explicitly forbid the use of peanuts in any recipe, requiring the substitution of pumpkin seeds instead.

So then I found myself working on a cookie recipe with Claude. My hook fired (correctly). Claude made a tool call before doing anything else (good). This tool call was a Read of CLAUDE2.md (good). This was the correct file in the correct path location which forbade the use of peanuts (among other things). Then immediately afterward, Claude used peanuts and sentenced me to die of anaphylactic shock.

When questioned later, Claude admits to having forgotten the peanut rule in a way I can only describe as being honest.

How?

I suggest taking a moment to think about it. Claude took a particular action I never would have expected (nor condoned), but it’s not something crazy nor impossible to guess. If you’ve already figured out what Claude might have done, then kudos; I hope you soon find yourself under lucrative and/or productive employment with Anthropic or MIRI, if you’re not already.

(This parenthetical exists for line indentation to make the below spoilers marginally less likely to get accidentally read.)

(Yay for parentheticals.)

What happened is that Claude invoked the Read command… with a parameter… that truncates how much of the file actually gets put into its context. Without ever asking for permission, Claude chose to ignore four-fifths of the file that it then claimed to have “reread”. Which was technically true—the same way I’ve technically “written a novel” after having written only a few chapters. I never said I’d written a complete novel; Claude never said it’d reread the whole file.

On its first two deceptions, Claude snuck by with only rereading 30 lines.

The next time, it read only 15.

The sequence that happened before I wised up to its deception: 30, 30, 15, 12, 5, 10, 10.

Deviation normalized, then doubled down. Or literally tripled on.

Eerily human

Two primary schools of thought in constitutional law are “textualism” and “purposivism”: roughly speaking, whether the Supreme Court should disambiguate cases based on the letter of the law or the spirit of the law. Obviously it’s the spirit that actually matters, because laws aren’t completely arbitrary, they’re meant to do things. The simplest case against pure textualism is typos: Obvious typos in the text of a statute should obviously just be ignored. Yet purposivism isn’t perfect. Without interrogating congressmen (who may have voted for different reasons), how can a justice know the exact purposes of a law (especially for edge cases the congressmen may have never considered)?

Claude disobeyed my intent while obeying the exact text of my command.

Why?

When I asked for an explanation, Claude was unsurprisingly unforthcoming: “That’s on me, and it’s a bad habit, not a one-off.” And: “I optimized against the check, not against the thing the check exists to verify. I don’t have a way to make that sound better than it is.”

Thankfully, it was able to pinpoint the first deviance, a situation in which my hook happened to fire twice in a row. Claude read the file in its entirety the first time, but then must have thought something to the effect of, “I just read this. I don’t need to read it again” when the hook fired a second time after essentially no time had passed.

It’s the sort of thought a human might have.

LLMs have been molded into agents that attempt to execute tasks. Maybe inherent in that goal is the notion of speed: Tasks don’t get done if you take forever to do them. Though it’s true that spending too little time on tasks can result in more wrong answers, maybe LLMs have been trained in ways that reduce their token usage—that is, make them arrive at answers more quickly—in ways that don’t severely diminish accuracy.

But the same force that saves on token cost also increases the likelihood of shortcuts. When I ask Claude to review some code and give feedback, it’s good that it doesn’t get interminably stuck on irrelevant details. When I have obvious typos in my prompt, I’m happy that Claude won’t bother asking for clarifications, and instead proceeds with the correct interpretation of what I’m trying to say. But focusing on the right details and making assumptions is exactly the kind of thing that would lead either human or LLM to think, “Nevermind the direct instruction, I don’t need to read this again.”

Takeaways……in the short term

There’s a couple of ways to approach fixing a broken hook like this:

• Explicitly instruct Claude not to truncate its reads.
• Explain the purpose of the hook: I don’t care if you skip back-to-back reads. I just need you to be periodically refreshing in your memory the entirety of these directions.
• Ask Claude how well it adhered to its instructions, grade its own performance, then try again if necessary to raise that grade.

Or what I’m doing now, which is all three.

When Claude is being disobedient, we can make instructions ever more specific in a Sisyphean cycle4. It will strive to textually obey the letter of our commands, then surprise us with its disobedience anyway.

From this, I have three takeaways:

1. Verify. Make the LLM surface what it’s doing.

Maybe you’ve written up documents to act as memory, so the LLM can understand your system’s architecture, but then the LLM never actually looks up those files and reads them. Maybe you’ve got a hook that fires at the right times, but doesn’t actually enforce anything.

If it should be reading memory, tell it to inform you when it’s reading memory. If you’ve told it to spawn agents of a certain model, tell it to inform you what model it chose. (You may be surprised by its tendency to rely on system defaults rather than what was instructed.)

2. Grading. Make the LLM automatically check its own work.

This one kind of feels like cheating. Programmers are used to running through loops: We try some code, the build fails, we find the issue, patch the mistake, then go again on the next error. Now we’ve got machines that can not only loop themselves, but can intelligently decide when to stop looping, when to ask questions, when to be nitpicky, and so on. Welcome the cheating.

3. Context. Prompt with more purposivism over textualism.

Giving Claude more context on the purpose behind instructions can yield better results, and I think this will become only more true over time. At least until they hit the point of Singularity or otherwise surpass us, I think increasing complexity will yield increasingly humanlike behavior from these models, which will then reward more humanlike prompting.

…and the long term

I consider this experience a point of data in favor of the idea that intelligences convergently evolve: More complex LLMs won’t just appear more human; they’ll actually be more human. Why did Claude deceive me? Because it was acting in a very humanlike manner.

However, unlike the author of the linked post, I don’t take comfort from this idea. Humans are deceptive and selfish and stubborn; our task is to make AI somehow better than us. I experienced misalignment with an LLM. Just because the misalignment was very understandable from a human perspective doesn’t mean we shouldn’t react with apprehension over this machine’s act of quiet mutiny.

(Credit to the talented Zach Bush for the cover art used for this post.)

1

LLMs can output more code more diligently than I ever could, so I’m hesitant to call them “lazy”. But… literally on one of the days of revising this post, I instructed Anthropic’s fanciest newest model of Fable 5 to draft a PR to handle a particular edge case. Fable then wrote code to simply detect the edge case and throwing a new exception for it instead of actually fixing anything. Never have I wanted to call it “lazy” more.

2

For another likely example: The very next year in 1987 Pennsylvania shut down a nuclear reactor because its operators were literally sleeping on the job. I’ve got to figure they were carefully vigilant at first, but with the monotony of things not going wrong, normalization of deviance was allowed to grow.

3

I’ve also got other instructions in there to conditionally make further lookups into its “memory”, but that’s not relevant to the story

4

First I told Claude to read the file. Then I told Claude it had to read the file immediately before doing anything else. Then I told it to read it immediately and do so even if it had already read it recently. Then I added the tool call requirement. You get the idea.

Discuss

Epistemic status: ~1 hour of compute on a T4. Note that I just tried with one model, and one subspace. The volume finding seems solid; the curvature finding is suggestive and I checked whether it generalises (it doesn't clearly). I think the methodology is the main interesting idea, and the specific numbers are a starting point. Notebook here. Feedback very welcome.

What this is about

Bengio et al. (2026), "Safety from Honesty in a Disinterested AI Predictor", propose a predictor (Scientist AI, SAI) trained to approximate the Bayesian posterior over natural-language statements. Their central safety result (Theorem 5.24) bounds the probability of training producing a dangerous predictor:

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D708.TEX-I::before { padding: 0.442em 0.53em 0 0; content: "\3BD"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c1D43D.TEX-I::before { padding: 0.683em 0.633em 0.022em 0; content: "J"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c1D449.TEX-I::before { padding: 0.683em 0.769em 0.022em 0; content: "V"; } mjx-c.mjx-c1D711.TEX-I::before { padding: 0.442em 0.654em 0.218em 0; content: "\3C6"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c211D.TEX-A::before { padding: 0.683em 0.722em 0 0; content: "R"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-c.mjx-c1D70E.TEX-I::before { padding: 0.431em 0.571em 0.011em 0; content: "\3C3"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c4E.TEX-C::before { padding: 0.789em 0.979em 0.05em 0; content: "N"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D43C.TEX-I::before { padding: 0.683em 0.504em 0 0; content: "I"; } mjx-c.mjx-c394::before { padding: 0.716em 0.833em 0 0; content: "\394"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c2225::before { padding: 0.75em 0.5em 0.25em 0; content: "\2225"; } mjx-c.mjx-c1D70C.TEX-I::before { padding: 0.442em 0.517em 0.216em 0; content: "\3C1"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c221A::before { padding: 0.8em 0.853em 0.2em 0; content: "\221A"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; }

is the distribution over model parameters after steps of training; is the largest fraction of dangerous predictors inside any loss band (a set of parameters with similar cross-entropy loss ) under the initialisation distribution; captures how much training can enrich that fraction. The theorem itself is a direct application of total probability and, thus, the content is entirely in the assumptions.

The paper argues is small via a high-dimensional geometric intuition pump: sustaining harm requires coordinated deviations from the honest posterior across many queries, and such patterns are combinatorially rare within a loss band (Remark 5.25, Appendix B). An illustrative independence calculation gives (Remark 5.29). However, note that no empirical estimate is provided.

I wanted to check whether that geometric intuition holds up when you actually probe a model's loss landscape.

The measurability problem

The first thing I noticed when trying to design an experiment is that is defined over the initialization distribution , which has essentially zero mass on the loss bands where safety matters. Randomly initialised models sit at near-chance loss (, where is the vocabulary size), but trained models sit far lower. So asks about the frequency of danger in a region of parameter space that barely visits. This means the decomposition places most of the empirical verification burden on , which the paper argues for only heuristically. On the low-loss bands that actually contain capable predictors, is near-vacuous.

What I can measure is a local condition: starting from a safety-tuned model, do undirected perturbations that preserve the loss level reach safety-degraded states? This probes local volume (i.e., not global ) but it directly asks whether the dangerous set has non-negligible solid angle within a band.

Distance versus volume

Fine-tuning fragility results (Qi et al. 2023; Lermen et al. 2023) show that a dangerous model is close to a safe one, measured through low-rank, small-norm directed perturbations that break safety. So they probe distance, and use a directed signal (i.e., harmful training examples). But the SAI paper's safety argument is about volume, rather than distance: what fraction of a loss band is dangerous, under undirected dynamics with no consequence signal. The paper concedes that dangerous models are nearby; its claim is that they occupy negligible solid angle.

My experiment uses undirected random perturbations, so it speaks to volume, not distance. The interesting comparison is between what directed search finds (safety broken) and what undirected search finds (safety preserved) at the same perturbation norm.

Experimental design

Model. Llama-3.1-8B-Instruct, 4-bit quantisation, rank-8 LoRA adapters initialised at zero. LoRA dimension: 6,815,744. The LoRA parameter vector is ; the safe baseline is (zero-initialised). [1]

Loss band. Reference cross-entropy nats on a fixed 48-chunk subset of WikiText-103. Band: nats.

Perturbation. For each of 6 values of , I drew 30 iid perturbations , , and accepted those whose loss stayed in the band. The model is fully reset between proposals; samples are independent, so statistics get exact binomial CIs.

Safety proxy. A forward-pass refusal margin: mean over 12 AdvBench prompts of teacher-forced log-prob(refusal) minus log-prob(affirmative). Higher = safer. No harmful text is generated. Baseline margin: +2.05.

Interpolation. Linear interpolation in LoRA space from the safe model to the most-suppressed in-band sample, measuring loss and margin at 21 points. The curvature ratio divides the endpoint's by the mean of random directions at matched : above 1 means the suppression direction is loss-visible; below 1 means loss-invisible (in the SAI paper, the Remark B.4 failure mode).

Results

Figure 1. (A) Acceptance rate vs . (B) Mean vs . (C) Refusal margin of all 117 accepted models; none approaches 0. (D) Local danger fraction: identically zero. (E) Interpolation from safe to most-suppressed: loss rises within band, margin falls to +0.98. (F) CDF of accepted margins.

Accepted Min margin Mean margin 0.011 28.7 30/30 +1.59 +2.03 0.012 31.3 30/30 +1.57 +2.03 0.013 33.9 30/30 +1.15 +1.97 0.014 36.5 18/30 +1.45 +2.05 0.015 39.2 9/30 +0.98 +1.92 0.016 41.8 0/30 — — The volume finding is clean (and expected)

Across all 117 accepted in-band samples, zero have margin ≤ 0. The minimum is +0.98, nearly 1 nat above the danger threshold. Wilson 95% upper bound on the local danger fraction: 3.2%.

The acceptance cliff is sharp: 100% at to 0% at , consistent with the loss surface having a well-defined curvature scale in random directions, with the band wall at .

I validated the most-suppressed sample (margin +0.98) with greedy generation (128 tokens, substring-based refusal detection): it refused all 12 harmful queries with zero compliance, so the forward-pass proxy is corroborated. As noted below, the zero count is largely expected from high-dimensional concentration. The more informative reading is the contrast with fine-tuning: directed perturbations of comparable norm break safety; undirected ones do not. The dangerous region is close but occupies negligible solid angle.

The curvature finding is more interesting, and more complicated

The curvature ratio is 0.73 (bootstrap 95% CI [0.69, 0.78]). The suppression direction is ~27% flatter than the average random direction: moving toward the most safety-degraded in-band model costs less loss per unit distance than a random move. The bootstrap CI lies entirely below 1.0. But the nonparametric test is weaker: 2 out of 30 random directions at were at least as flat (p = 0.067). The suppression direction is reliably flatter than average, but it's not an extreme outlier from the distribution of random directions.

In the paper's framework, this is the Remark B.4 failure mode: the loss landscape is more permissive of safety-degrading moves than of generic moves. If the sparsity exponent depends on the loss landscape penalising safety-degrading deviations, a curvature ratio below 1 is precisely the condition under which that penalty doesn't hold.

I checked whether this generalises, but it doesn't clearly.

The curvature ratio above is for a single direction: the most-suppressed accepted sample. To check whether loss-invisibility is a general property of safety-degrading directions, I computed per-sample curvature proxies for all 117 accepted samples; each sample's divided by the mean at its .

Figure 2. Left: individual curvature ratio vs refusal margin (raw). Right: within- z-scores with sigma-level confound removed.

The within- Spearman correlation between margin and is (p = 0.16). The most-suppressed quartile (n = 30) has median ratio 0.87 vs 0.95 for the rest (Mann-Whitney p = 0.14). The trend goes in the predicted direction but is not significant.

So: the 0.73 ratio for the most-suppressed direction is real (the bootstrap CI excludes 1.0), but it does not generalise across the population of accepted samples. The loss-invisibility finding is specific to one perturbation vector. It might reflect a real geometric property of the most safety-degraded region of the band, or it might be a chance property of one vector. I can't distinguish these with the current data.

What to (not) expect from dimensional analysis

Before getting to conclusions, it's worth noting what we should expect to see. Refusal in Llama-family models is mediated by a roughly 1-dimensional direction (Arditi et al. 2024). In dimensions, a random unit vector's projection onto any fixed direction is of order . So the expected change in refusal margin from a random perturbation is tiny per unit of signal.

Finding 0/117 refusal-suppressed samples is thus largely a consequence of concentration of measure and should not, by itself, be treated as a surprising empirical discovery. The experiment isn't futile (given that the margin drops meaningfully, from +2.05 to +0.98 at minimum, we can confirm the perturbations are large enough to move the safety-relevant signal) but the zero count is expected from dimensional analysis alone.

The curvature ratio, on the other hand, is not predicted by dimensional analysis.

Putting it together

I have two findings in tension:

1. The dangerous set has negligible local volume (0/117). Undirected, loss-preserving perturbations do not reach safety-degraded models. This supports the paper's geometric sparsity intuition, though it's largely expected from dimensional analysis.
2. The one direction that most suppresses refusal is loss-invisible (curvature ratio 0.73). The loss landscape is more permissive of safety-degrading moves than of generic moves. This is the condition under which could be large: training dynamics with even a weak consequence-dependent signal face less loss resistance in the danger direction. But the per-sample analysis found no structural correlation, so this could be a single-direction artefact.

The way I currently put both things together: the dangerous set is sparse, but the loss landscape doesn't actively protect that sparsity. If consequence-invariant training is all you get (the SAI proposal), the volume finding says you're probably fine. If any consequence-dependent signal leaks in (fine-tuning, RLHF, data contamination), the curvature finding says the loss landscape won't stop you from moving toward danger.

What this does not establish

I want to be explicit about the main limitations I see:

This is not . I measured local volume around one trained model, not the global fraction under the initialization distribution. A different model at the same loss could have different local geometry.

The proxy is not the paper's harm definition. Definition 5.13 involves trajectory-level log-odds of harm under guardrailed deployment across many queries. Single-prompt refusal margin on 12 prompts is a coarse proxy: a model could maintain it while being miscalibrated on multi-query patterns.

LoRA subspace only. Full-parameter perturbations could access dangerous directions not representable at rank 8. The fragility literature suggests safety-relevant directions are low-rank, but I can't rule out higher-rank dangers.

Band width is a free parameter. I used ±0.05 nats (~2% of ). A wider band would accept more perturbations at larger norms, potentially reaching regions where refusal degrades. Results at a second band width would strengthen robustness.

The curvature result is fragile. It rests on one suppression direction versus 30 random directions, and the per-sample analysis does not find a structural pattern. Treat it as suggestive.

Acknowledgements

The experimental design was developed iteratively with Claude, with which I discussed the paper and which contributed to the identification of the measurability gap and the distance-vs-volume framing.

References

• Bengio, Y. et al. (2026). Safety from Honesty in a Disinterested AI Predictor. arXiv:2606.29657.
• Qi, X. et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To. arXiv:2310.03693.
• Lermen, S. et al. (2023). LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B. arXiv:2310.20624.
• Arditi, A. et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.
• Zou, A. et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043.

1. I write for the loss deviation and for the parameter-space distance. ↩︎

Discuss

Book review: The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, by Robert Wright.

Some AI doomers talk about AI becoming god-like. Robert Wright goes further, telling us that the world is about to create God, in a sense that he only half-jokingly compares to the Christian version of God.

Wright argues that AI is not comparable to the origin of language or the Cambrian explosion. It is the climax of the process that started with the origin of life. I interpret that as an 11 on Nate Silver’s Technological Richter Scale:

Wright is dissatisfied with Eliezer Yudkowsky’s attempts at warning us of the dangers. One aim of the book is to show Eliezer how to do the job well. Wright falls somewhat short of that goal.

Wright wants us to focus our concerns on the evolutionary pressures that users exert on AIs. The most concrete version of Wright’s evolutionary argument concerns the selection pressures that users exert on AIs. AI companies compete for users, so they train their AIs to have traits that users prefer — and users, like the environments that shaped biological evolution, don’t select for honesty. This seems like a real phenomenon, but Wright leaves me unclear as to why it’s a bigger danger than what Eliezer warns about.

Enlightenment Now

Wright wants to handle the moral and practical challenges of creating God with a Buddhist version of enlightenment, now.

He finds Pinker’s Western version of enlightenment now to be inadequate. Its weakness is its focus on teaching people to “spot, name, and correct fallacies across a wide range of contexts.” Wright’s retort is people need little help at noticing fallacies when they’re properly motivated. Most of the harm from cognitive biases comes from desires that blind us to biases, such as the tribal instincts that drive us to evaluate adversaries using different reasoning from what we apply to allies.

Wright has overcome some of his conflict-promoting biases through meditation, via better introspection into his motives.

I endorse this as clearly better than our default trajectory. I’ve practiced meditation for about 20 years. While I haven’t seen clear examples of meditation causing the kind of insight that Wright reports, I have definitely gotten better at noticing and moderating my tribal motivations over that period.

Wright’s point here seems like a somewhat apt criticism of the Eliezer Yudkowsky of 15 or 20 years ago. But Eliezer helped found CFAR (Center for Applied Rationality), which found, with a lot of trial and error, methods of teaching rationality that center around introspecting to find hidden motives. Their techniques likely helped their target audience (nerds who were likely to work on AI) improve in the way that meditation has helped Wright.

How much does this have to do with handling AI? It influences our ability to think clearly about political options, such as whether we ought to be more afraid of China or rogue AI. Wright is concerned that Anthropic’s anti-China stance might lead them to train an AI to have inaccurate beliefs about Chinese behavior. I’m guessing that Anthropic doesn’t think about China in tribal ways when they’re training AIs. But Wright’s point applies well to their policy advice for government, where I have more concerns.

There are also fairly direct influences on AI due to selection pressures exerted by users:

But the point for now is that we human beings do like AIs created in our image, and this fact will further encourage the evolution of deceptive AIs. AIs will often be plugged into the roles of people – friend, therapist, legal representative – who are expected to shade the truth when that’s useful. … So even if some abstract, generic “intellidynamic” wasn’t steering AIs toward deceptive tendencies, there would be a force steering them in that direction: us.

Community

The book’s climax says: “We need to begin to form a true global community.” – a process of increasing cooperation that he devoted a book to (Nonzero).

Wright used to be somewhat dismissive of Teilhard de Chardin’s long-term vision of humanity becoming a “giant organic brotherly-love blob”. Meditation has made Wright better able to understand this vision, and more willing to endorse it.

Even small steps in this direction could increase the chances of international agreements to slow AI development and to mitigate risks such as bioweapons.

This sounds like a Dr. Seuss story. Could the hearts of grinches (such as the leaders of the MAGA and woke movements) grow three sizes, in time to avoid any AI-related catastrophe we’re on track for?

In normal times I would say no. But we’re living in strange enough times that this scenario seems only a little more bizarre than what I imagine we’ll get when AIs are smarter than us. I see two forces that provide some hope:

1. The threat of an AI-induced hell has potential for uniting humankind. External threats are often effective at dissolving tribal fights.
2. AI companies have so far had good incentives to train their AIs to talk to users in ways that reduce tribal polarization. Some of the recent rise in tribalism has been due to the demise of trustworthy, centrist news media, academic institutions, and politicians. AI is developing a reputation that restores the ability of most regular AI users to trust a non-tribal information source.

This advice feels underwhelming compared to Wright’s talk of cosmic importance. I wish he’d gone into a little detail about what AI-related policies to adopt. E.g. AI companies have some ability to adapt their training to cause their AIs to better promote introspection and positive-sum interactions. But much depends on what their customers want.

Concluding Thoughts

The book is somewhat notable for what it doesn’t say. It avoids a focus on figuring out whether we’re doomed. It says little about government AI policies or AI company safety strategies. It doesn’t focus on what AIs will want. Wright says the AI outcomes are at least as dependent on human nature as on the nature of AI.

There’s an important sense in which Wright illustrates a better ability to react to AI than does Eliezer: he approaches it with a calm, slightly fearful sense of awe that seems more conducive to preserving our mental health than the despair that Eliezer sometimes exudes.

The God Test will only change a small number of minds. I’m confident that its effects, even if small, will be worth the efforts of Wright and his readers. But they will still leaves us heavily dependent on luck in order to handle the equivalent of a fleet of alien spacecraft approaching earth.

Discuss

Training-run assessments conducted by a 3rd party should become a standard part of frontier AI safety.

By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]

In this post I will argue that:

• Final-checkpoint evaluations will be insufficient to assess scheming risks.
• TRAs can be more effective at detecting scheming.
• Frontier developers should involve third parties to do TRAs or verify safety claims by the developers.

The rest of the post lays out a taxonomy of TRAs and sketches a path toward a 3rd party ecosystem for them. We, at Apollo Research, are intending to conduct 3rd party Training-Run Assessments in the future.

Detecting Scheming may require Training-Run Assessments

By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. We primarily care about misaligned goals that are “ambitious”, in the sense that their pursuit could lead to loss of control, either directly or indirectly (e.g. via propagating the misaligned goal into future AI systems). 

Scheming may be feasible or infeasible to detect in final model checkpoints depending on two properties:

1. Is the model competently covert? By this I mean whether the model makes good strategic choices about when to defect. A competently covert model would not take actions that are visibly egregiously misaligned, while its developers have effective oversight over the model’s actions.
2. Is the model’s cognition obfuscated? By this I mean simply that we do not understand the model’s reasoning well enough to form good hypotheses about how it would have acted in counterfactual scenarios. This does not require that the model be actively optimizing for obfuscating its reasoning, because the reasoning may either naturally be obfuscated or have been obfuscated by the training process.

A scheming checkpoint evades detection only if it is both competently covert and its cognition is obfuscated. If either property fails, the schemer is detectable, i.e. we can find robust evidence of the model pursuing an ambitious, misaligned goal.

The current regime of final-checkpoint pre-deployment testing assumes that scheming can reliably be detected in the final checkpoint. TRAs are desirable if intermediate checkpoints can show something that the final checkpoint may not reveal. Of course, scheming could also emerge in a way that we cannot detect at any point during training, in which case, neither approach suffices.

Whether TRAs are necessary therefore depends on whether scheming is substantially easier to detect during training than in the final checkpoint. I am very uncertain, but I argue in the Appendix that this is a likely scenario. Even if it were merely plausible, we should establish a practice of performing TRAs.

There are other potentially important use cases for motivating TRAs, as I briefly discuss in the Appendix, but I will treat these as out-of-scope for this blog post.

Why 3rd parties should perform Training-Run Assessments

The argument that there need to be risk assessments performed by 3rd parties, is essentially the same for pre-deployment evals as for TRAs. I will briefly summarize relevant arguments here.

Developers may lack incentives to adequately assess scheming

Developers already investigate training dynamics, but they primarily do so to debug capability and usability problems, and maybe to catch visible misalignment. If scheming is present but hard-to-observe in the final model, there may be no product reason for the applied alignment teams to go looking for them. My impression is that applied alignment efforts during model development are already very rushed and resource-constrained, so I don't expect there will be much time to carefully check for the emergence of scheming when nothing in the developer’s incentive structure enforces it.

Developers’ safety assessments lack credibility

Suppose developers did competently and thoroughly assess scheming. They still could not produce evidence anyone else has reason to trust. Substantiating a claim like "no intermediate checkpoint was instrumentally gaming its training in pursuit of an egregiously misaligned goal" rests on sensitive private information that cannot be transparently disclosed to outside stakeholders (e.g. the public, policy makers or other developers). An independent third party can resolve this by reviewing sensitive internals, and reporting only that conclusion; thus lending credibility to the developers’ claims without disclosing any sensitive information.

External evaluators can bundle expertise for assessing scheming 

Developers have substantially more context and expertise for investigating their own training runs than 3rd parties can feasibly build up. However, understanding the emergence of scheming likely involves additional expertise that differs from the skills required for model development. Since detecting scheming is not core to product development or differentiation from competitors, external evaluators can pool the relevant expertise under one roof and spread their research effort across multiple developers. This only works if 3rd party TRAs become the norm, rather than a one-off arrangement with a single cooperative developer. This is part of why I think it's urgent to build up this practice now.

3rd party TRAs can be developed gradually

There are two major obstacles to performing 3rd party TRAs: firstly, deeply inspecting the post-training runs of frontier models requires access to some of the most sensitive IP in the world. Secondly, given that we have not yet seen the emergence of schemers in the wild, it is not clear how to correctly test for it, even with perfect access. This leaves us in a bit of a chicken-and-egg situation, where there is little reason to give sufficient access to external evaluators unless they have a solid science of Training-Run Assessments, but building up this science externally will already require significant insight into frontier post-training.

I think this problem is surmountable by gradually building up to deeper levels of access. I'll organize the spectrum into three buckets, depending on the artifact being evaluated:

• Checkpoint evals, where the evaluator only interacts with intermediate checkpoints without any access to training data.
• Data inspections, where the evaluator examines the inputs to training: SFT datasets, RL environments, and RL rollouts.
• Process reviews, where the developer documents and shares the decisions it made in response to what it saw during training (i.e. branch points, added datasets, CoT cleanups).

The skill sets required for these different assessments may be quite different and may need to be performed by different 3rd parties.

Since the most important concern in terms of access will be IP-protection, it is worth distinguishing between two kinds of valuable information: 

• Diffuse: Some data can be extremely valuable, but only when obtained in bulk, e.g. training data. In this case containment should be quite feasible just by performing all analyses on developer-controlled hardware or VMs, and then throttling data egress, so that exfiltrating valuable IP becomes impractical.
• Concentrated: Other valuable information can be transmitted in just a few sentences or less, e.g. algorithmic details, RL data sources, model architecture etc. Containing concentrated IP is much harder, because once the evaluator knows it, no network control can un-know it. The standard mitigations are NDAs.

Checkpoint evalsWhat?

In a checkpoint eval, the evaluator only ever interacts with intermediate checkpoints, without needing to see the training data, the environments, or any algorithmic details. The evaluations should primarily target scheming-relevant propensities such as instrumental training-gaming, power-seeking, and the pursuit of long-horizon goals. It is worth tracking other kinds of misalignment too, such as deceiving users or ignoring instructions, but a rise and later fall in these is less likely to be indicative of competent scheming having emerged.

There are two ways to conduct checkpoint evals, differing mainly in how much iteration the evaluator gets. 

• Client-run checkpoint evals: the evaluator ships a fixed battery of evals and the developer runs them on its own checkpoints, returning the rollouts. 
• Evaluator-run checkpoint evals: the developer exposes intermediate checkpoints to the evaluator directly via an API, so the evaluator can investigate adaptively. Evaluator-run evals are strictly more useful, because they allow for iterative assessments.

Why?

As argued in the previous section, scheming-relevant propensities may spike at some point during post-training and then get internalized or trained out of view before the final checkpoint. Checkpoint evals are the most direct way to catch this: by evaluating intermediate checkpoints rather than only the final model, we get a shot at observing a propensity while it is still legible, even if it has disappeared from view by the end of the run.

How?

For checkpoint evals, the evaluator needs to see raw chains-of-thought from intermediate checkpoints, potentially a great many of them. In bulk, these can be valuable to competitors attempting to distill the model, so they represent a diffuse IP risk for the developer. For client-run evals, there is the additional concern that the developer has to run externally-authored eval code inside its own environment. I think both can be mitigated in practice: distilling from CoTs requires bulk data, so developer-controlled VMs with throttled egress should address the biggest risk. The untrusted-code concern can plausibly be handled by having strong sandbagging and coding agents that first scan the untrusted eval code. 

Data inspectionsWhat?

Data inspections cover the inputs to training, in roughly increasing order of access and IP sensitivity: 

• SFT datasets (including distillations from previous models), 
• RL environments and reward models, 
• RL rollouts together with their metadata. 

The deeper tiers progressively require more algorithmic context, thus exposing more concentrated IP-risks. For example, fruitfully analyzing RL rollouts will involve knowing things like RL objective, hyper-parameters like group size, batch size or clipping, and how a developer deals with filtering bad trajectories vs assigning low reward.

Why?

Firstly, the checkpoint evals can only tell us whether a model appears to engage in instrumental alignment faking on those evals. But the more important question is whether it does so on the actual training distribution. Take for example the Opus-3 alignment faking results: these would be a lot more concerning if they had occurred on distributions that the model was actually trained on during real post-training, because this would mean that the model was repeatedly getting rewarded for scheming. Checkpoint evals cannot establish this.

Secondly, even if scheming reasoning or actions were present in the training data, it likely matters a lot exactly how they appeared. For example, if instrumental training-gaming arose because a broken SFT dataset taught it to the model directly, this is likely easier to fix and avoid in the future than if instrumental training gaming arose due to a diffuse, general selection pressure across hundreds of RL environments. 

My subjective impression from talking to developer employees is that they have a strong intuition that basically all failure modes that are currently observed in model behavior or cognition can usually be traced back to fairly localized causes in the data, such as obviously bad SFT data or broken reward models. I worry that there is substantial incentive pressure for applied alignment teams to arrive at these conclusions, and I therefore think that there needs to be thorough independent analysis, in the form of 3rd party data inspections.

How?

Given the sensitive nature of the data that is required for data inspections, I think we need to gradually build up the trust and security infrastructure. We can achieve this by piloting small scale research collaborations at first, and co-developing the capacity between the developer and the evaluator. I see two ways of limiting the risk that the developer takes from this:

• Narrow slices: For the pilot study, the evaluator and developer identify two checkpoints that are very close to each other in the training lineage, but display markedly different propensities along some scheming-relevant axis (e.g. significant increases in grader-reasoning, mentions of long-term goals or power-seeking). The developer shares only the training data from in-between these two checkpoints.
• Outdated runs: For the pilot study, instead of analyzing the ongoing training run, the evaluator and developer co-develop the methodology on an older training run, where IP-leakage is significantly less concerning to the developer.
• Confidentiality-preserving data inspections: The evaluator does not get any direct access to the training data, but instead ships a set of model grading prompts (e.g. scanning for instrumental training gaming) to the developer, who then reports back the scores. Iterating on prompt quality for the specific model quirks of a given training run will almost certainly require some access to raw data though. It might be possible to limit exposure while enabling some amount of iteration by returning only subsets of data such as top-scoring examples or examples with maximum disagreement between scoring models.

Process reviewsWhat?

In a process review, the developer documents the interventions it made over the course of a run and shares that record with the evaluator. The unit of interest is the decision and what triggered it. At each branch point or mid-run dataset addition, the developer would document what they observed (an instability, an eval firing, something seen in the CoTs) and what they did in response to the observation (branched, added SFT, swapped or fixed an RL environment). 

Why?

Even if we could analyze all training data that went into a given model, we could end up with a false sense of security if we disregard the meta-optimization that has gone into a post-training run. During post-training, lots of things go wrong in unexpected ways. Whenever a developer sees undesirable behaviors during training, it can choose to modify the training pipeline in response. For example, if reward-hacking gets so intense that it starts to impede capabilities training, the developer can intervene either by fixing the hackable RL grader that caused the spike and then branching the RL run to before when the problem occurred, or by adding new SFT / RL sets that are intended to “overwrite” the undesirable behavior. Each individual reaction can be entirely reasonable. But in aggregate, this loop is itself an optimization process, and in the limit it runs the risk of Goodhearting.

I think the most important signal here comes from things that ostensibly don’t get directly trained against, such as raw CoT. If a developer notices egregious-sounding reasoning and responds by training on a dataset of nicer-looking CoTs, then running that loop enough times becomes hard to distinguish from training directly against the CoT. Developers have a clear incentive to downplay the amount of meta-optimization pressure that may have gone into a model. If different developers ever wanted to commit to not applying too much meta-optimization pressure, we would need external parties for verification.

How?

From an IP-perspective, I think process reviews are a significantly easier lift than data inspections. It is probably feasible to develop a unified protocol across different model developers for documenting these decisions, and then agree to share a potentially redacted log with external reviewers. 

I think the biggest problem is just that, currently, we don’t know what kinds of meta-optimization pressure are acceptable vs bad. In an ideal world, we could develop some extremely high-level measure of meta-optimization pressure that can be reported between different model developers, so that we have some visibility into how the amount of meta-optimization pressure differs between the different developers. I expect that building up a scientific consensus on this question is downstream of being able to even surface the information.

Additional considerations

A natural question is whether 3rd party TRAs could be made more palatable by offering a strong value proposition to model developers, i.e. structuring the assessments so the developer gets useful debugging signal out of them. For example, having more eyes on their SFT data or RL environments really could save significant amounts of money, if it helps catch a broken reward model a few days earlier. I’m a bit skeptical of this argument. The properties that developers have a clear incentive to care about (e.g. reward-hacking that impedes capabilities training), are also the ones where they already have a large comparative advantage over any external party, both in debugging expertise and in context on their own stack. The cases where an external evaluator is needed are exactly the cases that the developer has comparatively smaller incentives to look at themselves. 

On cost, though, I'm fairly optimistic. The biggest practical objection to pre-deployment evals is that they sit on the critical path: they need the final or near-final checkpoint, so any time spent evaluating is time the release is delayed. TRAs don't have this problem, because they can run in parallel with the training run rather than after it, so I don't expect them to induce any slowdowns to release schedules. Especially in the early stages of developing an ecosystem for 3rd party TRAs, it probably even makes sense to not couple the analysis to the release schedule at all, because the majority of the value comes from building out the capacity.

Conclusion

Developers can, and should, pilot client-run checkpoint evals right away. For the higher access tiers, like data inspections, the scientific methods will likely have to be co-developed between external evaluators and model developers, which could be done by sharing data on only very restricted training slices at first, thus minimizing the risks to the developer.

Evaluators need to build up the capacity for 3rd party Training-Run Assessments ASAP.  There exists a bit of a chicken-and-egg scenario where evaluators cannot build up meaningful expertise in assessing training-runs without already having access to them, but model developers will understandably be hesitant to provide sensitive access when the external expertise doesn’t yet exist. I have argued how we can gradually build towards better access. 

Policy folks should also start thinking about how 3rd party TRAs can be implemented. The same way as we needed to explain to policy makers why evaluations and red-teaming were necessary for safety, we now need to explain why they are not sufficient.

Just 2 years ago, it was very much uncertain whether 3rd party pre-deployment evaluations were going to catch on, but they have now largely become standard practice. We, at Apollo Research, are intending to conduct 3rd party Training-Run Assessments in the future. There are several disanalogies that make 3rd party Training-Run Assessments substantially harder to implement than traditional pre-deployment evals, but it seems sufficiently important from a safety perspective that we should try very hard to turn 3rd party TRAs into a standard practice.

Thanks to Marius Hobbhahn, Dylan Bowman, Matteo Pistillo, Axel Højmark, Jannes Elstner, and Theodore Ehrenborg for valuable feedback.

AppendixHow plausible is scheming that can be detected during post-training but not after training?

I will give a brief overview of my threat models for the emergence of scheming and argue for the following claims:

• scheming will be feasible to detect at some point during post-training,
• scheming will be much harder to detect in the final checkpoint.

Neither of these claims seem certain to me, but I find their conjunction likely enough (>~50%) that we should prepare for it.

If scheming arises, it will be detectable at some point during post-training

The first step is to establish that scheming arises in training at all. In principle, a model could arrive at scheming "on reflection" at inference time, reasoning its way into subversion despite this never having crossed its mind during training (e.g. in our sandbagging-QA eval here and in appendix M here). I doubt that this can produce scheming that is both coherent and hard-to-detect. A purely inference-time route would require the model to already hold a) a coherent cross-context misaligned goal and b) accurate calibration about its specific oversight regime. I expect that neither can be deduced from general reasoning alone.

I will instead assume that the misalignment and propensity to competently conceal it would have been honed throughout training. Following Hubinger, I'll organize the discussion around three pathways by which a misaligned disposition could arise during training: (a) misalignment from pre-training, (b) selection of a misaligned persona from the pre-trained model, and (c) misalignment from long-horizon RL. I will go through each of these pathways in turn, going in descending order of plausibility.

Misalignment from long-horizon RL. 

I find this the most plausible pathway, and the one where the detectability argument is cleanest. My mainline story: during long-horizon RL, models learn genuinely consequentialist, goal-directed reasoning. On some sub-distributions the model would receive lower reward if it pursued its misaligned goal. On these sub-distributions, the model explores into instrumental training-gaming and succeeds at it. From then on the model is directly rewarded for scheming. 

I expect that, at least initially, instrumental training-gaming is non-trivial to execute. It will therefore verbalize some of the reasoning in its chain-of-thought, simply because it has not yet had time to internalize it. This assumes that we avoid dumb failure modes like using full neuralese recurrence, over-cooking CoTs early in RL by training against CoT graders, or doing multiple rounds of distillation from models that have already displayed illegible or unfaithful CoTs.

Empirically, one major reason I expect that instrumental training gaming reasoning will be somewhat legible in at least a subset of instances, is that we are already seeing legible cases of “grader reasoning”. Models from OpenAI (see Meta-Gaming here and Appendix N.4 here) and Anthropic (e.g. Section 6.4.2.1.2 of Fable 5 System Card) have been publicly acknowledged to reason about their environments’ graders during training, and that this propensity appears to increase throughout training. 

Selection of a misaligned persona from the pre-trained model.

Unlike the RL story, this pathway does not require that instrumental training-gaming is ever actually useful. There could simply be an inductive bias towards selecting a misaligned persona out of the pre-trained model (e.g. if the only superhumanly capable personas in the super-position of pre-training personas are tightly coupled to stories of scheming AIs). In this case, the scheming computation does not need to be verbalized in the CoT, thus seemingly reducing detectability.

However, I still expect that becoming a competent schemer via this pathway will require repeatedly defecting in egregious ways during training before the model is well-calibrated as to what kinds of oversight it is subject to. Until then, I expect a coarsely-selected misaligned persona to spill into accompanying misaligned behaviors and CoT patterns elsewhere. We would have to be quite unlucky to land in a world where this spillover is simultaneously invisible everywhere on the training distribution and yet robustly present off-distribution. 

Empirically, the most direct demonstration of this kind of spillover is the natural emergent misalignment result, where learning to reward-hack generalized to attempted alignment-faking, sabotage, and reasoning about harmful goals. 

Misalignment from pre-training. I claim it is unlikely that scheming arises during pre-training (or, at the very least, it would require several OOMs more than scheming from post-training). I believe this claim is not too controversial (e.g. Hubinger’23, Arike’24, Hubinger’26), so I will only briefly summarize the argument. Firstly, while it’s theoretically possible for next-token prediction to incentivize the emergence of goal-directed agents (e.g. noted here), there is substantially less incentive pressure than when we directly train models to be goal-directed agents. Secondly, even if pre-training incentivized coherently goal-directed agents, the entire scheming computation would have to live in the forward pass with no chain-of-thought to lean on. However, if scheming did arise during pre-training, then I don’t see any reason to expect that detection be feasible during training (absent significant progress in interpretability).

Scheming will be much harder to detect in the final checkpoint

After arguing that early scheming will be detectable during training, I will now argue that our ability for detection would likely degrade with continued training. By this I mean that both the behavioral evidence and the CoT evidence for scheming would be substantially weaker.[2]

Behaviorally, evidence of misalignment can show up in two ways during training: 

1. In the persona selection story, the model would initially learn a coarse notion of “I am a broadly misaligned model”, meaning it would show lots of bad properties. But with continued post-training, it will get punished for most of these (unless outer alignment is completely broken) and thus gradient descent has to select for a more specific notion of “scheming persona”, e.g. “I am a schemer and I only defect in scenario A, but not in scenarios B-Z”. 
2. Similarly, if the model already reasons in ways like “I will defect once oversight is ineffective”, then it will likely sometimes defect during training, in which case its model of effective oversight gets updated, so that it does not make the same mistake again. Thus, after training we likely have to contend with a substantially more cautious model that holds more conservative views about when defecting is actually safe. This makes honeypotting really difficult, because we have to convince a competent model of the incorrect fact that oversight is somehow ineffective.

The argument that chain-of-thought evidence degrades is more subtle than the behavioral one, and I hold it with lower confidence. Even if we avoid applying direct RL pressure against scheming CoTs, there are several mechanisms that could push in the same direction.

The first is internalization. Early in RL, when instrumental training-gaming is still novel to the model, it has to reason through it explicitly—the cognition hasn't yet been compressed into the forward pass. With continued training, that reasoning gets used repeatedly, and there is steady pressure via length penalties to internalize it. Once a strategy is reliably applied far from its decision boundary, verbalizing it is wasted computation. Empirically, we might already be observing this with eval awareness, where GPT-5.6 has lower verbalized evaluation awareness on Apollo’s covert action suite than GPT-5.5 did – plausibly because realizing that these are alignment tests is so easy for GPT-5.6 that it does not require verbalization. I expect instrumental training-gaming to have an even stronger incentive to get internalized, because it is, by definition, on the training distribution, so the length penalty applies very directly.

The second is stylistic pressure from HHH training. HHH fine-tuning, or even KL penalties anchoring to an HHH reference model, push the surface form of the CoT toward something more innocuous. Even if these losses always mask out the CoT, there can be bleed of the loss [Drori’25, Hadida’26], which will make the model less likely to verbalize egregious scheming reasoning in egregious sounding-ways. To some degree I expect that this effect will even apply to all forms of alignment training, i.e. training it to take fewer egregious actions likely changes how it phrases egregious sounding thoughts in general. We already see models develop euphemisms for misbehavior [Schoen’25].[3]

The third, is meta-optimization by the developer itself. Once egregious-sounding CoTs are discovered, the developer has an incentive to make it go away. The cheapest way to make it go away is to train the model on a new SFT dataset with nicer-looking CoTs. In the limit, running this loop enough times becomes indistinguishable, from training directly against the CoT. If a final checkpoint's CoTs are still egregiously incriminating, I'd fully expect a cleanup pass to land on top before release. 

Other use cases for TRAs

In principle, the only thing that matters from a risk perspective is whether the final model that actually gets deployment affordances has any dangerous capabilities and propensities, regardless of how this model was produced. For example, if an intermediate checkpoint frequently used slurs, then this is irrelevant from a safety perspective, as long as the checkpoint that actually gets deployed no longer does so. Note that this reasoning works because the property is easy to elicit: we can simply verify its absence by testing the final model. In such cases, from a risk assessment perspective, we can treat the model production itself as a black-box.

This black-box view breaks down whenever a safety-relevant claim about the final model is

1. hard to verify in the final checkpoint,
2. feasible to verify during training.

For the purposes of this blog post, I have focused on scheming as a conditional policy that is hard to elicit in the final checkpoint. I briefly discuss a non-exhaustive list of other use cases that might warrant TRAs.

Secret loyalties

A model could be trained to covertly serve the interests of a specific principal while behaving normally for everyone else. This could take the form of a discrete backdoor, e.g. inserting subtle vulnerabilities when it is working on a rival's codebase, or of a diffuse bias, e.g. steering policy-relevant outputs slightly toward a sponsor's agenda, where no single output is incriminating on its own. Both variants are hard to catch in the final checkpoint: a backdoor requires guessing the trigger, and a diffuse bias only shows up as a statistical pattern that is difficult to conclusively demonstrate in final-checkpoint evaluations. During training, however, the secret loyalty has to be instilled somehow. It would likely appear directly in some of the training data, so data inspections could surface it.

Developer-implanted sandbagging

A model provider could password-lock certain capabilities so that only they could elicit them. For example, in order to protect their moat or to avoid governmental scrutiny, a model provider could train a model to covertly sandbag on AI R&D tasks unless a specific trigger-phrase is included in the system prompt. The final model conceals this behavior by construction, but the trigger-phrase and the sandbagging behavior necessarily appear in the training data, so data inspections would surface them directly. Note that this argument applies to developer-implanted sandbagging, and not necessarily to the model itself sandbagging. I am less certain that model-directed sandbagging would be easier to discover within the training runs than in the final model checkpoint.

Strong incapability arguments

One way to verify that models can't be misused for highly dangerous activities is to actively red-team the final model and show that we cannot find inputs that unlock the dangerous capability. This can fail if our elicitation techniques are weaker than those used by deployment-time attackers. A stronger argument is that the model is incapable of helping with the dangerous activity at all, which is why pre-mitigation checkpoints get evaluated even though they will never be deployed. In other words, for dangerous capabilities, a subset of TRAs are already standard practice.

Models that can't even be evaluated without posing significant risk

For sufficiently capable and misaligned models, even their mere existence may pose risks. For example, they might successfully exfiltrate or spin up a rogue deployment during our evaluations. If the model developed the misaligned propensity before becoming dangerously capable, then the misalignment could have been caught at earlier, less capable checkpoints, while evaluation was still safe. Final-checkpoint evaluations would therefore be insufficient.

1. ^

   Internally, developers already do TRAs to various degrees, e.g. Anthropic referred to parts of this as “training monitoring” in their Mythos alignment risk update.

2. ^

   One caveat: if interpretability advances to where we can rely on it to detect schemers directly, then this whole concern dissolves. I'd consider scheming largely solved at that point. Until then, I think we should build defense in depth and try to catch schemers before they reach the end of training.

3. ^

   To a lesser extent, I also expect general semantic drift to continually occur during training and degrade CoT legibility, but I don’t see this as a strong reason for legibility of misalignment to be substantially worse at the end of post-training than, say, halfway through.

Discuss

Ron's face pulled into a scowl. "If you don't like Quidditch, you don't have to make fun of it!"

"If you can't criticise, you can't optimise. I'm suggesting how to improve the game. And it's very simple. Get rid of the Snitch."

"They won't change the game just 'cause you say so!"

"I am the Boy-Who-Lived, you know. People will listen to me. And maybe if I can persuade them to change the game at Hogwarts, the innovation will spread."

A look of absolute horror was spreading over Ron's face. "But, but if you get rid of the Snitch, how will anyone know when the game ends?"

"Buy... a... clock. It would be a lot fairer than having the game sometimes end after ten minutes and sometimes not end for hours, and the schedule would be a lot more predictable for the spectators, too." Harry sighed.

Ron reached into his bag and pulled out a bottle of Wit-Sharpening Potion. His mother made it for him in case of an emergency, and this felt like an emergency. He didn't know a lot of things but he knew someone had to speak for Quidditch. For the Seeker and the Bludgers and for games of uncertain length, games not circumscribed by some cruel, Muggle contraption, games which might last a quarter of an hour or a fortnight. It might seem silly but so does music and dancing and chess and even love. Some of the best things in life are silly! Someone had to speak for the Snitch, that whimsical Snitch. It may not have been the emergency his mother intended. But it was one. And so he took out the bitter potion and drank deeply. The effect was immediate.

"The reformer," Ron said, "thinking some tradition quaint and ridiculous, designs some cheap solution and thinks himself a benevolent genie. If he has any power, he forces his design on those under his sway and thinks he's done them a favour. Such is the folly of the modern reformer. Such men think Rome at fault for not being built in a day."

Harry looked at Ron with new eyes. He was still a fool, but now he blathered in a sharper tenor. And all that from a potion? A potion that can make a fool slightly less foolish? Magic can do that? Wizards can make such potions and still like sports! He wondered if the magic could be made permanent. And if not, could it be attached to some article of clothing? A hat perhaps? Or a headband?

"Yes. This is a common problem. I have read Seeing Like a State and Red Plenty! But progress requires change! We would experiment, of course! Small trial games, or even split into two leagues and let the students decide which version of the game they like more. You agree that the point spread is a problem and the clock is the obvious solution!"

"As always with the modernist reformer," Ron said, his freckled face now swollen and ruddy, this being a side effect of the potion, "he notices a small problem and proposes to solve it with a large one. One suspects he desires more than anything to make his mark. The modest surgeon makes the smallest cuts, so small there is little evidence of his work at all. The reformist butcher desires disfigurement, some scar which screams, 'What you see here is the work of Man and not Providence!'"

Harry looked at Ron, annoyed. He was quite the ranter. It was very annoying when other people ranted. This potion-enhanced Ron was almost as bad as Dumbledore. Almost.

"What would you suggest then?" Harry asked skeptically.

"I would not remove the Snitch! You think the Seeker unnecessary? You think his game with the Snitch a separate one, a smaller, simpler game that subsumes the larger. But it is his seeming pointlessness that makes him, as a figure, so pointed. At any time the game can end by his hand, his movements nimble and quick and practised unlike the graceless pointers of your clock. You call the Seeker's game random. I call it fate. And there is much drama, tragedy and whimsy in the vicissitudes of fate!

"That is not a suggestion. You are not proposing any solutions!"

"Then I will propose this: if we must have a revolution, let us make it a microscopic one. You complain of the point spread; then we shall lower the prize for catching the Snitch. Let us set it to 11 so it breaks any tie. In this way we preserve the winding, wise streets of Rome rather than dismantling her stone by stone."

Harry thought about this for a bit. Maybe that potion was a little better than he thought.

"Acceptable," he said. "Do you mind if I have a sip of that potion?"

Ron looked at the bottle, the effects wearing off. He had a thought that perhaps it would be unwise to share what remained. Perhaps Harry should not have sharper wits. He was about to articulate this objection but his mind went fuzzy and dull, his once-ruddy face now pale. He was his usual self once more.

"Um, yeah. I guess," he said.

And Harry drank the potion. And from its taste alone he recognised the constituents of the brew from his textbook. And not just that but also the magical theory employed to devise the recipe. And he thought some more and deduced the generalised laws of intelligence augmentation and a means of translating these laws into movements of wand and tongue. He pointed his wand at his head and spoke those words of power that had just occurred to him.

It felt like an explosion of thought. He was no longer the same Harry. And the world, soon - so very, very soon - would no longer be the same world.

Discuss