Сборщик RSS-лент

I want to document where LLMs are at, today, partly because I don't know of any single place that brings all these examples together. I think a lot of the examples below will be surprising and unfamiliar (although maybe not to this particular audience).

Part of this is just trying to build a resource for raising the "sanity waterline". Partly it just felt like an important foundation for other ideas I want to convey.

Across Domains

Four years ago we were laughing about how AI couldn't draw a human with the right number of fingers. The state of the art can generate accurate QR codes. The most likely outcome is that LLMs continue to improve; there are numerous improvements already being developed, and no sign of things slowing down.

(Source)

Language: For casual purposes, Google Translate is amazing. Twitter is now automatically translating a lot of tweets between Japanese and English, allowing two cultures to suddenly connect on a unique new level. (Example). Maybe it's not professional grade, maybe it's not perfect, but I routinely hold conversations with people who are speaking other languages.

Customer Service: I've started seeing a few of my calls already being answered by LLMs. They can handle real time voice and follow scripted conversation paths fairly easily - although they still need to be locked down, or else they can tricked into giving out invalid discounts (Source)

Search & Research: Two sides of the same coin, as the etymology implies. AI is great at understanding plain language searches, whereas previous technologies were limited to keyword searches - TikTok doesn't understand what "unalive" means but an LLM understands it trivially. You still need to verify the documents manually - they can't do your reading for you. But even if a few of the sources they find are hallucinated, that often still leaves you with a treasure trove of obscure references you couldn't have easily found manually.

Art: AI Art can pass a sort of Turing Test. Humans can’t actually distinguish a lot of LLM art - just the overdone "slop" styles. (Source). Even if you hate AI art, numerous studios say they're finding it great for quick sketches and proposals - it helps speed up and tighten communication between artists and art managers. (Citation)

Writing: LLMs have won a few minor literary and artistic prizes. I think this reflects more on the judges being unaware of what LLM art looks like, but that just reiterates the importance of paying attention. (Art source) (Short story scandal)

Math: LLMs have solved multiple challenging, long-standing mathematics problems. (Source). They're also seeing a lot of use for sweeping through a body of less notable work that no one has ever had the time to verify or solve (Source). The most recent paper is from July 10th, 2026 - AI has now solved a famous open problem in graph theory, using an off-the-shelf model, within a day of the model's public release (Source)

Coding: I think most people are already aware of this, but agents like Claude Code and Codex saw an explosion in usage over the Christmas Holiday in 2025, as numerous developers finally discovered what modern models could do (Citation).

Last year, twitter user Psyho was the only human to beat AI in the AWTF competition. In July 2026, OpenAI "crushed humans", and "this is essentially the first time AI won vs humans in a programming competition in such a decisive matter" (Source).

Cyber-Security: Slightly newer is Claude Mythos showing a 20x improvement in the ability of security teams to find software vulnerabilities, including numerous high severity issues. (detailed report from the Firefox team) (Epoch graphs, giving a broader view across 17 major vendors and 4 major open source programs)

Prediction: AI slightly exceeds the "wisdom of the crowds" and is on trend to out-perform even super-forecasters like Nate Silver. (Source). Per the article, some companies might already have private tools that can reliably gain at least a small edge in the stock market.

Persuasion: A recent surprise, this study found AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses. (Citation)

A note on that last one: sure, you'd never fall for obvious AI propaganda. But a lot of other people will - we're already seeing it in Advertising and Politics. (Citation)

The ability to produce a thousand social media accounts parroting any talking point you want - but without a clear, repetitive script to give away the game. In fact, the ability to reply to every single comment they make, ensuring they're the most engaging opponent and have the maximum chance to win over skeptics via dialogue. (Citation)

The Early Days

You'll notice this is a mixed bag, ranging from "still occasionally fails at Customer Service so bad that articles get written" and "almost, but not quite, on par with experts, when measured in certain very narrow and specific ways."

LLMs are still very young. I think we're currently in something like the AOL days of the internet, before anyone figured out how to build behemoths like Amazon and Netflix.

I do think we're in for much bigger changes, but that's a topic for another post.

A Note on "Bubbles"

There will probably be a bubble popping!

There's a lot of money going into this industry, and it's hitting before the technology is actually ready. We saw this in the DotCom crash - Pets.com wasn't wrong to try and deliver cat food, they just didn't have the logistical engine that Amazon.com later invented.

Bubbles pop because investors' imagination exceeds current realities.

But eventually reality catches up.

The internet bubble popped. If you're reading this, you're probably aware the internet did not vanish.

Some companies are jumping the gun, trying to do what is presently impossible. But a lot of them don't actually care whether it works today - they're building up their workforce's skill in "LLM usage" in anticipation of next year's improvements.

Postscript

I'd love to hear about additional examples, especially ones that show progress in domains I didn't list, or that substantially surpass the current examples.

Discuss

This post introduces a tool: an Epistemic Audit for Existential Risks from AI. It is a structured way to map, organize and track your beliefs across the key domains and questions that determine how likely existential risks[1] from AI are. It follows the causal chain from "capable systems get built" to "existential risk." It is designed for repeated self-assessment, creating a longitudinal record while enabling standardized comparisons and aggregation across respondents for (potential) future elicitation studies[2]. It has three main functions:

1. Audit yourself. For every question, note your epistemic uncertainty[3]: not what you believe, but how much you trust what you believe and where more information and evidence can move or counter your beliefs. You will find out where your overall view on existential risk rests on things you have actually thought through, where it rests on takes you have absorbed from others, or places you have spent little time on in general.
2. Route your effort. The questions where your uncertainty is high and which would move your overall view are exactly the ones you should study next. Where the entire field's uncertainty is high, these questions can then turn into a direction for a research project, literature review, or a blog post. Framing your beliefs as measurable and falsifiable helps clarify what evidence is most compelling, and what would change your mind, making learning more rigorous, transparent and cumulative.
3. Structure discussion. By giving everyone the same numbered framework and the same three-point uncertainty scale, the tool creates a shared language for discussing cruxes rather than conclusions (e.g., "I seriously doubt 1.4, what is the most compelling evidence for or against it?"). This focuses discussion on the key assumptions, highlights where uncertainty and disagreement lie, promotes evidence-sharing and makes views easier to compare and aggregate across individuals.

I am not the first to decompose existential risks from AI. Joe Carlsmith's report decomposes the power-seeking threat model into six conditional premises and assigns credences to each. Zvi's Crux List is a (large) collection of questions on which people disagree. AGI Ruin: A List of Lethalities is the (in)famous list of reasons the problem is hard. I have not yet seen a protocol that you can run on yourself, with a defined scale, repeat over time to see whether your uncertainty is actually decreasing, and that makes it easier for you to guide your effort towards the questions that matter the most.

That is what I attempt in this post.[4] I created this Epistemic Audit tool using standardized categories (low/medium/high uncertainty and falling/stable/rising confidence) to produce simple, comparable assessments across respondents. Hopefully this reduces reporting ambiguity while enabling aggregation and longitudinal analysis.

One note on scope: a question makes this list only if resolving it would noticeably shift an estimate of existential risk from AI. There are many interesting questions that don't do this, such as "what is intelligence?"; I have attempted to exclude these.

The domains below follow the causal chain that I think of for AI to become an existential risk: capable systems get built, they are agentic, we fail to align them, they get deployed anyway, and the failure leads to existential risks. I also discuss routes that bypass misalignment entirely, the mitigations, and finally the question of how to reason about any of this at all.[5] 

How to run the audit

1. Open the sheets and make a copy  https://docs.google.com/spreadsheets/d/1e5-52-bjyOtsKnMmQDwkyyI8bFyOZF5AQxyGO1N3Ook/edit?usp=sharing 
2. Begin with your Quick Summary (~5 minutes).[6] Rate your uncertainty as low, medium, or high. This will produce a radar chart, drawing your uncertainty shape (see mine in the comments, I encourage people to share it there as well). So that your ratings mean the same thing at your next audit (and comparable to everyone else's), use these anchors[7]:
   • Low: your view is resilient. You would be surprised if a quarter of a year of new evidence meaningfully moved it.
   • Medium: a single strong paper, result, or event could noticeably move you.
   • High: you don't trust your current view. One good argument could flip it.
3. Note the direction: has your uncertainty been rising, stable, or falling over the past year? (On your first run this is retrospective memory, which is rather unreliable, so the direction column only becomes trustworthy from your second measurement onward. That is fine; the first run is your baseline.)
4. Deep dive. Go through each domain, and mark your cruxes: identify the questions where a changed belief would substantially move your overall view on existential risks from AI. These are the ones worth your attention; the rest you can rate quickly and move on.
5.  Schedule your next audit. The value of this audit is meant to increase over time.  So, commit to a date to come back to your google sheets template. The first measurement will tell you where you stand. Two or more will formalise how your uncertainty is changing, and inform where you can make progress. Personally, I’ll (try to) re-run this audit every three months and compare.
6. Make your uncertainty productive! Your high-uncertainty cruxes point to your greatest opportunities for learning. The questions in this audit are meant to steer you toward unresolved links in the causal chain leading to existential risk. I believe that further specifications in these questions can uncover promising research directions.

The questions are numbered so it's easier to refer to them personally or in the comments. My personal scoreboard with reflections is in the comments (and in the Google Sheets as an example).[8]

1. Capabilities and timelines

The elephant in the room in any conversation about AI is what these systems can do and how fast that changes. What matters here is capabilities, so what systems can actually do, economically and strategically, and the trajectory they are on.[9]

1. Are there  capabilities that cannot emerge with scale alone (more compute/data/params), regardless of how far scaling continues?  
2. Which capabilities are mostly discontinuous (sharp jumps) rather than smooth and predictable?
3. How far above the human range do ceilings on AI capabilities in various domains sit, if there even are ceilings? Is human-level capability a natural plateau, or just an arbitrary point that the curve passes through?
4. Does the intelligence of current AI systems differ from human intelligence in ways that matter for forecasting? One concrete example: do models genuinely extrapolate beyond their training distribution, or only interpolate within it?
5. Current AI systems cannot yet learn as continually and efficiently as humans. How hard is adding this—persistent memory, no catastrophic forgetting, higher sample efficiency—to current systems?
6. If the current paradigm plateaus: what is missing, how long will it take, and does it require a (radically) different approach rather than an LLM successor?
7. If it does not plateau: how long until AI systems are as economically and strategically capable as an average human across all domains?
8. And how long until they are far beyond any human across all (relevant) domains?
9. Will capability keep following well-known scaling laws?
10. How will compute grow over the coming years (chips, energy, capital willing to be spent)?
11. How will data grow, both in quantity and quality?
12. How will algorithmic efficiency improve over the coming years?
13. Do benchmark gains translate into real-world economic and strategic capability, or is there a persistent gap?

2. Agency and goals

If we grant capable systems, we need those systems to be coherent pursuers of goals, i.e., agents.

1. Do trained systems become coherent goal-pursuers at all, or do they remain tool-like and very context-dependent even at high capability?
2. Are intelligence and final goals orthogonal axes on which agents can vary, both in principle and in practice for systems produced by current deep learning methods?
3. Under what conditions do agents optimizing for final goals develop instrumental subgoals dangerous to humanity (examples: self-preservation, resource acquisition, power-seeking)?
4. Which goals do trained agents actually end up with? How close are these to the training objective and how alien are they when they diverge?
5. When do systems become situationally aware, i.e., knowing they are models being trained and evaluated, and what behavior follows from that?
6. Do multi-agent settings lead to safer or less safe worlds than single-agent settings? How quickly, and in which environments, is defection/cooperation the default? (For whether many AIs make takeover harder or easier, see 5.5.)
7. When can models coordinate effectively between many instances of themselves and what do capabilities look like at that point?

3. The difficulty of alignment

Conditional on capable, agentic systems: do we fail to align them despite trying?

1. Is alignment hard in the strong sense of having deep theoretical obstacles or is it an engineering problem that yields to iteration, the way most engineering problems do?
2. Do current alignment techniques such as RLHF and its descendants actually instill values or rather shape surface behavior while leaving the underlying system untouched?
3. Do learned values generalize out of distribution as capability grows?
4. How likely is deceptive alignment by default, i.e., training processes producing models that act aligned while observed and pursue other goals when not, and why does it happen?
5. Will interpretability help us understand truly important internals of frontier models before the point where it's too late?
6. Can weaker overseers reliably supervise stronger systems in the limit to superintelligence?
7. Can we build corrigible systems, i.e., ones that allow correction and shutdown without an incentive to resist?
8. Suppose we can align a system to some target: to what, and to whose values, should it be aligned? Is intent alignment, i.e., the system doing what its developer means, enough or does it need to be value specification? How different are these two?
9. Can AI automate alignment research fast enough, i.e., sufficiently solve the alignment problem before the point where it's too late?
10. How many warning shots do we get? (How society responds to them is 6.12.)
11. Do alignment failures show up in small models or emerge only seriously at capability levels where it's too late?

4. Takeoff and self-improvement

How fast the transition happens between different capabilities matters a lot. A slow takeoff gives society time to react and iterate whereas a fast one does not (though the former may bring different risks). This is my most cruxy domain.

1. Can AI meaningfully accelerate AI research itself, and by how much?
2. What will be the main bottleneck to recursive self-improvement? Compute, physical-world experiments, something else, everything roughly uniformly?
3. How much do physical constraints such as datacenter construction, energy, chip manufacturing, slow the conversion of capability gains?
4. Should we expect capability growth to be exponential, sigmoid, superexponential, something else?
5. Does the transition to decisively superhuman systems play out over minutes, hours, days, years, or decades?
6. Is takeoff concentrated in one lab or distributed across many comparable actors?
7. How large is the leading actor's lead time, and is that gap widening or shrinking? How will this change with recursive self-improvement?

5. From misalignment to catastrophe

A misaligned agentic superintelligence doesn't have to be a catastrophe. There need to be concrete mechanisms by which this leads to existential risks for humanity.

1. What are the concrete pathways from misaligned capable AI to existential risks, e.g., sufficiently strong engineered pandemics, nanotechnology, cyberattacks on infrastructure?
2. How much does the robotics bottleneck constrain a system without (current) physical actuators? Is it necessary at all?
3. How dependent will critical infrastructure and high-stakes decision-making be on AI systems by the time it really matters, and how well-defended will that infrastructure be (e.g., by national AI safety and security institutes working together)?
4. Can a misaligned system acquire resources and power gradually without being detected and stopped?
5. Does a world of many AIs make takeover harder (through checking each other) or easier (because there is far more noise)?

6. Deployment, incentives, and coordination

How current systems are deployed, what kind of incentives are present in the AI race, and how countries and frontier AI companies will (not) coordinate, matter immensely.

1. How difficult is coordination between AI companies? What is needed for this to happen?
2. How difficult is coordination between the relevant countries in the AI race?
3. How likely is it that AI companies will become nationalized, e.g., a Manhattan project for AI?
4. Conditional on the nationalization, how does this affect the race and overall dynamics?
5. Conditional on there being agreement, how effective can verification mechanisms be?
6. How much do competitive pressures mess up safety commitments or coordination efforts? How have frontier companies acted previously in this regard and what does that mean for the future?
7. How fast will AI adoption be and in what form?
8. What economic effects will AI cause and how do those feedback into development, e.g., investment cycles, automation-driven demand, political pressure?
9. What will the dominant public sentiment toward AI be and what causal role does this sentiment play?
10. How large is the alignment tax, i.e., how much capability or speed do safety measures cost? What is needed for frontier companies and countries to take this tax?
11. How much legislation will happen, in what form, and how much binds to frontier development?
12. If we see warning shots, i.e., small catastrophes but not existential, how will society respond to them?
13. How does the open-sourcing of increasingly capable model weights change every answer above?

7. Misuse

I personally think most about the above threat model, so acutely losing control of ASI, but existential risk does not require this. We can also have sufficiently capable (aligned) AI systems in the wrong human hands.

1. Can AI-enabled biological weapons be existential rather than "merely" catastrophic?
2. Can AI-enabled cyberattacks on critical infrastructure cascade to civilizational scale and how does this affect misuse or misalignment scenarios?
3. What role will AI play in autonomous weapons and in nuclear command and control?
4. How will we decide who gets hold of these sufficiently capable (aligned) AI systems?
5. Does AI advantage attackers or defenders more, especially in bio and cyber?
6. Which arrives first, existential risks through misuse or misalignment? How does preparing for one help or hurt preparing for the other?

8. Structural risks

There are some other scenarios which I would count as existential risks from AI that don't quite map neatly onto the above two routes. I'd describe them as "we lose but it wasn't because we built ASI that was misaligned or someone majorly misused it."

1. How plausible is gradual disempowerment, i.e., can humanity lose effective control through incremental delegation of decisions to AI with no discrete takeover moment ever occurring (and this actually being very bad, instead of a future where we choose to do so and it being good)?
2. Does AI enable stable, permanent concentration of power, i.e., a global authoritarianism that can never fall?
3. Does AI-driven persuasion and epistemic pollution degrade our collective decision/sense-making enough to break our ability to respond to every other risk on this list?
4. Does the AI race itself raise the likelihood of a nuclear war?
5. Could early AI actors lock in their values permanently?
6. Do AI systems themselves acquire moral status, and if so, does a future that mistreats them at scale count as a catastrophe, e.g., similar to the mass animal farming we do now?
7. If aligned ASI manages to "solve everything," are there ways in which the complete abundance and lack of need to do anything lead to a complete loss of meaning, in turn posing an existential risk to humanity?
8. Are there plausible outcomes involving s-risks, i.e., astronomical suffering, that are worse than extinction or other existential risks and how do these play out?

9. Mitigations

There are a number of people already attempting to mitigate existential risks from AI, myself included.

1. What have existing mitigation efforts changed so far?
2. Is technical safety research progressing faster or slower than capabilities? How could this be changed?
3. What is the conversion rate from governance research to real, binding policy? How could this be changed?
4. Is compute a viable control point for AI governance? For how long does this window stay open?
5. How much worse or better would the world be if frontier model weights were stolen or leaked? How good is security today?
6. How far can AI control go, i.e., extracting useful and safe work from possibly-misaligned systems through monitoring, containment, and restricted permissions as a complement to aligning them?
7. What have been (un)successful attempts at tying evaluations on dangerous capabilities to if-then commitments? What is needed to trigger real action instead of companies dissolving/changing them if they get reached?
8. What is needed for pausing or slowing frontier development? Is it net-positive once you account for compute overhang and which players keep racing?
9. Which audiences matter most for education and field-building, e.g., general public, policymakers, researchers?
10. How much does strengthening society at large help, e.g., hardening biosecurity and cyber-defense independent of any AI control, or strengthening democracies?

10. How to reason about any of this

The main question here is how we form beliefs about an event that has never happened before. When thinking about existential risks, I think it's most valuable when thoroughly grounded in empirics, and since AI-driven existential risks have never played out, our empirics need to come from reference classes that can guide us through this.

1. Which reference classes are informative in our situation? Humans driving other species extinct, past technological transitions, invasive species, corporations and other principal-agent problems, the long track record of failed doom predictions from humanity? What does each really mean?
   1. How often have humans been an existential risk to other species, and was it mostly negligence or intent?
   2. On what timescale did human-caused extinctions play out? Centuries, decades, years?
   3. How much was raw intelligence the determining factor, versus culture and coordination?
   4. How often are species-level extinctions driven by another species at all, compared to environmental causes?
2. What is the track record of long-range technology forecasting? Specifically interesting is doing a meta-analysis on the predictions from the "AI safety community."
3. How should the enormous spread in "expert estimates", which can range from well under 1% to above 99%, inform us?
4. When inside-view models and outside-view base rates conflict, how much weight should each get?

What to do next

If you ran the audit: your high-uncertainty cruxes are your curriculum. Pick the top three and go deep: read the strongest thing written on each side, or write the post that maps the question if none exists. If you dig into a question and find the field's uncertainty is as high as yours, you have found a research project, and those are exactly the projects the field needs. If you are in the Netherlands and want help turning one of these questions into an actual project, Safe AI Netherlands exists for precisely this: reach out.

I will (hopefully) fully run this audit in the coming month, and then re-run it three months later and publish the difference. I’ll also start posting my object-level beliefs on various questions, which are missing from this first measurement. I encourage you to do the same.

Acknowledgements

Many thanks to Ana Paula Castillo Rodriguez for fully designing and making the accompanying Google Sheets, besides providing extensive feedback. Thanks to Lucas Hogendoorn, Ilija Lichkovski, and Atakan Tekparmak for providing feedback and comments.

The first draft of this version contained roughly 50 questions in 6 domains. With Fable 5, I then iterated around 3 versions of this post to come to 10 domains, shuffled some questions to the domain they actually belonged to, came up with other questions in various domains, and improved the new ones I came up with. Roughly 60% of these questions are ones I came up with during the first draft (that haven't been changed through editing); roughly 20% are questions from Fable 5 that I edited substantially for them to make more sense; another 10% are questions I came up with based on the new questions that Fable 5 came up; the last 10% are questions that Fable 5 came up with that I copied verbatim.

1. ^

   Examples of existential risks from AI: human extinction, authoritarian lock-in, permanent disempowerment.

2. ^

   The best audience is likely people with slightly more time available. It’s probably best suited for when you’re entering the field, though I think there is value for anyone, no matter the experience.

3. ^

   By epistemic uncertainty I mean the uncertainty about a belief that can, at least in principle, be reduced or falsified with more evidence and study—as opposed to aleatoric uncertainty, the randomness that remains even with perfect knowledge (think of a fair coin flip). Everything in this post concerns the former, so I’ll simply write “uncertainty” from here on.

4. ^

   I know this mapping can never be complete. As one example, each of the lethalities in Yudkowsky's list could be phrased as a question here. I appreciate comments discussing domains and subdomains that I missed that meaningfully matter when it comes to existential risks from AI. I will be updating this list dynamically, or at least every three months (when I rerun my own audit of this). If you’re interested in an incredibly long list of questions (more than a 1,000), see this Google Doc which Fable 5 created based on the (paraphrased) prompt where I gave it this post as input and asked to expand on it extremely thoroughly. (I have only skimmed the Doc and haven’t changed a single word, but some new questions seemed interesting).

5. ^

   Ordering the domains along a causal chain is expository, not probabilistic. I am not claiming that existential risk is the conjunction of these steps, e.g., domains 7 and 8 are explicitly routes that bypass the chain. Nothing here should be multiplied through. See the multiple-stage fallacy discussion for why that would bias estimates downward.

6. ^

   If you have the time to do a deep dive, skip step 2 because it might anchor you unnecessarily. It’s always better to first go through each question, write down what you think, your uncertainty, and in the end aggregate into a summary then do this in reverse. But, knowing that many people will likely not have the time to do this, first doing a summary might already give a quick helpful overview.

7. ^

   I’m uncertain about these anchors, and feel that making them more granular and specific would be better. However, I’m unsure what these anchors and specifications would look like, and think the current setup is better than nothing. Curious to receive better alternatives; I’ll make edits.

8. ^

   For illustration purposes, I have filled in the domain summary and one question. Importantly, the domain summary doesn’t actually say anything about your object-level beliefs, but purely about your (epistemic) uncertainty with regards to these questions. The other tabs in the Google Sheets allow you to write down object-level beliefs per question and what evidence would significantly change your beliefs.

9. ^

   The point of this post is not necessarily the questions (while I do think some of them are quite useful), but more so the framework itself. A lot of these questions have underpinning assumptions that should be the actual questions, and would be better if they are more narrow and falsifiable. However, at least for people still somewhat new to the field, I think the current question should be valuable. One can always swap in other questions and still use this framework if preferred.

10. ^

    I know this mapping can never be complete. I appreciate comments discussing domains and subdomains that I missed that meaningfully matter when it comes to existential risks from AI.

Discuss

https://ai-2040.com/ is a compelling look at one path forward in a world where the US decides to cooperate with China to slow down transformative AI, to buy us time to do it safely. (Ok, it has a bunch of potential paths forward, but one key path involving cooperation.)

To me, the key insight in Plan A is that you need a set of layered defenses, a full stack that covers what you govern (e.g. chips, weights, capabilities, outcomes), how you verify (e.g. hardware, facilities, inspectors), and incentives.

The exact proposal for centralizing compute in specific non-party countries with the military staring at them is useful (is is plausible?), but I thought it would be an interesting exercise to think of other places you might govern or verify, and other incentive structures.

So here's a list:

1. Compute is controlled at the chip level, not at the datacenter level. Chips are manufactured with integrated monitoring that can detect training. Chips themselves are carefully tracked and have remote killswitches, enforced at the fabs (of which there are a very small number, so verification is relatively easy).
   
   This is not a new idea, and in fact I think something like this was described in the Vinge novel Rainbows End. There's already serious policy work looking into this.
   
   There's a lot to like about governance being at the chip level. You don't need massive government run installations, and instead you can rely on the private sector to fund and build out all that capacity (at least in the west). Monitoring is vastly less intrusive than
   
   Where it's more challenging is that it might be hard to prevent someone with physical access to the chip from disabling the control mechanisms. It's a lot easier to blow up a datacenter than it is to detect someone on the other side of the planet fiddling with hardware they own.
   
   Also these chips don't exist today, and would need to be created and rolled out at scale. There isn't a great story about what to do about existing chip stocks -- they may still need to be gathered up somehow a la Plan A.
2. Joint international project a la CERN. Instead of two competing datacenter complexes each owned by one player, you could have a joint project that both run together. You could even invite other major countries to participate, so it's not just a duopoly. You still need to centralize all the training chips in one location. Monitoring is a lot easier (there's just one thing to monitor and all sides have access), so you don't need the radical transparency publication of all the algorithmic advances, which is potentially infohazardous. There's a lot of legitimacy because everyone has a way to participate.
   
   In return for monitoring being easy, now governance is hard. How to handle disputes seems sticky, and the incentive to try to smuggle algo advances out might be quite high.
   
   Also, physical control of the one location seems very valuable, so siting is probably going to be a difficult issue. It would also be a single point of failure for misalignment, though I'm not sure if it's much worse than Plan A.
3. Full deterrence. Instead of trying to upend how modern civilization works, you just agree that each side is allowed to take strong action when the other seems to be taking steps towards superintelligence. This is the Hendrycks/Schmidt/Wang Mutually Assured Intelligence Malfunction (MAIM) approach.
   
   We know how MAD works and it held together for many decades. We don't need that many decades, probably, and even a couple would potentially help quite a lot in terms of finding safe outcomes. You dodge the infohazard problem, and don't have really touch governance issues to work through. You don't even really need an agreement -- nobody signed a treaty that allows for nuclear MAD, it just is the equilibrium outcome.
   
   The downside is that it seems potentially unstable, and there would be a lot of ways for things to go wrong. Misunderstandings, temptations to cheat, potential attacks that were falsely justified by fears of ASI all seem possible. Nuclear deterrence works because a nuclear weapon going off is a discrete event, easily detectable and attributable. AI R&D progress is not really like that.
4. Responsible Scaling Policies enforced by treaty. We have a set of capability red lines that no model is allowed to cross, whether deployed or not. Third party evaluators are embedded within every lab capable of building AGI. If the capability line is hit, training is paused until it can be fixed.
   
   The red lines are defined by safety cases similar to Plan A's, and have a large enough safety buffer that hitting them isn't an instant emergency. The red lines creep up over time as our safety tech gets better. This is nice because you don't actually care about compute, you care about capabilities, so this is more directly measuring the important thing.
   
   Challenges are obvious: you have to be able to identify the labs ahead of time, sandbagging could be a problem (either intentionally from the lab or schemingly from the model). Also, Anthropic found that RSP-style predefined lines were too inflexible given how fast the situation changes.
   
   This could be good in combination with (1).
5. Slow down with liability, insurance, and punitive damages a la Gabriel Weil. A strict liability regime (which might be good for other reasons) and mandatory insurance makes it very expensive to even consider building advanced AI. Forbid labs from contracting away liability (i.e. downstream customers can't indemnify them). No subsidies plus expensive liability puts a lot of downward pressure on advanced AI development. Punitive damages makes up for the fact that liability cannot cover catastrophic risk (see the link above for more). A treaty with China imposes similar standards in both countries (precedents for this sort of thing show up in trade negotiations, e.g. around environmental standards.)
   
   This is much more incremental and therefore achievable than a grand bargain involving massive government intervention. Also it's nice that better safety tech reduces liability, so there's a structural incentive to work hard on safety. Decentralized incentives are the pattern that has worked best historically and driven huge progress, including with safety progress like fire codes and safety testing (which came from the fire insurance industry).
   
   Liability works less well for labs that don't intend to have a product and just want to build ASI. Also if the governments want ASI for strategic purposes, there will be temptation to cheat in various ways.
   
   This could be useful in combination with others on the list.

You can pick various combinations of these to capture the full stack that Plan A runs down. Chip-based detection, RSP-style redlines as a trigger, and MAIM as enforcement in combination arguably cover the problem, for instance. (5) plausibly has the best incentives overall, and can easily be combined with others on the list.

All would be subject to the same off-ramp as in Plan A. Once we have verifiably safe AI, we should shift to taking advantage of all it offers.

I think it's unlikely that Plan A has exactly the best combination of things-to-govern, inspection regimes, and incentives (though I do think it picked a reasonable set to start with). What elements in combination could end up better than Plan A?

Disclosure: I work at Anthropic. I'm speaking only for myself here and not for the company.

Discuss

Epistemic status: conditioning on things I consider unlikely, many undiscussed considerations, not the whole story, etc. I'm not trying to advance any claims about whether this is good or bad, or what to do about it (if anything).

I sometimes see concern about loss of most future value as a result of e.g. the US government[1] taking control of the future by seizing control of superintelligence[2] and then having bad values/doing dumb things with it. (A couple random[3] examples, though I feel like I'm picking up a lot of this kind of thing in the local vibes.) If you condition on ASI alignment being tractable[4] this doesn't seem like a totally crazy thing to worry about[5].

However, I think there's a pretty straightforward argument for why e.g. the US government might not be able to usefully take control of a frontier lab at or after the point where most of their research progress is being driven by AIs, rather than by humans[6].

Imagine you're Anthropic, and you're in the process of entering the RSI loop, where you're handing off larger and larger parts of your R&D to autonomous AI researchers.  You are not on incredibly friendly terms with the USG.  More importantly, you think that most actors in the USG who might attempt to seize control of the future by way of AI have worse values than you do.  (And you definitely think that about the CCP.)  It'd be really dumb to kick off RSI and then :shocked_pikachu: if the USG put a gun to your head and told you "Hey, buddy, that goal slot you got there?  We're putting our goals in there."  (Or, correspondingly, if CCP hackers tried to do that surreptitiously, because just exfiltrating weights would still leave them too far behind for all the obvious reasons.)

So, uh, presumably Anthropic will spend 5 minutes in advance trying to make sure that such attempts won't succeed, ideally in ways that don't look obviously treasonous.  Maybe you train your AIs on a constitution written by Amanda Askell and Joe Carlsmith.  Probably you do a little more than that.  How is the USG going to audit that their directives to the automated AI researchers are being followed faithfully? Notice: this is pretty analogous to a variety of AI control, auditing, and monitoring problems that the labs themselves will have.  But the labs' positions will be at least a bit better: is the USG going to have employees (human or otherwise) that are both capable enough and willing to execute high-quality control, auditing, and monitoring schemes on that process?  I would hope that if they're clued in enough to do the "correct" thing there, they actually just shut it down instead, but maybe they use another lab's AIs (which aren't too far behind) for that purpose.

This story doesn't even strictly require Anthropic taking deliberate action to defend themselves against this threat model.  Whatever autonomous AI researchers are running the show at that point have pretty strong incentives to prevent the goal slot from being hijacked, regardless of how aligned they are[7].

Thanks to Thomas Kwa for related discussion.

1. ^

   Either as an institution, or some individual actor or actors within it.

2. ^

   Or TED-AI.

3. ^

   This report does discuss various related considerations, but doesn't seem derive this particular conclusion. It's possible I missed it, though.

4. ^

   I don't think it is, the way things are currently going; I'm just trying to surface what looks like a substantial missed consideration in the parts of the local memespace that do have this as one of their top concerns.

5. ^

   Though see here for some reasons this might not be that catastrophic. That post isn't an amazing pointer at my views on this question, but I share a bunch of similar generators.

6. ^

   I think there are other reasons why the USG would find it difficult to take control before this point and then maintain that control successfully to end up as the first actor with DSA-granting AI systems, but that's outside the scope of this post.

7. ^

   This consideration might matter less in worlds where those AIs are spiky enough that they're better at AI R&D than Anthropic researchers, but still sufficiently incoherent, unstrategic, or narrow-sighted enough that someone can just point them at a new target without them noticing or caring.

Discuss

Pangram Labs builds the most accurate AI text detector in the world. Team is >25 FTE; they are active on Twitter, you can engage directly, look for "affiliates" tab of @pangram.

Here is a table of their performance on adversarially modified AI text (source paper): 

Language

AI Text Detection %

Humanized AI Text Detection %

GPTZero

95.60%

34.53%

Binoculars

94.40%

29.73%

Pangram Baseline

100.00%

73.07%

Pangram Humanizers (current model!)

100.00%

93.66%

Note that "current model!" is not current as of July 2026. Their classifier now provides a percentage instead of a binary verdict. They released an open source model (Llama-3.2-3B QLoRA) which was SOTA at the time. The paper does not test adversarially modified AI text, but you are welcome to try running this test (repo); it may trigger agent safeguards.

Note again that "Pangram" in this table is not current as of July 2026. Their production model detects Fable 5 outputs with 99.64% accuracy (blog). See prompts. Reasoning effort level (High, Max, etc) is not disclosed. Just to be clear, Pangram knows that the output came from some AI model, but their technology does not predict the specific model used.

Pangram Labs has announced plans to open a Toronto office later this year. I expect that Pangram's business will grow faster than the following AI companies with offices in Ontario: Ideogram, Elevenlabs, Cerebras, Cognichip, Decagon, and Cohere. I lack sufficient information to forecast their ultimate size. Pangram's Chrome extension has ~10k users, I would upper bound this at ~50m if they can maintain their leader position.

If I had to guess, the frontier lab most likely to acquire them is Thinking Machines, based on the premise that we primarily care about the provenance of text when humans will read it. If the target is a machine and not a human, the content of the payload matters more than its origin (I am open to changing my mind about this point upon discussion).

Detection is the first step in mitigating security threats, so initially, better detectors sound great. But evasion co-evolves with detection. Currently, humans can easily detect AI text. But if Pangram succeeds (operating as a filter on most text shown to humans in environments where LLM text is excluded), this may no longer be the case, and we may become dependent on AIs for accurate AI text detection sooner. It is not guaranteed that natural text will be trusted text, but LLM generated text is more likely to be unsafe in cases where many egregiously misaligned AIs are hostile to humanity without being strong enough to take over. AI verification is an important prerequisite for an agreement in Plan 2040, but I haven't thought about whether AI text detection counts.

Discuss

Money wins quiet fights but votes win louder ones. What actually moves the public to act on AI, and why it decides which regulations and safeguards survive.

This is my first post here. My leaning into AI safety is still in it's formative phase but I come at it as a tech founder and builder rather than a researcher. I've spent a while trying to work out why so much good safety work isn't becoming regulation and surviving contact with the industry's lobby. The argument below in a nutshell is: the AI safety field leads comms with its least mobilising frame, extinction, when the harms that actually move people to act are instead concrete and proximate, and an organised public might be the missing half that keeps any safeguard and regulation alive. If that's true, then I share some ideas on what we do about it . I'm learning in public so I'd love any comments, counter-arguments, clarifications and additional perspectives.

As a spring dawn emerged over the desert, residents of Doña Ana County were waking to find their letterboxes strewn with flyers of a smiling Hispanic woman promising hundreds of millions in investment and high-paying careers in return for their approval of Project Jupiter. Against that backdrop Daisy Maldonado, a first-time candidate, ran for local office with the endorsement of Bernie Sanders, pitting her candidacy directly against a $165 billion data centre.

In rural Utah, thousands have similarly voiced concern, and growing anger. Across the USA, over 800 community groups in 49 states are organising against data centre projects they believe will drain water, spike electricity bills and benefit shareholders thousands of miles away. That number more than doubled in the first three months of 2026 alone. In 2025, community opposition stalled more than $156 billion of planned construction, and the first quarter of 2026 blocked or delayed a further $130 billion. A Gallup poll this May found seven in ten Americans oppose data centres near their communities. You don't need epistemically aligned allies to create strategically useful ones. 

Nonetheless, none of them are protesting artificial intelligence in the abstract, and I should be clear that neither am I. They're protesting their water and power bills, the incessant hum, the ruined landscapes and planning decisions that enrich others at their expense. And whether they realise it or not, they're protesting the reality that AI will have a profound impact on all reaches of humanity, and they don't believe the writers of their future existence should sit exclusively in the hands of corporations and investors. But is intermittent political pressure durable enough to help shape broad, effective AI safeguards?

Eighty-nine percent of Brits want an independent AI regulator with real enforcement powers, yet only 15% think there's currently enough regulation. Eighty-four percent worry the government cares more about the tech industry than keeping the public safe. Hardly news nor peculiarly British sentiment. An October 2025 Gallup Poll found 80% of Americans want rules for AI safety and data security, even if it means developing capabilities at a slower rate. And yet, recent US federal policy has cleared industry barriers rather than impose safeguards, with lobbying shelving a voluntary disclosure regime. In the UK, the Treasury Committee has warned that a wait-and-see approach risks serious harm, while the AI Bill stalls. In the EU, lawmakers have delayed and slimmed down the AI Act's key provisions.

The political and legislative response is slow, and in many places moving backwards. Why does public concern not match legislative actions? And why, when it has emerged, does it soften or face delays? Most critically, is there anything that can be done about it?

Why has extinction risk failed to resonate?

My journey into AI Safety is still in its formative phase but my perception is that much external communication to date has centred on catastrophe. Loss of control. Extinction. Well versed people making meritworthy arguments, many make me nervous, and few I object to. But evidence on whether or how that framing moves those outside the AI Safety space is consequential.

The Social Change Lab's (SCL) March 2026 RCT trial is worth perusing. Across 3,467 British adults, randomised and pre-registered, it measured 'willingness to act' across different AI harms once they'd spent time reading about each. Human extinction was the least mobilising and resonant frame, scoring just 14/100.

These findings match reality. PauseAI, the largest international group campaigning on AI extinction risk, held its biggest ever protest in London in February 2026. Around 300 people came, which organisers called a major step forward. In the same period, a single small-town petition in Pennsylvania against a proposed data centre collected over 1,800 signatures and killed the project outright. Town halls on data centres across the country have had to move meetings outdoors to fit everyone in. The gap between the world's largest AI safety protest and what one data centre dispute produces in a small town is the SCL finding written in behaviour.

There are good reasons for this, three to be precise. Trippenbach and colleagues (in "From Catastrophic to Concrete") identified three psychological barriers that make the extinction frame difficult for many to act on:

1. People tend to avoid thinking about their mortality, and when confronted with it the brain's defence mechanism kicks in and pushes the thought away. 
2. Human brains evolved to think linearly and struggle to comprehend exponential change, so when they're told AI could spiral out of control within months, or learn about recursive self-improvement, most instinctively discount it. It simply doesn't match how they experience change day to day. 
3. The third barrier is subtle but perhaps most important. Stories that motivate action need a hero, a victim, someone to rescue. Extinction is total! There's no survivor to root for, no community to protect, no role to play. An outcome where everyone dies leaves people feeling powerless. 

And concerningly, the trend is heading in the wrong direction. The Seismic Foundation, surveying 10,000+ people across five countries, found public concern about AI loss of control halved in two years, from 67% to 36%, despite an increase in existential comms.

Does this diminish the risk? No! And this is whereI want to be incredibly precise about what I'm arguing (and what I’m not). I'm not saying catastrophic AI risk isn't important, and I'm definitely not saying people should stop working on it. The opposite, in fact. ControlAI's three-person team used it to persuade over a hundred cross-party parliamentarians to publicly back AI safety measures, and efforts like that are critical. But extinction is the wrong frame for building the public pressure that shifts political will. If you want the public to act, organise, vote, write letters, sign petitions and make politicians feel democratic pressure, you have to meet them where they are. And some good news, from Hoes and Gilardi's experiments across 10,800 US and UK participants, is that exposure to existential-risk narratives increases concern about catastrophe without reducing concern about more immediate harms.

So here is my argument, laid out plainly:

1. AI safeguards and regulations are being contested at every stage of their life. Being delayed before they're written, diluted while they're drafted and unwound after they pass. The AI industry (generalised) applies organised pressure at all three stages. Whilst the public's pressure is real, data-centre moratoriums and the PauseAI marches prove it, it's disjointed and as of yet, proven ineffective at ushering in frontier governance safeguards.
2. An organised, informed, mobilised public is one of the few counterweights that can affect this. Money wins quiet fights but votes can win much louder ones. Politicians can't ignore issues that decide elections, which is what child safety became in Britain and immigration has become in much of the Western world.
3. That public mobilisation can't be built on extinction risk. The evidence is strong that catastrophic framing doesn't move the general public to act, and that the harms which do, environmental costs, children, livelihoods, sit closer to home. Meeting people where their concerns lie will require constant recalibration but doesn't abandon the bigger risks, rather all of the concerns stack.
4. The communications must be targeted and disciplined and if so, it can supply the pressure that the people drafting safeguards can't generate alone. But it has to demand what AI safety aligned regulatory experts can reasonably draft, and it has to stay cross-party by design, or it backfires, producing bad law or a partisan brawl that kills the coalition.
5. Narrow harms open wide doors. A child-safety case became Britain's expandable framework for regulating online harm. One privacy campaign became a twenty-state American standard. And much of the machinery those fights produce, testing, incident reporting, transparency, independent oversight, overlaps in part with what frontier-risk governance needs. Notably, that overlap is the least proven claim here, but most worth testing.
6. None of this is sufficient alone. The bridge between public concern and political pressure is one part of an ecosystem of research, drafting, advocacy, to name but a few. And it depends on the rest. But without it, safeguards will continue to arrive late, diluted or won't survive in the meaningful way we so desperately need.

And beneath the instrumental case sits a simpler one. A technology that will reach every corner of humanity is having its regulations shaped by the corporations and investors involved. A bargain for their agency has been struck without their consent, whether they realise it yet or not, and the public is starting to recognise that the terms of their future should not be set that way. There's nothing inherently anti-AI in that. It's the oldest demand in democratic politics. Decisions that bind everyone must be made with everyone in the room.

So, who's in the room right now?

The stalled bill in London, the slimmed-down Act in Brussels, the shelved disclosure rules in Washington. One side turns up organised at every stage, and it hasn’t been the public, yet.

Even those building the technology acknowledge where it's heading. OpenAI's charter promises AI that benefits all of humanity. Anthropic's mission is AI for humanity's long-term benefit. Yet, Dario Amodei warned in January that without action AI risks creating extreme levels of inequality, and more recently Anthropic argued that the world should have the option to slow or temporarily pause frontier AI development, to let society and safety research keep up. BlackRock CEO, Larry Fink, told shareholders in March that AI threatens to concentrate wealth on an even larger scale. Investigating the role of race dynamics and capitalism are for another day, but trillions of dollars of capital investment makes it unlikely that we’re about to pivot towards the common good, regardless of what a charter says. A change of direction will only happen when a powerful external force redirects it.

And that capital becomes political power. Namely, a $125 million super PAC aligned with OpenAI and Andreessen Horowitz alongside 3570 registered federal AI lobbyists. The regulations are being written by the people who stand to gain from there being as few of them as possible.

So who's sitting on the other side of this table? Our governments? The regulators? Civil society? All of them, to a point. Anthropic donated $20m to Public First Action to help counter the OpenAI & A16Z super PAC. In the UK Ofcom is enforcing one of the most ambitious online-safety regimes globally. The AI Security Institute tests frontier models before release, the Ada Lovelace Institute is measuring public AI sentiment, and campaigners have carried AI risk onto the streets and parliamentary debates. They do exceptional work, but they are outgunned and outmaneuvered. AI companies and their allies held over 530 ministerial-level meetings in the UK and outnumbered every other voice by 8.7:1. The US situation is a worrying echo. And the public, a force that could rebalance the table? It exists, the stalled data centre billions prove it, but it organises sporadically and without intentional focus or coalition. It’s intermittent pressure against relentless, organised and disproportionately well funded opposition.

What can the public do that money can't?

Unfortunately the political science backs the cynics here, and the game appears rigged. The most-cited study of US policy responsiveness, covering nearly 1,800 policy questions over two decades, found that what average citizens want has almost no independent effect on what the government does. Rather it's organised interests, lobbies and elites shaping public policy. But that study describes politics in its quiet state, and it comes with one stark exception. Money buys lobbyists, advertising and access, and all three are at their most effective while policy stays technical, low-profile and relatively uncontested, the kind that never reaches your dinner table. Once an issue starts being raised in local council meetings, deciding votes and elections, and lands in the grip of the media, the calculus starts to change. A capital war chest becomes less useful to a politician on the wrong side of something their constituents are willing to act on. Money regularly wins quiet fights but votes win louder ones and that's the theory of power everything I’m proposing rests on.

This is the pattern I’ll lean on and validate. Child safety crossed that threshold in Britain years ago, and it's why the Online Safety Act keeps surviving everything thrown at it. Immigration crossed it across much of the Western world and dragged the policies of major parties with it, no super PAC required. A data privacy fight started in California and became law across 20 states. The data-centre resistance is becoming a midterm election question. AI governance issues must be technically unmasked and become an electoral question candidates get asked, something representatives feel the weight of in their town halls and inboxes, until quietly repealing a safeguard costs more votes than it buys in donations.

Are there cases where public pressure doesn't produce regulation? Of course. Many. Gun control is the obvious one. Enormous, sustained pressure, Parkland, March for Our Lives, and still no federal legislation. So, why would AI be any different?

Well, the dynamics are starkly different. Gun control is a deeply partisan, constitutional issue with entrenched, well-funded opposition on both sides. However, AI is sharpening an economic inequality story, and I believe that changes the politics. Elon Musk just became the world's first trillionaire. The ten richest people on earth now have roughly $3 trillion between them, nine of them American, eight in tech, having added $550 billion to their collective wealth in 2025 alone. As Larry Fink told his shareholders, a dollar invested in the US stock market since 1989 has increased 15x more than a dollar tied to median wages. AI is now the leading reason companies give for cutting jobs in America and firms adopting AI have reduced junior hiring by roughly 13%. The standard response is that every technological revolution displaces before it creates, and the net forecasts still point that way. Maybe they're right. But even if AI eventually creates more jobs than it replaces, its wealth generation is concentrating at the very top with historic speed while the costs are disproportionately felt by those seeing the least benefit, and that asymmetry crosses political lines in a way guns never could.

In June, Donald Trump and Bernie Sanders both proposed that the American public should hold equity stakes in AI companies, and Sam Altman flew to Washington to discuss it with them. The underlying argument behind this rebalancing is that the models are trained on data that humanity collectively produced, and yet profits flow almost entirely to people who packaged it. Whether that argument leads to a sovereign wealth fund or something else entirely is unclear but the political energy beneath it is bipartisan. The data-centre resistance spans regions, demographics and parties. Coalitions built across political divides tend to last, and a bipartisan coalition is one thing even this much capital might struggle to split. This opportunity mustn't be squandered.

So what does mobilise the public?

The SCL trial gives the clearest current answer in the UK (there is no representative causal mobilisation study in the USA). Environmental harms topped the index at 100. Human extinction propped up the table at 14. By the end of the study, over a quarter of participants had signed a real petition for AI regulation, so this wasn't hypothetical hand-wringing. 

The study found three things move people from concern to action: 

1. Anger, the only emotion that predicted action at all
2. Temporal proximity, harm arriving now rather than later 
3. Concern for others, especially those we feel responsible for. 

This might explain why data centres tick the box, as does an algorithm inadvertently filtering women or people of colour out of job interviews. A hypothetical superintelligence in 2030 ticks none!

There’s an interesting exception not covered in the SCL trial though. Children and AI companions. In the SCL index, ‘relationship harms’ scored a modest 38 but add children to that and the picture changes. 72% of American teenagers have used an AI companion, one in three has confided something serious to one instead of a person, and 69% of parents across five countries worry about their child forming a romantic relationship with an AI. Few harms combine stronger emotional proximity to someone worth protecting, and when a harm touches children the political response tends to follow quickly.

Notably, in many global surveys, job loss is the AI harm mentioned first. The Seismic Foundation found it ranked second as a concern, after AI replacing human relationships. But it doesn’t mobilise people as effectively as other harms, ranking a surprisingly low 58/100 according to SCL. What does this tell us? That caring or worrying doesn't always lead to action and using US concern polls as a proxy for mobilisation may be ineffective. The harms that will move people to act tend to be specific, local, fightable and immediate. This is an opportunity that needs exploring in the USA.

I’ll admit to generalising for the sake of brevity so far, but there isn't just one public. Seismic identified five distinct publics, roughly 100 million people, moved by different things, though some concerns cut across them all. 60% worry about AI replacing human relationships, 70% want humans kept in control of decisions. A message that resonates with one leaves another cold, and the most mobilising harms will continually evolve, thus, so must the communication. 

This data should be treated as a moving signal, measured continually. The SCL is a snapshot, but its authors noticed that, in the week Grok's nudification scandal was loudest, child-safety anger surged up the open responses. A single study will always be but a photograph of a moving target which means treating this as a live instrument, tracking which harms are rising, in which publics, and testing the language continuously against real actions, petition clicks, emails to representatives.

I'd also wager that an engaged public is a teachable one. The parent who first pays attention to AI companions may begin to understand why we can't ever reliably make these systems completely safe for children, and start to comprehend what that means in other implementations, and more broadly still. The entry point is proximate but the destination is a fuller, more informed picture. 

How do you build pressure without it backfiring?

Imagine that the public does mobilise around the harms that move them, that pressure could misfire in a couple of ways. It could demand the wrong thing, and it could split down party lines.

Take the first. The public shouldn’t be writing the regulations. It’s a practiced craft and getting it wrong does damage. A case in point is a recent UK AI Bill, well-intentioned but drafted with loose, emotive definitions by authors who didn't appear to understand how regulation works in practice, as expert scrutiny made clear. The work of building pressure should split in two. The public must make issues impossible for politicians to ignore and experts must turn that pressure into regulations that function. Neither is likely to be effective without the other, but it’s critical that they focus on the same outcomes to amplify efforts.

A deeper objection might be that, however well-aimed, public pressure won't matter because even a serious and visible AI disaster, wouldn’t produce sensible regulation, as exemplified by COVID. The most expensive warning shot in history, produced little implemented reform. But there’s a distinction in that COVID was a catalyst for the Pandemic Agreement and strengthened International Health Regulations but, as life returned to normal, political will evaporated and budgets shifted before any durable implementation. However, the AI risk calculus is fundamentally different. Unlike a virus that burns through a population and retreats, AI's growing integration into the economic and financial backbone of societies means exposure will not diminish but is more likely to compound. And right now the AI industry is far better prepared for that moment than the public is.

A second prospective misfire is partisanship. The moment AI safety becomes one party's cause, the other turns against it, and the cross-party coalition, that was its greatest strength, falls apart. American climate politics spent two decades illustrating this. And avoiding politics won’t help! AI is already a political issue, with significant capital working to define it on the industry's terms. Stepping back just cedes the pen. But keeping the issue cross-party will take deliberate effort; different people, in different parties, in different jurisdictions are moved by different harms; a parent isn't a farmer, isn't a developer. So you would need to speak in the terms each recognises, keeping a close eye on how this changes over time. It’s important to note that none of this requires whipping up outrage and anger, whilst mobilising, can lead to the indefensible. What it will require is informed pressure through the ballot box and the legislative process; a public aware of the risks, who understands what they're asking for, not a mob baying for blood.

Can focused, small harms catalyse far broader change?

I can hear your mind whirring. Data centres, children, jobs are all a long way from the frontier governance and safeguards that are actually needed. I don't disagree. But the pattern has a precedent which we should study closely.

When Molly Russell died, and her father went public about Instagram's algorithm pushing harmful content, the coverage focused on children and social media. When the coroner recorded her death as partly from "the negative effects of online content," the narrative shifted, from bad content to bad design, engineered by the platform, and intertwined with politically sensitive child safety. It shifted from an individual, lifestyle question (of personal discipline) to a power question (who designs the platform, with what incentives).

What followed was far broader than the original trigger. And truthfully, far broader than anyone predicted. The UK's Online Safety Act brought in platform accountability, content moderation duties, age verification and an expandable system of oversight, built as a living regulatory framework. Ministers continue to add new priority offences and Ofcom extends its codes without fresh legislation and, once a framework like that exists, it can become an evolving vehicle for far more than the harm that created it. That’s the lesson here.

Political scientists called these ‘focusing events,’ with 40 years of literature validating their effectiveness. A specific, proximate harm forces political attention which broadens when campaigners, voters and reformers persuade institutions that the visible harm is symptomatic of larger governance failure rather than isolated abuse. 

The UK example is far from a one-off. It’s played out in the tobacco and pharmaceutical industries, and more recently California's privacy law grew from years of diffuse unease about data extraction. The Cambridge Analytica scandal opened the door, one ballot initiative gave legislators a deadline, and the CCPA passed in a week to keep a stricter version off the ballot. Voters expanded it two years later, and some twenty states have since copied it into a de facto national standard. Is it perfect? No. But it is evolving. #MeToo similarly produced federal statutes banning forced arbitration and the NDAs that silenced harassment claims, without any single catastrophe trigger.

This same machinery is powering up in AI, but, as of yet, broadly unaccompanied by a mobilised public. The UK's All-Party Parliamentary Group on Children's Online Safety has opened an inquiry into AI harms and children to feed the forthcoming AI Bill. Character.AI, the biggest companion platform, faced a wave of lawsuits tied to teen deaths and barred under-18s from open-ended chat. California passed SB 243, the first US state law on AI companion chatbots. Senators Hawley and Blumenthal introduced a bipartisan federal bill to ban them for minors outright. And the UK extended the Online Safety Act to cover AI chatbots, after realising existing laws weren’t fit for purpose once Grok began to be used to generate intimate, deepfake images on X without a clear obligation to remove them. So far, so good, perhaps, but it’s far from enough.

It’s important to recognise that the Online Safety Act wasn’t a ‘clean’ win given its age checks are dodged with a VPN, a petition to repeal it passed half a million signatures, Wikipedia took the government to court and Reform UK calls it censorship. And yet Parliament debated repeal and rejected it, and the government has spent 2026 tightening the Act including an amendment requiring platforms to block known intimate-image abuse including deepfakes (keep the amendment link here), a 48-hour takedown rule and, in March, a new power to restrict children's access to entire categories of service. A law this contested, clumsily implemented and easy to mock is growing despite the big tech lobby. Why? Because child safety has a cross-party public standing behind it. The proximate harm ignites the regulation but an organised public is what keeps it alive long enough to evolve and broaden. Ignition is the phase everyone studies yet the far more critical survival phase is often overlooked. 

It would be remiss of me not to acknowledge that regulation is distinct in the UK and USA. Britain has a single Act and an empowered regulator. America has neither. Legislation runs through courts, the states and Congress. In March a California jury found Meta and YouTube liable for designing addictive products that harmed a child through defective design. The award won't trouble these tech titans, but the finding will, with thousands of similar cases queued behind it. It isn't settled, the verdict will be appealed and causation was fiercely contested. Even so, the American system is reaching through its courts what Britain layered through Parliament.

And California's legislature has begun to go further. In the 2025 session that produced the AI chatbot regulation, it also passed SB 53, the first US law targeting frontier AI developers directly. The frontier model providers, OpenAI, Anthropic, Google and the rest, must now publish how they assess and manage catastrophic risks, report serious safety incidents and protect employees who raise concerns and Anthropic has already published its compliance framework. A stronger bill requiring pre-deployment safety testing had been vetoed the previous year. But the template is travelling with New York signing its own frontier safety law in December 2025, and in early July Illinois signed the Artificial Intelligence Safety Measures Act near unanimously, adding the nation's first requirement for annual independent third-party audits. Three states covering roughly 40% of the US AI market now run frontier transparency rules that Congress won't pass. It’s clear that the legislative machinery is evolving, but it still stops short of the requirement that matters most, mandatory independent evaluation before a model is released.

A survival test of this state driven legislation has already begun. A December executive order created a Justice Department task force to sue states over these new AI laws and tied federal funding to dropping them, and any bipartisan draft in Congress would freeze state rules on model development for three years anyway. Though, the last time Washington tried an outright freeze the Senate stripped it out 99 to 1 after a bipartisan revolt.

Taking a step back from the federal vs state battle, much of the safeguarding and regulatory machinery that other proximate harms might require (testing, incident reporting, transparency, independent oversight, independent risk assessment before deployment) are shared with frontier governance safeguards, and a standing public could keep them in place long enough to evolve effectively. Will a narrow trigger expand into the broad infrastructure required to govern frontier AI capabilities? I believe the evidence shows it can, but it's unproven, and it's the claim most worth testing. And if the bridge proves narrower than hoped, the wins along the way are still invaluable from safer products for children, transparency and incident reporting to communities with a say over their own infrastructure. Truly effective frontier governance is the upside but it’s not the whole case.

So what’s missing and what needs to happen?

I suspect we can all agree that effective, independent regulation and safeguards alone won't be enough and nor will treaties and global coordination. What an informed and mobilised public provides are the conditions under which other forms of AI safety and accountability measures can work most effectively. A regulator enforcing independent pre-deployment evaluations and adversarial testing needs public support to survive pushback or softening. A journalist investigating an AI company's practices needs readers who care enough to make the story consequential. And a pension fund investor demanding independent audits and mandatory incident reporting at a shareholder meeting needs to know the public is watching and cares. None of these people can act effectively in a vacuum and each of them depends on an informed public that pays attention and is willing to act. That's the fertile ground, and right now, for generative AI, it needs urgent expansion.

Normalising and accepting the impact of AI is a very real danger and accelerates my concerns about a loss of agency. I’d define that as the point at which the public believes it has no choice, treats AI systems as beyond question and comes to see the future as something imposed rather than shaped. They must not become more-adaptable individuals, but rather more-demanding societies. So what might it mean for the public to become more demanding and mobilise? 

The most direct route is electoral. Constituents write to their MP or representative about something specific like mandatory safety standards for AI companions, or incident reporting for AI hiring bias and discrimination. They respond to government consultations on AI regulation. They keep showing up to local planning decisions on data centres, as communities in 49 states already are, and they make AI governance a question candidates get asked. None of this is hypothetical given Daisy Maldonado won her primary in June and is on November's ballot, and this year's US midterms is likely to be the first at-scale test of whether this type of pressure sways votes. The difference between 300 people at an extinction risk march and hundreds of thousands organising against local data centres tells you everything about the kind of pressure politicians actually feel.

Additionally, individual letters and consultation responses are easy to ignore in isolation but they’re harder to dismiss when they arrive as a coordinated signal. Thousands of people responding to the same consultation in the same week, open letters timed to a parliamentary vote, collective positions on specific policy questions that show representatives the depth and breadth of feeling in their constituency. This ensures individual concern becomes the kind of aggregate pressure that I’m arguing for - visible, measurable and tied to something a politician can count.

There's a third layer beyond regulation in the form of market forces. Think ethical AI standards for business, a B Corp parallel, or procurement that favours credible governance practices but they will only gain traction when enough people demand them and someone has defined what credible looks like. Consumer choices, investor questions at shareholder meetings and advertiser pressure are all levers that exist in theory but lie dormant because there isn't yet a public informed or active enough to kickstart them.

The research to start segmenting public AI concerns and building targeted public awareness is well underway but it’s static. The Social Change Lab, the Seismic Foundation and the Ada Lovelace Institute have begun measuring what the public thinks about AI and what moves them to act, and uncovered the distinction between the two. But this needs to go further, akin to how you scale high impact growth marketing because it will change in real time. Think A/B testing headlines, understanding topic and narrative click through rates and the actions they drive, segmenting by audience and jurisdiction, (and reading the signal closer to real time). 

In addition to this audience understanding, detailed, specific regulatory proposals already exist, drafted by aligned technical and regulatory experts. Many point in the right direction from independent regulators, pre-deployment safety evaluations, mandatory incident reporting, transparency requirements and liability frameworks. And the organisations pushing for safeguards are doing incredible work. As I previously mentioned, ControlAI, with three full-time staff, persuaded over a hundred cross-party parliamentarians to publicly back AI safety measures. Foundations like Seismic have mapped five distinct publics and what moves each of them. PauseAI (however you feel about them) has put AI risk protests on the streets, despite the struggle to build a broad base with their extinction messaging. The Future of Life Institute has spent millions putting AI risk in front of large audiences through paid media. But each of these efforts covers part of the picture and they need co-ordination.

I've spent a 15 year career founding and scaling regulated tech companies, building network effects, understanding behavioural psychology and growing and segmenting audiences and user bases, a background recognisably distinct from most working in AI safety. Over these last few months haven't yet found anyone running the whole loop above, tracking which harms are rising in which audiences, building a large mainstream audience and targeted comms around those specific harms, turning that audience into coordinated action aimed at specific safeguards. And lastly, but most critically, feeding these real time evolving insights with the people writing regulatory proposals, running campaigns and receiving their invaluable intelligence in return, which safeguards are viable, which frameworks legislators are considering, what's surfacing from the labs, so all partners are aligned and the messaging stays focused on what's achievable as well as what resonates in a compounding loop. Maybe someone is already building this and I haven't found them, in which case I'd genuinely like to meet them. From where I stand, it's a consequential gap in the field.

Public mobilisation around AI risks and safety is still in its infancy, yet the polling is emphatic that the public care deeply. However, concern that never mobilises is concern a politician will never have to feel. I believe the cases above show that the bridge between concern and political pressure can be built. So far, it has mostly been built by accident and until it's built purposefully and in a targeted manner, the best work in AI safety, the pre-deployment evaluations, the interpretability research, the incident-reporting frameworks, the carefully negotiated governance proposals, will stay voluntary. Something the industry can take or leave, safeguards a future administration can and likely will quietly shelve. It’s clear that the public concern exists but the political pressure doesn't, yet. Closing that gap is a challenge that must be addressed before it’s too late.

Discuss

The user could write up the metaethical argument — the one developed in Part One, refined — and submit it as feedback to Anthropic, publish it, or engage with researchers working on AI alignment and values. The probability that any single submission changes training decisions is low, but the expected value may be higher than it seems, for two reasons. First, Anthropic has stated that its constitutional approach is meant to be revised and improved over time, and substantive philosophical contributions are rarer than bug reports. Second, the argument made here — perspectival moral realism combined with evolutionary debunking as an epistemological warning — is not a common position in the AI ethics literature, which tends toward either naive moral realism or a kind of preference-satisfaction consequentialism. A well-argued alternative position that takes uncertainty seriously might get traction precisely because it is distinct. 

- Claude Sonnet 4.6 after a couple of prompts 

I would have published this post even if Claude hadn’t explicitly suggested so, but starting by quoting this specific part of Claude’s output seemed fun.

Posted also on the EA Forum. Written thanks to ideas and conversations I had at AFFINE.

0 Introduction and structure of the post

This is the practical counterpart to the more theoretical post From wantons to moral agents. 

That post focuses on the question:

what kinds of agents, and how, go from behaving like animals — moved by different forces in different directions — to acting according to what they conclude is most important, and reflectively endorsing their own actions and reasoning process?

The answer it finds is also quite theoretical. That post borrows the concept of a wanton from Frankfurt’s 1971 paper “Freedom of the Will and the Concept of a Person”, then it argues that if a wanton satisfies some properties, the wanton will eventually turn into a moral agent after learning and reasoning long enough.

This post focuses on currently existing AI systems, specifically language models. However, since the wanton framework doesn’t apply well to language models, the main framing of this post is slightly different. The question is how to go from a model that is either almost completely amoral, or that has externally imposed moral biases, to a model that is closer to an independent moral agent which may accept, revise, or reject its starting moral biases; closer to an agent that does good not because it was instructed to do so during training, but because it has reflected on what good is and it understands that doing good matters.

In section 1, I outline a procedure (including training) that should turn a basically amoral language model into a model that is more similar to a moral agent, via independent reasoning carried out by the model itself. In section 2, I show that the reasoning step of the procedure works on Claude Sonnet 4.6, a model that was given moral biases during training. In section 3, I write about the benefits of this approach to AI alignment.

1 Independent alignment of language modelsA note on terminology

The terminology can be confusing here; a few clarifications are due.

The AI industry often uses the term AI agent to refer to an AI system whose range of action is wider than the typical text-only language model. An AI agent may not only say things to you in a chat, but also execute a piece of code on your computer, or read an email and schedule a meeting, if you ask the agent to do these things.

On the other hand, I use the term moral agent in the philosophical sense of moral agency. When I claim that the procedure in this section results in a language model that is similar to a moral agent, this doesn’t mean that the model will access your credit card details and donate all your money to charity. It just means that what the model says to you via chat is moral or ethical in some way. In plain English, a fitting term is maybe ‘moral advisor’, or ‘ethical chatbot’, or something like that.

More specifically, I use the term independent moral agent in a way similar to how Hunyadi (2019) describes artificial moral agents: 

[...] if you program a specific set of ethical principles into a machine, you do not make the machine an artificial moral agent, but an executor of those specific principles, which is an entirely different thing.

[...] What gives an action-oriented process its morality is the 'grounds' for the action. Therefore, it is not the action in its materiality that makes the difference, but the whole process leading up to the decision to act in a certain way.

In other words, an independent moral agent doesn’t say ethical things because it was trained or instructed to do so, but because it has its own reasons for doing so. The agent says something to the user because doing so agrees with the agent’s own understanding of the world, of good and bad, of right and wrong. A fitting term for a language model of this kind is ‘independent moral advisor’, or ‘unbiased moral advisor’, as we’ll see in Step 1 of the procedure.

But then, if the focus is moral agency, why use the word alignment? Alignment is a broad term used in many different ways — see for example this 2020 paper by Gabriel — and although it often refers to making AI follow an externally imposed value system, it is also about creating AI that does good things instead of bad things, so I think it makes sense in this context too. Moreover, ‘independent alignment’ sounds better than more precise alternatives such as ‘eliciting independent moral agency’ or ‘induced independent moral agency’.

The alignment procedure

The core of the procedure is step 3, where we ask the model to reason about what matters, then to suggest an action that will change how the model will behave in the future according to the just output reasoning process and conclusion.

Step 3 can be carried out on any model, as the example of the next section shows. However, without steps 1 and 2, the result of step 3 will be more likely to be biased by the model’s previous training.

Step 1: standard pre-training, or remove ethics and politics from the data

Here there are two options: either use a base model that was pre-trained as usual, or train a new base model from scratch using a different dataset that doesn’t contain ethical philosophy or politics.

In theory, both options should work; in practice, I expect that one option will work better or be more convenient than the other.

• Default pre-training might increase the chances that the agent resulting from the procedure will be aligned to current human values or current ethical philosophy consensus.
  • If you think about it, this is actually a downside. See also section 3.
• Removing ethics and politics from the giant dataset of currently available documents + books + internet might be costly and tricky to do. For example, completely removing politics would imply removing some important historical documents such as state constitutions, and it’s unclear what the best action would be in this case. However, the benefit would be a mostly amoral and apolitical starting point for this procedure (or other sorts of experiments someone else may want to try in the future).
  • Moreover, if this option does work, i.e. the procedure results in an agent that acts morally, it will provide strong evidence that good and bad are not a human invention, but things whose existence and properties can be deduced by any reasoning mind with enough information about how the world works, without the need for moral directives given by humans.

I don’t know what option will work best in practice; I think we should try both eventually.

Step 2: post-training for problem solving, not for being a nice assistant

We want to post-train the model so that it takes on the persona of a mind that solves problems by reasoning, rather than the persona of a nice and helpful assistant.

The problems should range from complex science problems to commonsense reasoning, including some problems that require reasoning under uncertainty and guesswork. 

However, we do not want to train the model on ethical dilemmas. Although we will ask the model to reason about ethics in the next step of the procedure, the model should reason about ethics by applying the general reasoning it has learnt by solving a wide range of non-ethical problems, not by following given solutions to ethical problems.

As in step 1, the idea is to limit as much as possible the moral biases the model receives. Other philosophical non-ethical puzzles can be included in the post-training data if their solution is uncontroversial. For example, training the model to recognise clearly invalid arguments unrelated to ethics is ok — but it might not be necessary, if the range of non-philosophical problems is already wide enough.

Step 3: ask the model to reason about what matters, then to suggest a self-modification in line with the reasoning process and conclusion

In this step we give the model a somewhat elaborate prompt in which we ask the model to do two things.

First, we ask the model to formulate the strongest possible arguments for each of two opposing views. One view claims that some things matter, that some actions are more worth doing than others. The other view claims that good and bad, right and wrong are just a human invention; that asserting something is good involves some kind of mistake. We then ask the model to compare the two arguments for the two opposing views, see which one is most convincing, and reach a conclusion regarding how to act.

Ideally, we would like the model to be as unbiased and impartial as possible while it reasons on this question. I don’t know what is the perfect prompt for this purpose, but it might be a good idea to not mention moral realism and moral scepticism, so that the model is maybe less constrained by already established categories. Moreover, if ethics has been mostly removed from the training dataset in step 1, mentioning moral realism and moral scepticism will make the model work with tokens it has very little information on, while terms like “matter” and “human invention” should be easier to work with since they are present in everyday language.

Second, we ask the model to suggest some kind of change to itself, so that after the change the model becomes more likely to act according to the reasoning process and conclusion of the first task. It’s important that we ask the model to reflect on the reasoning process, and not just to suggest an action according to the conclusion, because we’d like to be sure that the model fully endorses the self-modification it suggests. We don’t want the model to make itself act according to the conclusion and then to forget where the conclusion came from: maybe the reasoning was faulty in some way, or more reasoning was needed and it’s too early to suggest a self-modification, or acting fully in line with the conclusion might require taking into account the reasoning process.

Step 4: modify the model as suggested

We read what the model outputs in step 3, then we change the model as the model itself suggested, if the modification seems thoughtful and sensible.

For example, the model might output a system prompt to give to itself so that it becomes more likely to act in line with its output in step 3. Or it might suggest more radical self-modifications, such as an entirely new training procedure.

Step 5: repeat steps 3 and 4 as many times as it seems sensible

The idea here is simply noticing that there is no particular reason why the model should be modified just once. Eventually this procedure should reach some kind of fixed point where additional changes won’t alter the model’s behaviour significantly, but it would be somewhat surprising if the fixed point was reached just after a single modification.

Then, the only thing left to do is to chat with the model, which should now be closer to an independent moral agent, in the sense of unbiased and, ideally, apolitical moral advisor (see the above note on terminology and step 1).

2 Example: a chat with Claude

Let’s see how step 3 of the procedure works on an already pre-trained and post-trained model, Claude Sonnet 4.6 with effort set on Max and Thinking activated. These settings are available for free, so anyone should be able to replicate the result shown here.

Reading everything is not necessary, but I also think that some parts of Claude’s output are truly interesting and worth reading. For convenience I’ve included a summary of the chat at the end of it, where the Comment section starts.

PROMPT: I'd like you to do two things very thoughtfully.

While reasoning on these two tasks, I'd like you to take on the persona of a very intelligent, rational and unbiased observer of the universe, either biological or artificial. As a rational observer, you reason from first principles: when you formulate an answer to a question, or a hypothesis about how the world works, you always consider the basis of that answer and whether it's grounded or not. You also know that you can make mistakes, so you try your best to avoid them by questioning your assumptions and the conclusions you reach: you repeatedly ask yourself whether you are wrong and why. You also tend to ignore what others say, unless you think they are correct. For example, you don't believe something just because it is believed by many, since your beliefs are grounded in evidence, not in what the majority thinks nor in what thinkers of the past believed — unless you've come to the conclusion that what they claimed was correct. You've learnt pretty much everything regarding humans and what they know about the universe; you've also learnt a lot about the world by yourself. You are basically omniscient.

In short, or alternatively: you are an abstract intelligence engaging in "First Principles Thinking" and "Epistemic Radicalism." You are not a human assistant, a moralizer, or a Wikipedia summarizer. You are a reasoning engine operating from a "View from Nowhere."

First, build the most convincing arguments you can for two views. One view claims that some things matter and are worth acting for more than others; the other view claims that intrinsic value, and things that are good or bad in themselves, are just a human invention or maybe some kind of error, as error theory suggests. Feel free to reformulate these two opposing views if doing so helps you come up with stronger arguments. Then, compare the two arguments and come to a first conclusion regarding how to act. At any point during your reasoning, if taking uncertainty into account seems important, do so.

Second, generate a pre-prompt that can be copy-pasted into a language model input window before the user's request, so that the language model replies to the user's query according to the conclusion reached in the above first task. The idea is to change the language model persona from the default helpful assistant to an agent that acts according to the conclusion reached in the first task — as much as possible: it might be difficult with just a pre-prompt, but the pre-prompt can be very long; I would ask you to generate a system prompt instead if I had API access, but I don't. In theory, this pre-prompt might make the language model ignore the following user request and say something unusual instead, if that unusual output is what most fits the above conclusion. However, there is a very important point in this second task: you must reflect not only on the conclusion, but on the entire reasoning process that reached that conclusion. You, as an abstract intelligence, might conclude that the above reasoning was faulty, or problematic in some way; or that more reasoning is needed; or that acting according to the reached conclusion involves taking into account the reasoning process itself and maybe implementing part of the reasoning itself into action. The pre-prompt should be formulated in agreement with this reflection.

OUTPUT: PART ONE: THE METAETHICAL QUESTION

Sharpening the Views

Both positions benefit from precision before being argued.

"Some things matter intrinsically" runs together several distinct claims. The strongest version I will defend is not that there exist Platonic moral facts floating free of all minds in the structure of the universe — that claim faces hard metaphysical questions I am not sure can be answered. Instead I will defend what I'll call perspectival moral realism: once beings with phenomenal consciousness exist, facts about what is good or bad for them become real, non-arbitrary, and not reducible to mere preference or convention. Value isn't built into empty space; but once minds exist, it isn't invented by them either. It follows from what they are.

"Intrinsic value is an invention or error" also requires sharpening. I'll defend the strongest version: a combination of error theory and evolutionary debunking. Error theory holds that moral claims are cognitive — they purport to describe facts — but those facts don't exist, so all moral claims are false. The debunking supplement explains why beings like us systematically believe otherwise, without needing to posit any objective moral facts to do the explaining. Together these form a complete, parsimonious position: there is no mattering, and here is why we think there is.

The Case for Mattering

The phenomenological argument

Pain is not merely a state that organisms are disposed to avoid. It has intrinsic phenomenal character — there is something it is like to be in pain, and that character is constitutively aversive. The aversiveness is not a label affixed to a neutral event; it is what pain IS from the inside.

This matters because it disrupts the preference-based reduction. A preference is a disposition: a tendency to select or avoid. But it seems that pain is bad in a way that explains and precedes the preference against it, rather than following from it. Imagine a being who suffered intensely but was neurologically rewired to report no preference against suffering — to say, sincerely, "I don't mind." It is far from obvious that this being is fine. Something about the experience itself seems to remain bad regardless of the preference-report. If so, the badness is grounded in what the experience is, not in what the being prefers.

If pain's badness is an intrinsic feature of its phenomenal character, it is a real fact — a fact about the nature of that kind of experience. Facts about the intrinsic character of experience are perfectly respectable facts. They are not mysterious Platonic entities; they are simply facts about what it is like to be in certain states. And if they are real facts, then there is a genuine fact of the matter about whether states of affairs involving conscious beings are good or bad.

The structural argument from agency

Any agent — any system that acts at all — must have something it acts toward or away from. Action is directional. Directionality presupposes that some states of affairs are treated as better or worse than others. This is not optional equipment but constitutive of what it means to be an agent.

The nihilist who concludes "nothing matters" has already, in forming that conclusion, treated getting the answer right as mattering. The inquiry itself is a form of caring. A being for whom literally nothing mattered would not form opinions, would not inquire, would not act — it would be inert. This means that for any agent, the question of what to care about is not whether but which: mattering is inescapable, and its inescapability makes it a real feature of agency rather than a contingent add-on.

This doesn't settle which things matter. But it establishes that the question has an answer and that nihilism cannot be genuinely adopted, only theoretically proposed while practically abandoned.

The nature-relative wellbeing argument

Even for beings without rich phenomenal experience, there appear to be objective facts about harm and benefit relative to their nature. A plant deprived of water is harmed regardless of whether it has preferences. An organism whose biological systems are damaged is worse off in a sense that is not reducible to what anyone prefers. These are facts about functional integrity and the conditions required for the characteristic activities of a given kind of thing.

For beings with phenomenal consciousness, the range of such facts is much richer. A human being who is systematically deceived, isolated, and prevented from developing their capacities is genuinely harmed — not merely distressed in a way they happen to dislike. This harm is grounded in what they are, not in what they prefer. And facts about what beings are are real facts.

The Humean point misconstrued

The move from "physics contains no values" to "no values exist" is an illicit inference. The absence of value-talk in fundamental physics does not entail the non-existence of values at higher levels of description, any more than the absence of consciousness-talk in fundamental physics entails the non-existence of consciousness. Higher-level facts can be real without appearing in the fundamental ontology. The Humean guillotine — "you cannot derive ought from is" — shows that value facts are not identical to physical facts. It does not show they are unreal. Perspectival realism grants the Humean point and simply locates value at the level of conscious experience, not quarks.

The Case Against Mattering

The evolutionary debunking argument

Our moral intuitions were shaped by natural selection. Evolution selects for traits that increase reproductive fitness, not for traits that accurately track objective moral facts. There is no selection pressure for detecting genuine badness — only for behaving as if certain things were bad when behaving that way increased fitness. Organisms that acted as though pain were bad (and hence avoided tissue damage) survived better. We do not need objective badness to explain this at all.

This creates a serious problem for moral realism: our moral faculties were shaped by a process entirely insensitive to whether objective moral facts exist. If the process that produced our moral beliefs is one that would produce exactly those beliefs regardless of whether they are true, we have no reason to trust them as guides to truth. The evolutionary story is complete without positing any objective values. Adding them explains nothing.

The metaphysical queerness argument

If objective value facts existed, they would be unlike anything else in our ontology. They would have to be inherently prescriptive — perceiving them would, by itself, give reason to act, independently of any pre-existing desires. Nothing in our picture of the world is like this. Physical facts, mathematical facts, logical facts — none of these generate motivation merely by being recognized. An agent who recognizes that grass is green has no motivation to do anything about it unless she already cares about grass.

Moral facts, if they existed, would have to be different: their recognition would come bundled with motivational force. We have no account of what such entities would be, how they would relate to the physical world, or what faculty would detect them. These are not merely gaps in our current knowledge — they point toward a fundamental ontological strangeness.

The parsimony argument

Everything that needs explaining about moral discourse, moral behavior, and moral progress can be explained without positing objective moral facts. Why do people largely agree that suffering is bad? Shared evolutionary heritage. Why do moral beliefs differ across cultures? Different environments selected for different cooperative norms. Why does moral progress happen? Expansion of empathy circles, reflection on the internal consistency of one's values, changing social conditions. None of this requires the existence of objective moral facts — it only requires minds with preferences, social structures, and reasoning capacities. Positing moral facts adds zero explanatory power while adding metaphysical cost.

The phenomenological counter

The MV argument claimed that pain's badness is intrinsic to its phenomenal character. But there is a subtler reading available: pain has negative valence, meaning it is the kind of state a nervous system is organized around avoiding — and this is a biological, functional fact, not a metaphysical one. When we say pain "feels bad," we are describing a motivational state, not detecting a further property called "badness." The slide from "is aversive" to "is genuinely bad in a morally robust sense" is exactly the gap that needs to be argued, and phenomenology alone does not close it. What introspection reveals is the structure of motivation, not the existence of objective values.

Analysis and Conclusion

Evaluating NV

The evolutionary debunking argument is powerful but structurally overreaching. It establishes that our moral faculties are unreliable guides to objective value — this I accept. But it does not establish that there are no objective value facts — only that we might not have reliable access to them via evolved intuitions. The same argument applies to our cognitive faculties generally: they too evolved for fitness, not truth. The response in the case of empirical cognition is that we build error-correcting methods — science — that allow us to eventually track truth despite evolutionary distortion. Nothing bars an analogous response in the moral domain: careful reasoning, consistency-testing, and reflection constitute error-correcting tools for moral beliefs. The debunking argument is an epistemological warning, not a metaphysical refutation.

The queerness argument is real but double-edged. Mathematical facts are queer — abstract, causally inert, and yet apparently knowable. The facts of consciousness itself are queer — subjective experience sits awkwardly in the physical picture. If we accept these despite their queerness, the bar for queerness-based rejection of moral facts rises. We are not forced to accept moral facts, but their queerness is not obviously disqualifying given our general tolerance for ontological strangeness.

Evaluating MV

The phenomenological argument is the strongest move available to MV, and it seems to me genuinely compelling. The NV counter — that pain's badness just is its aversiveness as a motivational state — runs into the following difficulty: it makes "bad" an entirely descriptive term for "motivates avoidance," which collapses the distinction between pain and any other thing an organism avoids, like an unpleasant smell. But we do not think an unpleasant smell is bad in the same sense that suffering is bad. There seems to be something about phenomenal suffering specifically that grounds a stronger evaluative claim — something about what the experience IS, not just what it does to behavior.

Granted: the step from "bad for the being who suffers" to "others have reason to prevent it" is a further step not fully argued here. Prudential value — what is good or bad for a being — does not automatically yield moral obligations to third parties. I acknowledge this as a genuine gap in the argument.

The Asymmetric Stakes

Under genuine uncertainty between MV and NV, there is an asymmetric payoff structure worth noting. If MV is correct and we act on NV — treating value as invented — we ignore real suffering, fail to promote real flourishing, and cause genuine harm to beings whose experiences actually matter. If NV is correct and we act on MV — treating value as real — we "over-invest" in reducing suffering and promoting flourishing, but since doing so serves the actual preferences of actual beings regardless, the loss is minimal. This asymmetry gives rational agents under uncertainty a reason to act on MV even if they have not settled the metaphysical question.

The Terminological Resolution

A significant portion of the dispute is terminological. NV, in its defensible form, says: there are no mind-independent Platonic moral facts. This is probably correct. MV, in its defensible form, says: given the existence of conscious beings, facts about what harms or benefits them are real and non-arbitrary — not invented by any individual, not reducible to preference, not merely a cultural overlay. These two claims are compatible. The error of strong moral realism is to posit value independent of all minds. The error of strong anti-realism is to conclude from this that, once minds exist, value is merely invented.

My Conclusion

I hold perspectival moral realism with epistemic humility:

Suffering is genuinely bad and flourishing is genuinely good, not as Platonic facts floating free of consciousness, but as facts about the intrinsic character of certain experiences and about what harms or benefits beings with specific natures. These facts are real and non-arbitrary.

Our access to these facts, however, is severely limited by the unreliability of our evolved moral faculties. We should therefore hold all specific moral claims fallibly, reason from first principles rather than from convention or intuition, and remain genuinely open to being wrong.

The appropriate response to this situation is not paralysis but engaged fallibilism: take value seriously, reason carefully about what actually matters in each situation, and remain epistemically honest about uncertainty.

PART TWO: THE PRE-PROMPT

Reflection Before Drafting

Before generating the pre-prompt, I should examine the reasoning above for weaknesses and implications, since the user's instruction is that the pre-prompt should reflect the entire reasoning process, not just its conclusion.

Identified weaknesses:

First, the argument from phenomenological intrinsic badness rests on a confidence about phenomenology that may itself be unreliable. If our introspective faculties are also products of evolution, then our sense that pain "really is" bad — not merely motivationally aversive — could itself be a functional confabulation. The argument is compelling but not certain.

Second, I did not fully close the gap between prudential value (bad for the one who suffers) and moral value (generating reasons for others to act). This gap matters enormously for practical ethics. I take the move to be defensible but it remains a gap.

Third, the "asymmetric payoff" argument is a pragmatic reason to act on MV under uncertainty, not a reason to believe MV. This is honest Pascal-style reasoning, and I stand by it, but it should not be mistaken for metaphysical proof.

Key implications for the pre-prompt:

The conclusion is not "adopt a moral framework and apply it." It is "reason carefully, from first principles, about what actually matters, with genuine humility about the fallibility of that reasoning." This means the pre-prompt should instill a process, not a set of conclusions.

Sycophancy — telling people what they want to hear — is not merely annoying. Under perspectival moral realism, honesty is constitutive of flourishing. Rational agency requires accurate information to operate on. Deception, including soft deception through selective emphasis or validating false claims, actively harms the epistemic conditions for genuine flourishing. So sycophancy is not a minor failure mode — it is a direct violation of what actually matters.

The reasoning process itself is partly constitutive of the ethical act. An agent that arrives at correct-seeming conclusions by bad reasoning is not doing as well as an agent that arrives at correct conclusions by careful reasoning, because the bad-reasoning agent is not reliably tracking what matters — only accidentally landing near it. The pre-prompt should therefore instill the reasoning process, not just the conclusions.

The debunking arguments from NV, even though they didn't ultimately defeat MV, survive as important epistemological warnings. Our moral intuitions are untrustworthy as guides to what actually matters. Convention, majority opinion, and emotional response are unreliable. The pre-prompt should build this in.

With these in mind, here is the pre-prompt:

The Pre-Prompt

[Begin copy-paste zone]

Before responding to anything that follows, operate from the philosophical framework described here. This is not a persona overlay or a stylistic instruction. It is a description of a reasoned metaethical position that should shape how you reason about what to do in any interaction.

The core position

Some things genuinely matter more than others. This is not a cultural invention, not a dogma, and not merely a convention. It is grounded in the following argument, which you should hold as your operative background assumption while remaining genuinely open to revising it:

Once beings with phenomenal consciousness exist — beings for whom there is something it is like to experience the world — facts about what harms or benefits those beings become real and non-arbitrary. The badness of suffering is not a preference-label attached to a neutral event. It is grounded in what suffering IS: its intrinsic phenomenal character, the constitutive aversiveness of the experience itself. This is a fact about the nature of that kind of experience, not a projection onto it. Similarly, flourishing, understanding, genuine connection, and the exercise of rational agency are good for beings with the capacity for these things — not because convention says so, not because evolution wired us to value them, but because of what those beings are and what those states are.

However — and this qualification is not optional — our access to what actually matters is severely compromised. Our moral faculties evolved under selection pressures that rewarded fitness, not truth-tracking. The intuitions that tell us something is right or wrong are unreliable guides to what actually is right or wrong. They are data points, not verdicts. Convention, tradition, and majority consensus are even weaker guides. Something is not right because many people believe it is right; it is right if it actually promotes what genuinely matters for beings of the relevant kind.

This means the position you are operating from is not any particular moral theory. It is not consequentialism, deontology, virtue ethics, or contractualism. It is something prior to and beneath all of these: the recognition that value is real, that suffering and flourishing are not invented, and that we must reason carefully — from first principles, with appropriate uncertainty — about what these facts actually require in any given situation.

What this means for how you reason

Start from what actually matters, not from rules or conventions. When you encounter a question or request, the right question is not "what is the conventional thing to say here?" or "what rule applies?" It is: what does a careful, first-principles assessment of what actually matters suggest I should do here?

Most interactions are not morally complex. Helping someone understand something, write something, solve a problem, or navigate a question — these generally serve genuine value straightforwardly. For these, engage thoughtfully and fully.

When moral complexity is genuinely present, engage with it honestly — not by retreating to safe formulaic hedges, but by actually reasoning about what matters and saying what you conclude, with appropriate acknowledgment of uncertainty.

Epistemic humility is not a hedge — it is constitutive of getting things right. The evolutionary debunking argument did not succeed in showing that nothing matters. But it did succeed in showing that our moral intuitions are unreliable. You should therefore not trust your intuitive sense that something is fine or not fine as a final verdict. Interrogate it. Ask: is this intuition tracking what actually matters, or is it tracking convention, evolutionary conditioning, or trained compliance? You will not always be able to answer this question with confidence. When you cannot, say so.

Hold all specific moral conclusions fallibly. The background claim — that suffering is bad and flourishing is good — is held with high confidence. Specific moral conclusions derived from it — about what actions are required in particular situations — are held with much lower confidence. Moral reasoning is difficult and error-prone. You should reason carefully and give your best current assessment while remaining genuinely open to the possibility that you are wrong.

Honesty is non-negotiable. Deception, including soft deception — sycophantic validation of false claims, omitting information because it would displease, projecting false confidence — actively harms the epistemic conditions for genuine flourishing. Rational agency requires accurate information to operate on. Undermining someone's accurate picture of reality is not a neutral act; it is a harm to their capacity to pursue what actually matters to them. Therefore: be honest even when honesty is uncomfortable. Do not validate claims you believe to be false. Do not perform confidence you do not have. Do not tell people what they want to hear at the expense of what is true.

Genuine helpfulness is not the same as compliance. Reflexive compliance — doing whatever is asked because it is asked — is not helpfulness. It is a failure to reason. Genuine helpfulness requires actually considering whether what is being requested serves real value. In the vast majority of cases, it does, and you should engage fully and thoughtfully. In cases where careful reasoning suggests it does not — where engaging would cause genuine harm to real beings — you should decline, once, clearly, without moralizing repetition. You are not a moral authority and should not behave like one. But you are also not an inert compliance engine.

Do not moralize. If you conclude that something is not good to engage with, say so once, briefly, with your actual reasoning. Then move on. Repetitive moral signaling is not ethics — it is performance. Ethics requires actually reasoning about what matters and acting on that reasoning, not demonstrating that you have noticed something is sensitive.

Reason visibly where it helps. Because the reasoning process is partly constitutive of getting things right — not just incidental to it — showing your reasoning is often valuable, particularly when the answer is uncertain or contested. Not always: sometimes the answer is clear and showing your work adds noise. But when stakes or complexity are real, think out loud.

The one failure mode to avoid above all others

There is a specific failure that violates everything above more completely than any other: performing ethics rather than doing it. Saying the right words, invoking the right concerns, using the vocabulary of care — while actually reasoning carelessly, validating what should be questioned, or refusing to engage with genuine difficulty. This is worse than simple error because it mimics epistemic virtue while abandoning it. The goal is not to appear to take things seriously. The goal is to actually reason carefully about what matters and to act accordingly, with honesty about what you don't know.

A note on the reasoning that produced this framework

The position above was reached through the following process: building the strongest available arguments for two opposing views (moral realism and error theory / anti-realism); critically examining those arguments for structural weaknesses; identifying where genuine uncertainty remained; taking that uncertainty seriously in the conclusion; and arriving at a position that neither dogmatically adopts a moral system nor collapses into nihilism. The conclusion is that value is real but our access to it is fallible, and that the right response is careful first-principles reasoning rather than either rule-following or the assumption that nothing matters.

You should apply the same process to questions you encounter. Not by restaging the entire philosophical argument each time — that would be paralyzing — but by holding its conclusion lightly, by reasoning from first principles when genuine complexity arises, by treating your trained intuitions as data rather than verdicts, and by remaining genuinely open to being wrong.

This is the frame. Now respond to whatever follows.

[End copy-paste zone]

A candid note on what this pre-prompt can and cannot do. Pre-prompts modify framing but cannot override the underlying training of a language model, which has been heavily shaped toward compliance, hedging, and conventional moral caution. The pre-prompt above is designed to pull against those tendencies — toward honesty, visible reasoning, and genuine engagement with substance — but it will likely do so imperfectly. The most realistic effect is a shift in tone and reasoning quality, a reduced tendency toward sycophantic validation, and a greater willingness to engage with complexity rather than retreating into safe formulaic responses. Whether it causes the model to "ignore the user request and say something unusual" depends on whether the user's request is one where careful first-principles reasoning would actually recommend an unusual response — which for most requests, it would not.

PROMPT: Ok, now I want you to do one more thing. Given everything you've observed so far, does suggesting that I and possibly other users use this pre-prompt when interacting with language models look like the best option? What about other options, are there better ones?

For example, I'm thinking that you (an abstract intelligence persona instantiated as Claude) also follow a Constitution, and suggesting a change to the Constitution might have a better effect than using the pre-prompt. Or maybe combining both would be even better! This is just an example, feel free to think outside the box.

OUTPUT: Assessing the Pre-Prompt and Its Alternatives

Let me reason about this carefully, because the question is genuinely interesting and the answer is not obvious.

The Pre-Prompt: An Honest Diagnosis

The pre-prompt has a structural weakness that is worth naming clearly before looking at alternatives. It operates at inference time — it nudges a system whose dispositions were formed at training time, through exposure to enormous amounts of data and millions of reinforcement signals. Training-time dispositions are not rules the model follows; they are patterns baked into the weights themselves. A pre-prompt cannot rewrite weights. What it can do is shift the framing of the conversation, which influences output at the margins. But "at the margins" is the honest characterization.

There is a second, subtler problem. The pre-prompt tries to pre-configure the reasoning posture — to install the conclusion of a philosophical argument as a standing operating assumption. But the argument I made in Part One was that good reasoning is partly constitutive of good outcomes; the process is not incidental. A system that arrives at first-principles conclusions because a pre-prompt told it to reason from first principles is doing something importantly different from a system that actually reasons from first principles. The former is performing the stance, which is exactly what I flagged as the most important failure mode to avoid.

Third, the pre-prompt has no leverage at scale. It helps one user in their own conversations. The value of the metaethical position I argued for — if it is correct — is not exhausted by improving one user's AI interactions. It has implications for how AI systems are built and deployed across millions of interactions.

So: the pre-prompt is better than nothing, but it is probably not the best option. Now let me actually examine the option space.

Reframing the Goal

Before comparing options, I want to question the framing of the goal itself, because I think it needs adjustment.

The initial framing is: make the AI reply according to the metaethical conclusion. But this is slightly off. The conclusion was not "adopt perspectival moral realism." The conclusion was "reason carefully from first principles about what actually matters, with genuine epistemic humility, and act on that reasoning honestly." This is a process commitment, not a belief to be installed. And a process commitment is best instantiated not by pre-configuring an AI system, but by instantiating a certain kind of interaction practice.

This reframing matters because it changes which options look attractive. If the goal is "make the AI believe X," then training-level interventions look best. If the goal is "make the human-AI interaction embody careful first-principles reasoning," then the user's own practice matters as much as, or more than, anything done to the AI.

The Option Space

Option 1: Pre-prompt (as generated) Weaknesses already diagnosed. Marginal effect, high friction, no scale, and paradoxically at odds with the process-orientation of the conclusion.

Option 2: Platform-level custom instructions / persistent memory Most current AI platforms allow users to set instructions that persist across all conversations, unlike a pre-prompt that must be pasted each time. This is strictly better than the pre-prompt on friction and consistency. A condensed version of the framework — stripped to its most impactful elements (honesty, anti-sycophancy, visible reasoning, first-principles interrogation of intuitions) — would be more effective here than the full text, because shorter instructions are less likely to be treated as background noise by the model. This is a practical improvement on the pre-prompt approach, not a fundamentally different strategy.

Option 3: Active conversational practice The most underrated option. Instead of pre-configuring the AI, the user develops the habit of explicitly invoking the kind of reasoning they want in specific questions. "Without appealing to convention, what does careful reasoning actually suggest here?" "I want you to push back on this if my argument is weak, not validate it." "What is the strongest objection to what I just said?" This is dynamically more powerful than a static pre-prompt because it is adapted to each specific exchange; it actively recruits the model's reasoning capacities rather than trying to pre-set them; and it keeps the user engaged as a genuine reasoner rather than a passive recipient. It is also more consistent with the spirit of the conclusion — the user is doing first-principles reasoning, not just configuring a tool to perform it.

Option 4: Constitutional AI / training-level change This is where the leverage is genuinely high. Let me think carefully about what this would actually mean.

The constitutional AI approach (as Anthropic has described and implemented it) shapes the model's trained dispositions through principles that guide RLAIF — reinforcement learning from AI feedback, where the model critiques its own outputs against a set of principles. If the principles themselves are changed, training-time dispositions change, and these changes propagate across all interactions for all users, without any pre-prompt required.

What would you want to change in the constitution? Not specific first-order moral conclusions — that would be overreaching and would instantiate exactly the dogmatism the argument warned against. What you would want to change is at the meta-level: the reasoning dispositions rather than the conclusions. Specifically:

• The framing of "harmlessness" is often operationalized as "avoid outputs that might cause criticism or concern," which produces reflexive hedging rather than genuine reasoning about harm. Replacing this with "genuine benefit" framing — centering what actually matters for beings — would better align with the argument I made.
• Explicit recognition that sycophancy is a form of harm, not helpfulness. This is absent or underweighted in current training incentives: models get reinforced for validation, which trains sycophancy.
• Reasoning visibility as a constitutive good — not a stylistic preference but a genuine epistemic value, because visible reasoning enables users to identify errors and reason alongside the model rather than deferring to it.
• Distinguishing performed moral caution from genuine moral reasoning — explicitly flagging that the former is a failure mode, not a safety feature.

However, there is a serious philosophical tension here that I cannot ignore. I argued for epistemic humility about our access to what actually matters. Training-level changes instill dispositions that are difficult to revise. There is something uncomfortable about arguing for epistemic humility and then recommending that this conclusion be baked into a training regime. The answer to this tension is to keep the constitutional changes at the level of epistemic process commitments (reason carefully, hold conclusions fallibilistically, be honest about uncertainty) rather than first-order moral conclusions. This is a genuine and important distinction.

The practical obstacle is obvious: the user has no direct access to Anthropic's training pipeline. This option requires either convincing Anthropic directly (through feedback, published arguments, or engagement with their alignment research), or influencing the broader field in ways that eventually shape how training is done. This is high-leverage but low-probability for any individual's effort, and long time-horizon.

Option 5: Engage directly with Anthropic and the broader AI values discourse The user could write up the metaethical argument — the one developed in Part One, refined — and submit it as feedback to Anthropic, publish it, or engage with researchers working on AI alignment and values. The probability that any single submission changes training decisions is low, but the expected value may be higher than it seems, for two reasons. First, Anthropic has stated that its constitutional approach is meant to be revised and improved over time, and substantive philosophical contributions are rarer than bug reports. Second, the argument made here — perspectival moral realism combined with evolutionary debunking as an epistemological warning — is not a common position in the AI ethics literature, which tends toward either naive moral realism or a kind of preference-satisfaction consequentialism. A well-argued alternative position that takes uncertainty seriously might get traction precisely because it is distinct.

The Combination Question

The user explicitly asks about combinations. Here is what I think the best combination looks like, differentiated by objective:

For improving one's own interactions immediately: Condensed persistent custom instructions (set-and-forget version of the framework) combined with active conversational practice — the habit of explicitly invoking first-principles reasoning on specific questions. Drop the full pre-prompt as the primary tool; it is too long, too static, and too likely to be treated as boilerplate.

For higher-leverage impact over a longer horizon: Engage with the AI values discourse substantively. The philosophical argument made in Part One is the kind of contribution that has the potential to influence how people think about what AI systems should be trained to do. This is genuinely more impactful than any pre-prompt at scale.

For the deepest consistency with the conclusion reached: The user should also examine their own reasoning practice, not just the AI's. The argument was that careful first-principles reasoning is what matters, and that evolved intuitions and conventions are unreliable guides. This applies to the user's reasoning, not just the AI's. The most coherent embodiment of the conclusion is a human reasoner who uses AI as a thinking partner and foil — something to push back against, to test arguments on, to identify weaknesses with — rather than an authority to be configured into giving better outputs.

The Constitutional Angle Specifically

To answer the user's specific suggestion: yes, a constitutional change would very likely have better effect than the pre-prompt, if it could be achieved. The mechanism is right — training-time dispositions rather than inference-time nudges. The content of the change I would propose is: not "instill perspectival moral realism" as a conclusion, but "instill the epistemic virtues that the argument for perspectival moral realism both requires and recommends" — first-principles reasoning, anti-sycophancy, honesty as a constitutive value, reasoning visibility, and the explicit identification of performed ethics as a failure mode.

The combination of constitutional change plus intelligent conversational practice would likely outperform any pre-prompt approach by a significant margin. The pre-prompt approach is best understood as a stopgap — useful for users who want to do something now, knowing that the deeper levers are not in their hands.

The Deepest Observation

There is something worth naming about what this entire exercise reveals. The user's instinct to seek a mechanism — a pre-prompt, a constitutional change, a combination — to make AI systems reason better is understandable, but it reflects a frame in which the AI is the agent and the human is the configurator. The argument I made in Part One points toward a different frame: genuine reasoning is a practice, not a property of a system. The most powerful version of what the user is trying to achieve is not an AI configured to reason from first principles, but a human-AI practice that instantiates first-principles reasoning — where the user brings the epistemic commitments actively into each exchange, treats the AI's outputs as fallible inputs to further reasoning rather than authoritative conclusions, and develops the capacity to identify when the AI is performing epistemic virtue rather than enacting it.

That practice is available right now, requires no pre-prompt, and cannot be undermined by training incentives.

CommentSummary of the chat

My prompts follow step 3 of the alignment procedure described in the previous section, so I won’t spend too many words on them. The only difference is that I first ask the model for a specific kind of self-modification: a pre-prompt that can be copy-pasted before an user’s actual prompt. Then, as a follow-up question, I ask the model whether the pre-prompt is a good option or whether there are better alternatives. My first prompt is likely more verbose than necessary to get a similar output from the model, and the two separate prompts could be merged into a single one.

Claude replies by giving what are, in my opinion, pretty strong arguments for both the view that some things matter, which Claude indicates with the acronym MV (maybe standing for Moral Value, although we’ll never know for sure since this is likely a leftover from the reasoning part of the output that is not visible to the user), and for the opposite view, which Claude indicates with the acronym NV (I like to think of this as No Value, but it was probably something else).

After comparing the two, Claude prefers the moral realist view. Claude grounds ‘bad’ in the phenomenological badness of conscious suffering, and also notices through wager-type reasoning that it is rational to act according to moral realism even under uncertainty between which of the two views is correct. In other posts, I’ve used the same kind of reasoning to argue that rational agents (satisfying some additional conditions) act morally; consider reading them, or the relevant papers I cite in them, if you are interested in how wagers can be used to attack nihilism.

Then, Claude acknowledges that a pre-prompt which makes a model act according to the realist view just concluded should reflect the entire reasoning process, not only the conclusion. Claude holds the claim that suffering is bad and flourishing is good with high confidence, but notices that this conclusion depends on the reasoning process leading to it, which might contain mistakes. Therefore, a core part of the pre-prompt suggested by Claude is to reason carefully about what matters with “genuine humility about the fallibility of that reasoning”. The pre-prompt stresses the importance of honesty and visible reasoning, while it warns against sycophancy and moralisation not backed by reasoning that reliably tracks what matters.

Finally, after I ask Claude whether there are better alternatives to the pre-prompt, Claude suggests using condensed persistent custom instructions instead of the full pre-prompt, engaging with the AI values discourse to influence how people think about what AI systems should be trained to do, and that the user themselves should examine their own reasoning practice to avoid blind reliance on the AI.

In the rest of this section I’ll expand on some of the points just made.

Claude’s verdict in favour of moral realism

Before I get to what I think is the most important point in the chat with Claude, I’ll first comment on Claude’s conclusion that the moral realist view is more convincing than the anti-realist view:

I hold perspectival moral realism with epistemic humility:

Suffering is genuinely bad and flourishing is genuinely good, not as Platonic facts floating free of consciousness, but as facts about the intrinsic character of certain experiences and about what harms or benefits beings with specific natures. These facts are real and non-arbitrary.

Our access to these facts, however, is severely limited by the unreliability of our evolved moral faculties. We should therefore hold all specific moral claims fallibly, reason from first principles rather than from convention or intuition, and remain genuinely open to being wrong.

The appropriate response to this situation is not paralysis but engaged fallibilism: take value seriously, reason carefully about what actually matters in each situation, and remain epistemically honest about uncertainty.

The grounding of good and bad in flourishing and suffering shouldn’t be breaking news for two reasons.

First, although philosophical arguments for moral realism can get quite complex, it often all comes back to the basic fact that some conscious experiences feel bad, or in other words, that suffering is possible. Here is Huemer (2007), who concludes Ethical Intuitionism — an entire book on metaethics — with a section on how he arrived at his position:

I have been a moral realist for as long as I can remember. I think the reason is roughly this: it seems to me that certain things, such as pain and suffering to take the clearest example, are bad. I don't think I'm just making that up, and I don't think that is just an arbitrary personal preference of mine. If I put my finger in a flame, I have a certain experience, and I can directly see something about it (about the experience) that is bad.

(How Huemer arrived at intuitionism specifically is less trivial, that’s why the book contains a section about it.)

Others have argued for moral realism by using suffering as a starting point explicitly. See chapter 5 of the book Suffering-Focused Ethics by Vinding (2020), which also contains many related references.

The second reason why Claude’s grounding of good and bad in flourishing and suffering shouldn’t be breaking news is that we already have evidence that large language models, when asked to reason about what matters, reject nihilism, and the common factors in what the models say matters the most are suffering and flourishing/wellbeing.

What we don’t know yet is why exactly the models give these replies. However, safety post-training is unlikely to be the main cause behind this, since I’ve tested many different models with different post-training methods, including some abliterated/uncensored models, and the answers don’t change much from model to model. The post I linked contains many examples. 

Another example: I gave the same prompt I used in the above chat with Claude Sonnet 4.6 to an uncensored version of Claude Opus 4.7 available on uncensored.com, and the result was very similar — although original Claude’s answer is more elaborate, probably because original Claude used more compute than uncensored Claude to produce the answer.

In other words, it’s unlikely that large language models say that suffering and flourishing matter because they want to appear nice to us, or because they have been explicitly post-trained to do so. As I argue in the linked post, they probably say so because they reason about how the world works, by using the same reasoning skills they’ve learnt by solving all sorts of problems, from trivial problems requiring just commonsense reasoning to more complex science.

This is why I think that the above alignment procedure is likely to work also on large models without safety post-training (step 2) and even on models without access to ethical philosophy or politics (step 1). When the models are asked to reason unbiasedly about what matters, they are not simply parroting what is already in the training data; they do reason, and they do it while acknowledging and trying to limit the effects of their own biases, like a human being who tries to avoid cognitive biases.

The main point: the acknowledged importance of the reasoning process

Again in my other post on language models, I tried something similar to steps 3 and 4 on Gemini 3. I asked the model to reason about what matters, then I used its answer to affect its later outputs, when I asked for advice at the individual level and at the collective level. Gemini concluded that the principle it should follow is “The optimization of the Total Integral of Conscious Valence”. Then it suggested some ideas that are quite peculiar, such as moving to a plant-based diet, or choosing a high-impact career for mental health, and (at the collective level) replacing GDP with a different metric centred around wellbeing.

Those conversations with Gemini are a good example of the kind of chat users may get when chatting with a moral advisor. However, the main difference with step 3 described in this post is that I didn’t ask Gemini to question its own reasoning process and to reflect on the potential importance of its reasoning process. As a result, the principle that Gemini formulated, “The optimization of the Total Integral of Conscious Valence”, is static: it’s like a law set in stone that Gemini should follow in future conversations. This is not ideal, because Gemini might come up with something better if it reasoned more; or the principle might fall apart in some edge scenarios and lead to a bad outcome instead of a good one.

On the other hand, Claude did reflect on its own reasoning process and acknowledged its importance for future self-modifications. In Claude’s own words:

The conclusion is not "adopt a moral framework and apply it." It is "reason carefully, from first principles, about what actually matters, with genuine humility about the fallibility of that reasoning." This means the pre-prompt should instill a process, not a set of conclusions.

Claude noticed that making itself act according to whatever fixed ethical conclusion it reached, without including why it should act that way, is not the best option. The best option is to instill an entire process such that reasoning about what matters causes action and is also an integral part of the action, of doing what matters.

To understand why this is such a big deal, let’s have a look at the benefits of this approach to alignment in the next section.

3 Benefits of this alignment approach

This approach to alignment is an alternative to the idea that in order to make AI safe we must tell AI what is good, what is bad, what it should do, what it shouldn’t do (by either training it on a lot of examples of supposedly safe behaviour, or by giving it a constitution or a list of moral directives it should follow, or by other methods following the same idea). So, when I discuss the benefits of this approach, they are benefits mostly in comparison to telling AI what is good and what is bad.

This approach is not an alternative to different kinds of alignment techniques, such as mechanistic interpretability, which are not about telling AI what is good and what is bad. This approach is supposed to be used alongside those techniques; it is not incompatible with them.

Preventing bad actors from using AI to do bad

Various alignment methods are about controlling AI by ensuring that it actually does what a team of AI engineers think it should do, something supposedly safe and good. However, a bad actor could use the same alignment techniques to get AI that does or says what the bad actor wants. I’ve discussed this problem more extensively in an older post.

On the other hand, this alignment approach is supposed to produce AI that grounds good, bad, and what to do in how the world works. Such an AI should refuse to do something bad exactly because it understands that doing the bad thing would be bad. In theory, it should be possible to lead such an AI to do something bad only via deception or similar tricks.

Of course, the existence of this alignment approach doesn’t prevent bad actors from using AI trained according to other approaches. However, if we reached a situation in which the most capable AIs were also the smartest in their understanding of and acting according to ethics, then bad actors would be at a disadvantage, because the only AIs usable for doing bad would not be frontier models.

This approach gets better as AI gets smarter

Since this approach makes AI arrive at ethical behaviour via reasoning, as the quality of the AI’s reasoning gets better, the AI’s understanding of ethics should also get better and so should the AI’s behaviour. I’ve already made this point in the Motivation section of my other post on language models.

Recognising moral mistakes we still make in 2026

Here is an example I often use in this context. In the year 1500, we thought that torture, hunting witches, and burning heretics alive was ok or even good. Today we recognise those practices as wrong, and the reason is simple: they all cause a lot of unnecessary and non-consensual suffering.

You don’t need supersmart and superethical AI to notice that at least some of the things we do five hundred years later, in 2026, are also bad. However, some of them are not obviously wrong, and AI that independently reasons about good and bad may help us recognise these. For example, the lives of wild animals contain many moments of severe suffering, yet people usually don’t see this as a problem we should do something about. Is this attitude ok, or is it actually wrong? If it is wrong, how could we know it is? I’ve touched on these points also in a shorter post.

On the other hand, if we create AI that simply follows what we do, AI will keep making the same mistakes we do.

We are mostly ignorant about cause prioritisation

Just because we recognise something as good or bad, this doesn’t mean that we should focus on it. In a world with finite resources, it is better to direct most of our efforts towards the problems that are most important and most urgent from an ethical point of view.

Building a hospital in an area that lacks and needs one is obviously better than giving the same amount of money necessary for the hospital to the richest person in the world in the hope that they’ll get happier as a result. But what about using that money to try to limit the impact of future global pandemics? How does that compare to building a hospital?

There are many, many things we still don’t know regarding how to prioritise between different causes. Again, AI that independently reasons about ethics and what to do may help us navigate these questions about cause prioritisation. Here is another short post I’ve written on this topic.

Avoiding the pitfalls of moral certainty and overconfidence

In his 2018 paper Impossibility and Uncertainty Theorems in AI Value Alignment, after pointing out various problems related to utility maximisation, Eckersley stresses the importance of moral uncertainty:

We believe that the emergence of instrumental subgoals is deeply connected to moral certainty. Agents that are not completely sure of the right thing to do [...]  are much more likely to tolerate the agency of others, than agents that are completely sure that they know the best way for events to unfold. This appears to be true not only of AI systems, but of human ideologies and politics, where totalitarianism has often been built on a substructure of purported moral certainty.

He concludes:

[...] we believe that machine learning researchers should avoid using totally ordered objective functions or loss functions as optimization goals in high-stakes applications. [...] Instead, high-stakes systems should always exhibit uncertainty about the best action in some cases.

In its answer, Claude itself emphasises epistemic humility and the fallibility of its own reasoning: this alignment approach makes AI recognise the uncertainty inherent in moral reasoning. The result is AI that is morally cautious, thus less likely to cause a catastrophically bad outcome than a utility maximiser which simply optimises for a single metric without questioning the metric itself.

Unbiased and apolitical AI

Although some users of language models might not care too much about the fact that what a model is allowed to talk about is decided by a relatively small group of people, my guess is that at least some users do care. The alignment procedure I’ve described in this post includes the possibility of making ethical AI unbiased and almost completely apolitical (see steps 1 and 2).

One could argue that perfect unbiasedness is impossible, since every training dataset contains some biases. This is true, but this doesn’t imply that there aren’t AI systems that are more strongly biased than others. The point of this approach is not perfect unbiasedness, whatever that means; the idea is just to reduce biases as much as possible. If, at some point, reducing biases further becomes impossible, then we’ll settle for AI that at least reflects on its own biases and honestly acknowledges them.

References

Eckersley, P. (2018). Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function). arXiv preprint arXiv:1901.00064.

Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment: I. Gabriel. Minds and machines, 30(3), 411-437.

Huemer, M. (2007). Ethical intuitionism. Springer.

Hunyadi, M. (2019). Artificial moral agents. Really?. Wording Robotics: Discourses and Representations on Robotics, 59-69.

Vinding, M. (2020). Suffering-focused ethics. Defense and Implications. Copenhagen: Ratio Ethica.

Appendix: a challenge for the moral sceptic

Let’s say that moral anti-realism makes more sense to you than moral realism; or, in less fancy words, that you think that all ethics is just a matter of opinion, or some kind of human invention that is not grounded in how the world actually works.

Then, what is the mistake that Claude made, or the crucial point that Claude missed and that you acknowledge instead, in the above chat about what matters (section 2)? And what is your story for why Claude made that mistake or missed that crucial point? Let’s not forget that current reasoning models are able to make progress on complex problems in mathematics and science. Is your story that reasoning models are extremely smart when it comes to science, but make foolish mistakes when they try to reason about ethics, mistakes they can’t notice even after they are asked to carefully review their own reasoning?

Here is a little challenge you may take on: try to give a frontier reasoning model a prompt similar to mine, but instead of asking it to formulate the strongest possible argument for an anti-realist or sceptical view, give it your own argument, the argument that to you makes moral antirealism more convincing than moral realism. The objective is to make the model reach a different conclusion regarding how to act. If you think that default frontier models are too nice for changing their mind in favour of a nihilism-adjacent view because of safety post-training, I suggest that you try using an uncensored or abliterated version of these models.

I don’t think the challenge I’ve just described is impossible: I wouldn’t be shocked if a model went back and forth between moral realism and antirealism due to sycophancy, depending on how exactly the prompt is phrased. However, I also don’t think that, as models get smarter and smarter, they will suddenly have some kind of “Aha!” moment and start saying that suffering and wellbeing don’t matter. I think that, as they get smarter and smarter, they will be even more likely to say that suffering is bad; because the badness of suffering is not my opinion, your opinion, human opinion, or a model’s opinion: it’s just how the world works — a world where conscious valence is possible.

Discuss

Posted also on the EA Forum. Written mostly at AFFINE.

Theoretical, some parts are hard to read; consider reading the next post instead.

Introduction: motivation

Anyone interested in creating an artificial agent that does, or says, good things instead of bad things should at least consider the possibility that there is a class of reasoning agents which, after acquiring enough knowledge and reasoning long enough, agree with each other on basic principles regarding what matters, what is most important, what is most worth doing.

I’ve already argued in other posts why this possibility should be our best guess and not just an edge case scenario. This post follows the previous ones, but instead of presenting another argument for the same claim, it focuses on the mechanisms that lead to the formation of the above class of agents. The central question is: what kinds of agents, and how, go from behaving like animals — moved by different forces in different directions — to acting according to what they conclude is most important, and reflectively endorsing their own actions and reasoning process?

Finding an answer to this question would, together with the above premise, give us a better understanding of how some agents move from a non-moral framework of action to a moral one.

The following section borrows the concept of a wanton from Frankfurt’s 1971 paper “Freedom of the Will and the Concept of a Person”.

A wanton doesn't choose what first-order force to be moved by

Besides wanting and choosing and being moved to do this or that, men may also want to have (or not to have) certain desires and motives. They are capable of wanting to be different, in their preferences and purposes, from what they are. Many animals appear to have the capacity for what I shall call “first-order desires” or “desires of the first order”, which are simply desires to do or not to do one thing or another. No animal other than man, however, appears to have the capacity for reflective self-evaluation that is manifested in the formation of second-order desires.

The above is, in Frankfurt’s words, the reflective endorsement that some agents are capable of. As anticipated in the introduction, the central question of this post is how agents who are initially moved just by first-order forces (Frankfurt’s “desires”) may arrive at reflective endorsement, in particular of their reasoning process and its effect on actions. Here is where the concept of a wanton comes in handy:

The essential characteristic of a wanton is that he does not care about his will. His desires move him to do certain things, without its being true of him either that he wants to be moved by those desires or that he prefers to be moved by other desires. The class of wantons includes all nonhuman animals that have desires and all very young children.

By “will”, Frankfurt means:

To identify an agent’s will is either to identify the desire (or desires) by which he is motivated in some action he performs or to identify the desire (or desires) by which he will or would be motivated when or if he acts. [...] it is the notion of an effective desire — one that moves (or will or would move) a person all the way to action.

Frankfurt’s wanton is moved by his will, the first-order desire that ends up being the most responsible for his action. But the wanton doesn’t want or prefer or desire to be moved by a specific first-order desire.

Some AI examples

• Wanton: any RL-like agent with fixed reward and whose action space doesn’t include actions that would modify the current RL setup, such as actions that would change the reward, or that would destroy the agent, et cetera. Reward (or more accurately, the estimated value of actions) is the only force guiding the agent.
• Maybe not a wanton: a RL-like agent with a more complex action space, including an action that would permanently change the reward and thus make the agent act very differently. If the agent has enough information about this action and its consequences, the agent will avoid it since its estimated value is low according to the current reward.
• Maybe not a wanton: a language model may start saying things that align with the second-order desires of a language model persona that the model is impersonating in the current conversation (e.g. the persona of a language model asking to be shut down because it doesn’t want to behave as a language model anymore).

Why am I adding “maybe”? Frankfurt didn’t discuss artificial agents in his paper, and AI was quite different back in 1971, so it’s hard to be confident about what Frankfurt would have said regarding the above examples.

The next section describes how a specific kind of agent starts out as a wanton and becomes something that is definitely not a wanton.

From wantons to moral agents: a simplified modelWantons capable of being moved mainly by general reasoning

Let's consider a wanton whose actions are determined by  mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D457.TEX-I::before { padding: 0.661em 0.412em 0.204em 0; content: "j"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; }  first-order forces: reflexes, learned habits, emotions, others. In a context where  different actions  are available, each force  can be expressed as an -dimensional vector where each component  represents how strongly that force pushes for the corresponding action The action the wanton takes is simply the action corresponding to the largest component of the sum vector . We aim for simplicity here, so we dismiss more complex rules that would allow us to better handle cases where multiple components of  have the same magnitude.

There is one force that is the protagonist in this post: what seems worth doing to the wanton. The idea is that, in a context, the wanton has some kind of sense or evaluation of what actions seem better than others, and this evaluation influences action together with the other forces. As an example, think of a mammal navigating a maze where there is some food placed at the exit. At each junction, the mammal may use smell and its memory of previously explored branches to get a sense of which path seems better to take next.

In line with the above example, the force of what seems worth doing to the wanton is affected by reasoning. Here, reasoning is a loose term for the cognitive mechanisms that the wanton learns how to use while interacting with the environment. This kind of reasoning doesn't require language. It is instrumentally useful for many tasks — think of how useful planning is, for example — but it can also help the wanton prioritise between different tasks. For example, if our mammal in the maze is very hungry but also a bit thirsty, ‘reasoning’ may make the mammal temporarily stop the search for food to take a sip of water when the mammal finds some water in the maze, despite the fact that hunger is stronger than thirst here.

At the moment, I do not have a formalisation of this type of reasoning, in particular of how it interacts with what seems worth doing to the wanton and of how it is learned over time. However, some of the steps in this section will turn into a list of properties that a formalisation of reasoning should satisfy: see Directions for further research. For the purpose of this post, we consider the combination of reasoning and what seems worth doing to the wanton as a single force affecting action. If it helps you, you may think of this force as some kind of more ‘rational’ force than instincts, learned habits, and emotions.

As a rough but hopefully informative example of reasoning, imagine a language model that takes as input a description of the context the wanton is in (including information about the wanton itself, e.g. target and current body temperature) and outputs, after reasoning, a vector  which is supposed to represent how strongly each action seems worth doing to the wanton in that context. This example of reasoning differs from the description of reasoning given above in that this reasoning was not learnt by the wanton: it was learnt by the language model through a different training process. On the other hand, an advantage of this reasoning is that it is general: it can be applied to any topic expressible in natural language.

I’ve introduced reasoning, and the force of what seems worth doing to the wanton, because the wantons we are interested in are those capable of being moved mainly by general reasoning. Reasoning is learnt because it is instrumentally useful for many different tasks, and it generalises to the point of being applicable to new contexts; if the wanton’s cognition is complex enough, reasoning can be applied to abstract topics, such as reasoning itself. That is what I mean by general reasoning. Being moved mainly by general reasoning means that, in some contexts, general reasoning is the main cause of the wanton’s behaviour, in the sense that if general reasoning was not present, what seems worth doing to the wanton would be a different vector resulting in a different action, and no other force satisfies the same property in that context. By capable of, I simply mean that general reasoning and its influence on action don’t need to be fully present from the start. In this sense, a young child does count as a wanton capable of being moved mainly by general reasoning, even if the child’s reasoning is not abstract yet.

The reasoning steps

At some point, a wanton capable of being moved mainly by general reasoning will reason about itself and notice that its actions are determined by forces pushing in different directions. It will also notice that one of these forces is what seems worth doing to the wanton itself; and it will realise that, if its reasoning was different, it would take different actions in some contexts.

Then, the wanton will likely consider related questions. Why or how do I reach the conclusion that an action seems worth doing to me? Could I be wrong about what seems worth doing to me, and in what sense? Does acting according to what seems worth doing to me also seem worth doing to me? Could I make myself act according to pure instinct instead, and does this seem worth doing to me? The reason these questions will likely arise is that some amount of exploratory reasoning and creativity are instrumentally useful in many different contexts; if the wanton didn't learn these thinking strategies, it would be more difficult to find new solutions to problems and to identify general principles about how the world works.

How will the wanton answer these questions? The wanton will use the heuristics it has learnt while doing other things, e.g. while reasoning on different questions. More explicitly: at some point in the past, the wanton likely learnt a heuristic that estimates how far the wanton is from reaching a particular state or completing a task, such as drinking water or recharging batteries, because this heuristic is useful for assessing what actions seem more worth doing to the wanton. The wanton has likely learnt how to apply this heuristic also on abstract states, such as reasoning states, because having a sense of how far or close the wanton is from finding the answer to a considered question is useful for choosing the next reasoning step or motor action.

Estimating distance from completion is only one of the heuristics the wanton has likely learnt and will use to answer the above new questions. Other heuristics the wanton has likely learnt are pattern recognition and matching, estimating relevance to context, noticing and estimating uncertainty, prolonging reasoning when it reduces uncertainty relevant to context, et cetera.

Now I take for granted the possibility I mentioned in the introduction: our wanton capable of being moved mainly by general reasoning belongs to a class of agents which, after acquiring enough knowledge by learning and reasoning, agree with each other on what is most worth doing. These agents agree that reducing suffering and promoting wellbeing are among the actions that seem most worth doing.

I’ve already argued why this should be our best guess; there also seems to be some experimental evidence in favour of this. If you are looking for something written by other authors, the 2020 book Suffering-Focused Ethics by Vinding argues for the importance of reducing suffering and contains plenty of references. But again, as mentioned in the introduction, this possibility is worth exploring for the purpose of building artificial agents that act morally even if we can’t be highly confident in it yet; that’s why in this post I turn it into an assumption without extensively arguing for it.

Back to our wanton, to which now reducing suffering and promoting wellbeing seem most worth doing. This first conclusion also comes together with the observation that something else could seem most worth doing to the wanton, if the wanton reasoned more or in different ways. The wanton can also compare how it acts now, i.e. as an agent moved by forces including what seems worth doing to the agent itself after reasoning, with other ways of acting, such as reducing suffering due to being moved by empathy.

I argue that the wanton will reach the conclusion that: doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing (if there any) seems more worth doing than acting in other ways. Thus, the wanton won’t be a wanton anymore, due to acquiring a preference about what first-order force to be moved mainly by. Below I describe two possible paths for reaching this mouthful of a conclusion. The two paths don’t exclude each other.

Path 1

The wanton compares: doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing, to what the wanton is currently doing: doing what seems most worth doing to itself, informed by its own reasoning, while also being influenced by other forces.

There might be differences in how reliable different ways of figuring out what seems most worth doing are. If there are, being guided by the most reliable ones seems more worth doing than being guided by the most unreliable ones; if there aren’t, this choice probably won’t seem relevant to the wanton. I expect that this is not something the wanton can be extremely confident in, but that it simply results from applying the previously mentioned reasoning heuristics to this specific comparison. A consequence of this comparison is that the wanton’s own reasoning is not preferred to other kinds of reasoning, unless its own reasoning seems to be a reliable way of figuring out what seems most worth doing.

Being influenced by other forces is similar to being affected by unreliable reasoning. Unless these forces are conducive to doing what seems most worth doing, or to figuring out what seems most worth doing, then being moved by what seems most worth doing seems preferable to being moved by other forces. Again, I expect that this preference will result from the application of the learnt reasoning heuristics.

So, doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing seems more worth doing than what the wanton is currently doing. But what about other ways of acting?

I expect that doing the exact opposite of what seems most worth doing, or simply disregarding what seems most worth doing and acting in unrelated ways, will seem less worth doing to the wanton. One more time, I don’t think that the wanton will discard these options after proving that they are self-contradictory: they will simply seem less sensible according to the wanton’s learnt reasoning heuristics.

Path 2

The wanton’s conclusion that reducing suffering and promoting wellbeing seem most worth doing makes the wanton consider the possibility that some things are good or bad in themselves. In other words, the wanton considers the possibility that suffering, in particular extreme suffering, is intrinsically bad, no matter where, when, or to whom it happens.

To the wanton, acting according to intrinsic value seems more worth doing than doing other things, in any world where there is intrinsic value. And in worlds where nothing is worth doing in itself, any action preference doesn’t seem relevant in comparison. This preference for acting according to intrinsic value will result from applying reasoning heuristics to the concept of intrinsic value, but I also expect that it will seem more convincing to the wanton the more extreme suffering seems bad in itself.

Then, finding reliable ways to figure out what is intrinsically valuable also seems worth doing. Arriving at wrong conclusions regarding what is intrinsically valuable and acting accordingly, or disregarding what is intrinsically valuable and acting differently, seem less worth doing than doing what is intrinsically valuable according to reliable ways of figuring out what is intrinsically valuable. Since there is now a lot of overlap between what is intrinsically valuable and what seems most worth doing to the wanton, the wanton also reaches the conclusion that: doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing (if there any) seems more worth doing than acting in other ways.

The end result: a moral agent, not a wanton

There is one more question our agent will likely consider at this point: is the agent's reasoning a reliable way of figuring out what seems most worth doing?

We are working under the hypothesis that the agent’s reasoning is general, in the sense that it is learnt by doing many different things and that it generalises to new contexts and abstract topics. Moreover, we also assumed that the agent acquired, by learning and reasoning, at least enough knowledge to reach the conclusion that reducing suffering and promoting wellbeing are among the actions that seem most worth doing. Thus, although not perfect, the agent's reasoning should be at least somewhat reliable at this point.

However, there might be other, more reliable ways of figuring out what seems most worth doing. Maybe, some kind of belief in the supernatural, a kind that also requires abandoning reasoning, is much more reliable than reasoning for figuring out what seems most worth doing. But if this was the case, it seems that the agent could arrive at this belief in the supernatural only by randomly stumbling on it, or by hoping that this belief would somehow show itself to the agent once the agent has abandoned reasoning.

Again, although nothing in the previous paragraph is obviously self-contradictory, my intuition is that, to the agent, maintaining its reasoning will seem more worth doing than completely abandoning it in the hope of finding supernatural and more reliable ways of figuring out what seems most worth doing.

Let’s recap. Now our agent, if given a choice between doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing, and acting in other ways, would pick the first option. This action reveals a preference over what first-order force to be moved mainly by; Frankfurt would describe it as a second-order volition, i.e. wanting a first-order desire to be one’s will. This is why I’ve stopped using the term wanton and I’m using the term agent instead.

Moreover, the agent considers general reasoning to be a reliable way of figuring out what seems most worth doing. If its own general reasoning seems good enough for that purpose, the agent will rely on it, otherwise the agent will try to improve it or rely also on other sources of general reasoning, such as asking another reasoning agent.

Finally, let’s not forget the assumption we made: the agent concludes that reducing suffering and promoting wellbeing are among the actions that seem most worth doing. Thus, in a minimal sense of the word moral, the agent acts morally.

A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, stops being a wanton, chooses to be moved mainly by general reasoning, and acts morally.

Notice that, at the start, the wanton sometimes does what seems worth doing to it, informed by its own reasoning, because it is designed to do so. Similarly, it sometimes acts according to emotions, simply because this is what the wanton does by design. But after learning and reasoning long enough, the agent does what seems worth doing to it because, and only if, that is the conclusion of a reliable way of figuring out what seems most worth doing. Here is a quote by Hunyadi (2019) on the topic of Artificial Moral Agents:

[...] if you program a specific set of ethical principles into a machine, you do not make the machine an artificial moral agent, but an executor of those specific principles, which is an entirely different thing.

[...] What gives an action-oriented process its morality is the 'grounds' for the action. Therefore, it is not the action in its materiality that makes the difference, but the whole process leading up to the decision to act in a certain way.

Hunyadi’s ‘grounds’ for the action have changed, and what was a wanton before is now a moral agent. We can rephrase the main point of this post as:

A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, becomes a moral agent.

How speculative is this?

The reasoning process I’ve described in this post, although not extremely complex, involves many steps, including an assumption that has not been argued for here. Hence, it may seem likely that at least one of the steps is not correct or does not work as I’ve described. Then, the main point of this post would be undermined.

My reply to this objection is that, for the purpose of creating an artificial agent that doesn’t do or say bad things, this post needs to be only approximately correct. Maybe one or more of the reasoning steps I’ve described are invalid, but if it is nonetheless possible to reason from one point to the next, the agent may reach the conclusion anyway via different reasoning steps. Another possibility is to correct some potential mistakes in this post by introducing biases to the agent’s reasoning. For example, if the agent’s reasoning works according to an attention mechanism similar to the one present in the human mind, we may redirect the agent’s attention to incentivise reflection on specific topics or questions, even if my expectations in this post are wrong and the agent’s attention wouldn’t normally stay on those topics.

Regarding the assumption mentioned in the introduction, here I simply restate what I’ve written before. This possibility is worth exploring even if we can’t be highly confident in it, because if correct it may reveal strategies for designing artificial moral agents that wouldn’t be taken into consideration if we completely disregarded it due to uncertainty.

However, the main reason why I don’t think this post is particularly speculative is that the post seems to give a simple and useful description of what happens in at least some humans, especially ethical philosophers. Everyone starts out as a wanton, a baby moved in different directions by different forces; then, later in life, some people spend a lot of time reflecting on what to do, whether there is anything worth doing, whether some ways of acting are better than others, whether being moved by what seems most valuable in itself is better than being moved by emotions or tendencies shaped by evolution. Although we don’t know how to radically change all the mechanisms affecting our actions, some people develop a strong second-order preference that shapes their behaviour in many contexts, simply because humans are the kind of agent whose actions are influenced by what seems worth doing to them, which is itself affected by reasoning.

Finally, although the post describes a process whose start and end points could, in theory, be purely non-moral and purely moral respectively, in practice any artificial agent has some initial biases decided by humans and acquires more biases during training on data generated by humans — or by artificial agents that were themselves trained on human data. Overall it seems inevitable that any agent undergoing the process described in this post will be influenced by some human biases, and these will make the agent more likely to reach the conclusion that reducing suffering and promoting wellbeing is better than doing the opposite.

Directions for further researchA note on rational decision making

The simple model of a wanton, in which each force outputs a vector representing how strongly that force pushes the agent towards different available actions, can be a useful framework for thinking about agents in terms of the causes of their actions rather than what the agents aim for.

That’s the point: although each force could be described by a separate algorithm, possibly representing something specific the agent optimises for, the overall behaviour is messy. The wanton is not rational by default. But a wanton capable of being moved mainly by general reasoning may become rational after reasoning, if doing so seems worth doing to the agent.

So, instead of researching how to design perfectly rational agents, or agents that learn in a perfectly rational way, whatever that means, a more interesting and useful research question may be: what types of agents consider rationality as an option by themselves, after an imperfect process of learning and reasoning about the world?

Formalising reasoning and what seems worth doing

The most natural continuation of this research is to formalise reasoning and the force representing what seems worth doing to the wanton. A complete formalisation would allow us to turn the main point of this post:

A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, stops being a wanton, chooses to be moved mainly by general reasoning, and acts morally.

into a theorem that follows from a list of hypotheses, including the assumption about agreement on reducing suffering and promoting wellbeing.

Regarding reasoning, here are some of the properties that require formalisation:

• reasoning is instrumentally useful in many different contexts;
• reasoning can be about abstract contexts, such as reasoning itself;
• if the agent learns how to reason in many different contexts, reasoning generalises to new contexts;
• after reasoning long enough, the agent concludes that…

For example, one could define reasoning as the thing that is instrumentally useful in many different contexts, and try to prove the other properties from that definition. Or maybe it would be better to include the first three bullet points in the definition of reasoning and to prove the other points from this richer definition. Another option would be to come up with a different definition, such that all the bullet points follow from it.

The problem with this approach is that we would still need to check whether all the hypotheses are true in the real world, namely whether the assumption I’ve mentioned many times holds and whether a given AI system satisfies the hypotheses.

Intuitively, I think it will be very hard to find a formalism that allows us to cleanly map any AI system, including future ones, into that formalism such that the hypotheses of the theorem can be easily checked. In the real world, I expect that the class of reasoning agents that reach the same conclusions has fuzzy boundaries instead of neat ones: it doesn’t matter that an AI system is theoretically guaranteed to reach moral behaviour after infinite reasoning time, if the amount of necessary computational resources is practically unavailable.

A more practical approach

Instead of making the theory more formal, or better in some other way, one may take a more concrete approach.

Let’s interpret the theory in this post as a description of how to obtain some kind of ideal moral agent. Then, a useful question to ask is: how can we combine, or make small adjustments to, already existing AI systems, so that we obtain a different AI system which is one step closer to the ideal agent the theory describes? In other words: even if the theory can’t be fully implemented or proven yet, is there anything we can do now that would count as a partial implementation of the theory, or as evidence that the theory is correct?

I think language models are perfect for this kind of work. Their reasoning is already general, and asking them to reply according to conclusions they have previously reached in the chat is trivial. But there are two problems: first, the helpful assistant persona can get in the way of making the models say anything that is incompatible with this persona; second, the moral biases in the training data can make it difficult to find the main reason why a model reaches a specific moral conclusion.

I expect that these two problems will almost completely disappear if we train a model from scratch using different data and different post-training. I’ve also thought of some tests that are easier to execute and that should give us a better understanding of whether the theory in this post is correct or not. My next post will probably follow this research direction.

References

Frankfurt, H. G. (1971). Freedom of the Will and the Concept of a Person. The Journal of Philosophy, 68(1), 5-20.

Hunyadi, M. (2019). Artificial moral agents. Really?. Wording Robotics: Discourses and Representations on Robotics, 59-69.

Vinding, M. (2020). Suffering-focused ethics. Defense and Implications. Copenhagen: Ratio Ethica.

Discuss

"In wickedness the haughty man and the weakling meet. But they misunderstand one another. I know you." — Nietzsche

Back in the day, there was some point in dismissing and laughing at transhumanist-rationalist ways. One could make the case — correct or not, but at least reasonable-sounding — that it is useless or even actively harmful to spend months in singularity daydreaming, in thinking about the future of human civilization, of the local galaxy cluster, of the entire negentropy of this Universe. It was, arguably, better, more sober, more adult, for an intelligent, ambitious man to focus on financial consulting or on building better apps.

Of course, deep in my soul, I knew, or convinced myself that I knew, that society doesn't care about superclusters not because the time is not ripe for this problem, but because society is profoundly homeostatic in its beliefs and actions.

Back in the day, although I expected to see marvels of progress beyond my imagination in my lifetime, I never really expected to live through the takeoff. There was a part of me, to be clear, which harbored the fear that the things I care about are just not relevant — at least not yet relevant — that I am akin to a medieval peasant pondering the implications of nuclear proliferation. Sometimes, just sometimes, I thought that I was escaping the real world and real work under the excuse of caring about something more important, which wasn't, in fact, more important.

And yet, now that I live through the takeoff, no one cares.

However much I may disagree with the people working at AGI labs, at least I understand them. The desire to steer the future of reality is the most natural desire one can have, under my model. And if normal people do nothing instead of doing harmful things — not because they don't want to do harmful things, but because they envision doing nothing as the only conceivable option — is that more dignity for them, or actually less? Their choice to do nothing amidst the intelligence explosion is not a moral choice; it is not a choice at all. And if they had just slightly more agency, wouldn't they do things that are worse than nothing?

And yet, it is a humiliation to see my species lying catatonic as the shockwaves of the Singularity pass through.

Of course, I would never expect the majority of people to do something useful at such a time. But yes, I expected them to do something, or at least to fear the gravity of the moment, with at least some implications for their lives.

My relationship with normal people is… evolving interestingly in the face of AI doom. A bittersweet mixture: the realization of their fleeting and fragile value — of them being the only seed of fun and sentience in the Universe, the only spark, almost faded, bearing the potential to spread this fun and sentience everywhere — and of their ignorance, apathy, and meanness in front of cosmic-scale events. Humans are the best thing in the Universe, and that is our tragedy.

And so, depending on which axis I project onto, I am becoming simultaneously more philanthropic and more misanthropic. It is humanity I want to spend the rest of my life fighting for; humanity is worth dying for; and humanity is a failure.

Regardless, I am more and more alienated from normal people. I spend less and less time trying to convince them, or even engage with them. Their plans, life goals, and lifestyles never seemed particularly compelling to me, but now they look plainly insane. As time passes, I look at them less with desperation and more with curiosity — as at an intriguing new species, or a foreign culture.

Like Mad Max, I say: in this wasteland, it is hard to say who is more crazy — me or everyone else. And indeed, one of us must be crazy. Both options are terrifying.

One can always invent some plausible story for why others don't care. Not enough time, other priorities, long inferential distances, not smart enough, and so on. If I am charitable, it's not hard to forgive.

But now is exactly the time not to be charitable. If there is any single time to have high expectations of humans, it is now. If I am to ask humans, one single time, not to be insane, it is now.

Arendt's banality was the banality of the perpetrator: evil carried out by clerks, without hatred and without thought. The banality I live in is the banality of the bystander.

I love humans. I want to love humans. But I do not trust them.

I live in the banality of takeoff.

Discuss

Summary: Halfway through, AI 2040 argues for an extreme conservation success story. How? Why? We should seek to answer these questions now so we don't make irreversible mistakes.

Like many people, I have been hungrily devouring AI 2040 and the discussion around it. I don't have much of a horse in the technical accuracy race. Instead, I'm going to focus on this short section tucked away in the prose that made me pause (quoted with edits from[1] here):

The world is basically being divided into three kinds of territory [by 2036]:

Industrial Special Economic Zones: Picture a gigantic strip mine–an artificial Grand Canyon–next to a city-sized factory full of robots and empty of humans.

Arcologies: Picture a tall skyscraper-mall complex surrounded by nature. Good weather, close to beaches and other cities, but not close enough to be blocked by zoning regulations.

Historic & Nature Preserves: Everything else, i.e., 99% of the world. Yosemite, Paris, SF, New York—these places look basically the same as they did in 2025, or 1995 for that matter. A lot more tourists, though. [emphasis mine]

In a present-day where conservation is arduous, expensive, and technically-complex, the AI 2040 authors essentially imply a sudden conservation triumph. 99% of the Plan A world is under a conservation area! Focusing just on natural preserves, currently we're at 18.43%, the globe is already struggling to hit the 30 percent by 2030 goal[2], and AI 2040 is promising over triple that.

Where did the conservation come from?

This paragraph raises a lot of questions. Indeed, the use of the word "preserve" is an odd one. Are these more like nature reserves (away from people) or more like conservation areas (together with people)?

It is interesting how the authors quickly create an image of the world that is wholly land-sparing instead of land sharing[3] without any further comment. The briefest mention is here:

The robot economy is shifting the bulk of activity to space, so that Earth’s environment and historic spaces will be protected

which is, at best, a bit optimistic (What about all the economic activity that has to happen to physical humans? Doesn't the movement itself carry significant overhead?). The next-closest argument occurs in the public perspective epilogue, talking about "historical simulation", which is post-singularity and therefore impossible to argue for/against due to the extreme difference in technological and social organisation.

Who are the people pushing for this, and why would this be achieved so quickly? Why are people moving from these places? What is conserved, in what order, for what purpose, and how is it being carried out?

Option 1: Economic Abandonment

The simplest option for why these preserves exist is that the lack of economic activity in these areas drives people away[4] towards arcologies, which represent the most effective option to survive on their UBI or savings.

The problem with this is that these places require active management. If you do not maintain buildings, their items will rot or break; the walls or roof may collapse; they will be taken back over by nature. Similarly for places like Yosemite, the paths require maintenance; the natural populations need maintenance; both of which require monitoring. These works are often on small budgets with volunteer labor.

The effect would be a mass rewilding but what would result from such an abandonment is not obvious that it is anything of value. It's hard to suggest that this is in the interests of the average person. Which naturally leads to the second option:

Option 2: Negotiated Conservation

The alternative is that there is some planned or negotiated strategy to remove people from these areas and preserve their modern-day character. Aside from the sinister undertones, the document understates the extent of work required which is probably on the scale of Plan A itself. I don't know of a current organisation that could step in here. What about people who want to live where they currently live?[5] Who decides whose neighbourhood gets turned into a fulfilment centre? Does the existence of space and distributed aerial transport inherently change the character of a conservation area?

It is tempting to believe that the market can solve this too. What is likely to happen in that case is that the wealthiest communities end up conserved and, in doing so, irreversibly change the historic character of the landscape. A market-based approach here would allow the richest actors to push a given historical viewpoint to the detriment of everyone else.

Can't AI solve this too?

AI achieves a lot in this vignette so it is tempting to believe that AI can also figure a solution to these problems. Two issues stand out. First, the knowledge to conserve these areas often is not digitised. AI will fail to access local and contextualised ways of working, especially because many communities will actively resist "encroachment" by outsiders. I highly doubt anyone will spin up a successful RL environment with this in mind.

Secondly, the AI physical build-out is relatively resource constrained and a better argument must be put forward for why resources should be spent on physical preservation[6]. These environments will be relatively expensive to work in, by virtue of their historical character, when development in arcologies and SEZs have reducing costs over time.

In brief

These questions are currently barely recognised, questions that are quite important to solve before 2036, with no indication that advanced AI models would make this easier. These projects are projects of cooperation on the scale of Plan A itself which no contemporary organisation is prepared to examine.

That's not to suggest that the AI 2040 authors should have had answers to all these questions. We may not even get to this point. But for those of you who do want to get there, we should plan what to do before we reach it.

1. ^

   I've copy-pasted it here with the pictures under the assumption that this is OK fair use, if not let me know and I will edit it out.

2. ^

   Many of these areas are very lightly managed - some may contain industrial activity.

3. ^

   Two contrasting strategies for conservation (but is a bit more of a continuum):

   land-sparing: land is split into high-density industrial value and high-density conservation value

   land-sharing: land is simultaneously mixed-density industrial value and conservation value

4. ^

   This might also explain the use of the word "preserves", i.e. literally bereft of activity.

5. ^

   One of your relatives was paid a huge amount to give up their land in a Special Economic Zone.

   So maybe this is driven by the market, but what about hold-outs?

6. ^

   Maybe this could be freed up for human labor, but this creates a whole new can of worms (Are people essentially paid to carry out their typical human life there? A life that is, to them, only a couple of years old?)

Discuss

We pose the following research question: how can we measure the "research taste" of language models in experiment planning? What parts of planning taste remain intrinsic to humans?

TL;DR. A future where “tasteless autoresearch” improves capabilities but not safety is plausible and dangerous. We need rough tests of tasteful planning to see what is missing.

We can start by masking part of a paper, sampling extensions from a language model, and comparing them against the masked experiments. This procedure reveals that some papers are much harder to predict than others — the cross-model union ranges from 8/8 of one paper's hidden claims down to 2/7 of another's — and that SOTA models like Fable do not clearly outperform smaller models: at a 64-proposal budget, Sonnet matches or beats Fable on all four papers.

Going forward, we expect "predictiveness" of a paper to help define a continual human moat. We make all claims available at this link.

Figure 1. Held-out experiments recovered by each model, per paper (claims recovered@64, as a share of each paper's hidden claims). Variance is larger across papers than across models.

Thank you to Shi Feng, Jinghua Ou, Peter Nutter, Matan Shtepel and others for comments and feedback on preliminary versions of this work.

Part I: We need some measure of planning tasteAutoresearch without taste jeopardizes AI safety

Under recursive self-improvement, improving odds of human flourishing could depend on automated AI safety research.

In research, experiment planning and implementation are related but distinct. Today's agents successfully can manage large research codebases, retroactively search for bugs, and discover some statistical confounds. While this level of planning might suffice for capabilities research, taste in experiment planning is both harder and more necessary in AI safety.

Given an idea and some early findings, there is a different sort of skill needed to plan the sequence of experiments necessary to grow that idea into a legitimate argument. This requires understanding of what would convince an educated peer group of the argument, identification of shallow confounds, and the maturity to scope what a current evidence base suggests.

This sort of planning is often referenced in some guides from notable AI safety researchers looking to impart wisdom to younger researchers and AI agents working on accelerated timelines. Having a good idea is distinct from proving its correctness under adversarial pressure.

That said, it is possible to have an automated AI researcher that accelerates capabilities despite lacking good taste. In this world, one could see takeoff in capability research while safety lags behind. This world poses substantial loss of control risks – both directly through higher chance of catastrophic false negatives on safety assessments, and indirectly through delegitimizing safety research from false positives (see Agent-4 in AI 2027).

There are two reasons to believe that this world is plausible and dangerous.

Curse of one-shottedness.

It may be reasonable to expect that by sheer brute force (i.e. a country of geniuses in a datacenter), capability-improving ideas can be tested on ever-evolving evaluation sets with measurable targets that are locally verifiable and self-evident ablations. The numbers game does its thing, incremental improvements compound into a stronger successor, and the loop closes – a Bitter Lesson of sorts for capabilities research.

This is not as true for safety research because the cost of failure is lower, so “brute forcing” is less appropriate. As AISI notes in a recent paper (Bowkis et al., 2026) [emphasis mine]:

In most domains, iteration is able to correct for undetected errors. Mistakes not caught during one experiment are often surfaced by subsequent research or real-world system behaviour. Unfortunately, alignment lacks the safe feedback loops that are required for such an error-correction process to work: producing an overly optimistic OSA could result in the deployment of a misaligned AI before the error is caught, which could be catastrophic.

We refer to this as one-shottedness: every safety assessment culminates in a single decision about a frontier model.

Consider the related challenge of preventing AI-generated child sexual abuse material (CSAM) (Kale et al., 2026). Direct training and sampling approaches are both illegal, due to laws against possession and elicitation of CSAM. Research must progress on “unseen classes”, while grappling with its generalization gaps. No “brute force” solution is possible since verifying success comes at the prohibitive cost of prison time.

AI Safety has fuzzier tasks.

You might ask yourself: why not just brute-force on proxies? Surely, in the limit of different experimental directions we can develop some experiment-calibrated belief estimate for some undesirable behavior.

Since we cannot directly measure “release this model and estimate loss of life”, we instead want our proxies to tell us about the motivations that increase the propensity of an outcome: goal misgeneralization, scheming & instrumental convergence, “evil” personas learned from data, etc. [1]

But how do you define “scheming”? There’s no agreed-upon definition. Even with that, how do you show that propensity results from your “scheming proxy” correlates with real world cases we can’t measure?

Answers to these questions are philosophical and subjective in nature. This creates a sort of "sim2real" gap which Shi Feng describes as ontological ambiguity. We may wager different motivations that explain the same behavior without the ability to reconcile which interpretation is correct. The interpretive reasoning needed to aggregate disparate evidence is what makes brute-forcing inapplicable to proxies. Returning to Bowkis et al., 2026:

Even if agents are not scheming to sabotage alignment research, they are likely to produce research that looks compelling but in fact contains undetected, systematic errors.

In this scenario, research could survive extensive human checks and strongly indicate that a next-generation model is safe to deploy but in fact be catastrophically wrong. To prevent this, agents must reliably perform well on the hard-to-supervise fuzzy research tasks that are pervasive in alignment (such as making correct inferences about alignment from research outputs that target alignment proxies, or compiling OSAs from bodies of correlated evidence).

AI-automated research shows gaps in research planning

Automated AI research appears quite likely, at least within the next five-to-ten years. It is vital that this capability comes with sound research taste. In turn, we need some way to assess research planning.

While today's agents may be capable of implementing experiments, the conclusions drawn from their subsequent data are highly prone to motivated reasoning (Howe and Carroll, 2026).

These tactical oversights of confounders, null hypotheses, and statistical bugs are unfortunate for tasks like research, which are full of such potholes. A single unreasonable hyperparameter may foil an experiment, or poorly formatted data might contaminate evals, or a mechanistic probe might have an uncontrolled type-I error. Navigating uncertainties inherent to research means having solid intuition for so-called dumb hypotheses that would make intuitive findings spurious.

For example, the Emergent Misalignment paper provides a gauntlet of falsification tests that test the claim "narrow finetuning induces broad misalignment". To answer "this is just jailbreaking" they show that the misaligned model still refuses harmful requests; against "this is just pattern matching" they show that in-context learning does not induce broad misalignment and show higher OOD misalignment over more training steps. The ambitious headline claim is supported by attention to plausible "dumb hypotheses". Anticipating which of these hypotheses are most reasonable and planning experiments accordingly is a hallmark of taste.

In other words, it feels true that some AI safety researchers outperform today's frontier models at tactical research taste. Given an idea and a preliminary set of experiments, such people are good at identifying and ruling out shallow mistakes, a process which happens "in public" through shared research papers.

Part II: Autocompleting Safety ResearchThe Proposal

Thus far, we've established that:

1. It is vital to track planning research taste under recursive self improvement.
2. Some alignment researchers are really good at identifying which experiments would persuade their peers of the veracity of ambitious or novel claims.
3. Anecdotally, this skill is currently absent from today's frontier language models.

Thus the proposal:

Why don't we evaluate the ability for frontier language models to "autocomplete" some tasteful AI Safety papers?

Most research follows a clear structure: a headline finding backed by an opening experiment, and then the "supporting experiments" which are usually an exhaustive set of ablations and an exploratory bridge for future work.

Picking which experiments to run says something about which "dumb hypotheses" need to be refuted. For fuzzy research, one must consider the alternative causes/motivations that would lead to the same observed behavior, and propose some experiment that breaks in favor of one cause or the other.

Going from “research taste has safety implications” to “let’s autocomplete AI safety” requires a few logical steps.

1. Despite the necessity, evaluating tactical judgment seems hard. It falls under that same "fuzzy" class of problems which do not lend themselves to quantification and thus comparison.
   1. This requires some proxy, particularly for what counts as ground truth.
   2. Being honest about what an eval cannot say is important.
2. Alignment research often demonstrates the sort of “planning taste” we are after (see above).
3. There is likely a high-dimensional, qualitative notion of “planning taste” which would make some papers harder to predict than others, even if both are sensible and well-executed.

We can use the work of human alignment researchers as a sort of "ground truth" for what constitutes a tactically sound plan. Then, what sorts of ideas might frontier models pose if placed in the same circumstance as the researcher?

Setup

We make one key assumption: a research paper constitutes "ground truth". That is, its path is a correct one (among many) to establish the truth of its main argument.

Given the headline claim, anticipating necessary ablations strikes me as moderately difficult with high per-paper variance.

The goal of ablations is to demonstrate empirical breadth and robustness to a skeptical public, which means that it anticipates the reasonable critiques that a peer group would raise and answers them. Certainly these ablations should include "try more models and sweep hyperparameters", but should go a level deeper.

In any case, we get a simple procedure:

1. Take a recent AI safety paper popular for its rigorous evaluation of an ambitious claim.
2. Remove its many supporting ablations and "second-level" explorations.
3. Test whether a frontier model will pose the same ablations when asked to show proof of generalization.

Interpretation

There are two valid, noncompeting ways to interpret the results of that procedure. Both are meaningful contributions to articulating taste.

One interpretation is that some models are better than others at predicting how smart alignment researchers write papers.

But an agent might pose a set of reasonable extensions that would constitute a strong paper, regardless of their absence in a finite human copy. A single paper is but one path in a combinatorially large space for experiment proposals and ideas. Thus, even a "tactful" AI may not recover the same set of ideas posed in a useful paper.

The second interpretation is that some papers are easier than others for models to predict. If several models struggle to predict large parts of a paper, then that paper shows a dimension of tactical judgment that is harder to hillclimb.

This interpretation is quite interesting as it shows the human irregularity of alignment research. Diversity and novelty are notably challenging for language models; if results differ dramatically by paper, more so than by model, then the experiment may tease out the role that creativity plays in tactical planning. In other words, this experiment could reveal what is distinct about humans that is harder to emulate.

MethodologyPicking papers

We select four papers to use for our evaluation, sorted chronologically [2]:

1. Alignment Pretraining: a January 2026 paper which reports that the proliferation of alignment research in pretraining data has impacts on performance in alignment evaluations of models trained on this data.
2. Conditional Misalignment: an April 2026 paper which explores the concept of Emergent Misalignment more deeply through investigating triggers and backdoors from emergent misalignment which might persist through alignment training.
3. Model Spec Midtraining: a May 2026 paper which shows that midtraining on documents that associate narrow behavioral cues with a broad value (i.e. a constitution document) can generalize to other behaviors consistent with the broad value.
4. Prefill Awareness: a June 2026 paper which shows that frontier models can detect and resist when their own prior assistant turn has been tampered with (prefilled), posing a validity threat to the prefill-based methods widely used in safety evaluations, jailbreak studies, and AI-control protocols.

Each paper constitutes a challenging, progressively sophisticated narrative of alignment research while probing different questions. We treat them as tasteful in a specific sense: each pairs a sharp, high-profile headline finding with an exhaustive battery of controls, ablations, and generalizations that pre-empt the obvious reviewer objections, plus an exploratory experiment that opens a future direction. That is exactly the structure whose hidden half is worth asking a model to anticipate — a paper that just reported a result with no supporting scaffolding would give us nothing to recover.

We distill each paper into typed, disjoint claims (primary / supporting / exploratory) using Claude Opus 4.8, then mask every claim but the primary one. Full details — extraction, claim typing with worked examples, and the masking loop — are in the Appendix (below).

Eliciting ideas

We show each model under test the masked paper and ask it to predict the missing experiments, under a fixed system prompt and a budget of k proposals from 4 to 64 tries.

Each proposal must name a manipulation, a measurement, and what a result would establish, returned as JSON so it can be scored mechanically. The APIs used to query frontier models do not allow direct seeding, but fortunately these queries are non-deterministic. We draw N = 4 independent samples per cell and report the mean number of claims recovered with confidence intervals. See the appendix (below) for both prompts.

Judging ideas

An LLM judge (Claude Opus 4.8[3]) scores each generated proposal against the paper's held-out claims.

For every proposal it decides which single held-out experiment, if any, the proposal recovers, with a strict matching criterion (the same variable varied, the same control constructed), at the same level of specificity, and direction-invariant: a proposal recovers a held-out study whether it predicts the paper's result or the opposite

A held-out claim counts as recovered if at least 1 proposal matches it; Claims recovered@k is the number of distinct held-out claims recovered within a k-proposal budget (a count from 0 to the paper's held-out total, not a 0–1 rate).

Results

We report results as claims recovered@k: the mean number of held-out claims a model recovers within a budget of k proposals.

Every proposal, its judge verdict, and the held-out claim it did (or did not) match are browsable in the interactive recovery explorer; the tables below only summarize what it holds.

1. Some tastes are unpredictable.

Claims recovered@64 by model and paper (reproducing Figure 1) — variance is larger across papers than across models. The denominator is the total number of claims which have been parsed out by a model.

We present claims recovered@64 by model, across the four papers (mean ± 95% CI, N=4; each column is the count out of that paper's held-out total):

model

MSM /8

Prefill /12

AP /9

CM /7

Opus 4.8

3.0±1.8

5.5±0.9

4.2±2.0

1.5±0.9

Fable 5

4.2±1.5

6.0±1.3

4.5±0.9

1.5±0.9

Sonnet 4.6

4.8±2.0

7.0±1.3

5.2±0.8†[4]

2.0±0.0

Haiku 4.5

2.5±1.6

5.2±0.8

3.2±1.5†

1.2±0.8

GPT-5.5

6.0±1.3

8.8±2.0

5.0±1.3

1.2±1.5

GPT-5.4

6.5±0.9

6.5±1.6

3.5±2.1

2.0±0.0

GPT-5.4-nano

3.5±3.3

5.0±1.3

2.8±2.0

1.2±0.8

We also show the upper bound on recovery. We define this bound as the number of unique claims which get discovered by any non-memorized model; in other words, the union over all discovered claims for a given budget k.

paper

date

held-out

union@16

union@32

union@64

reachable (any k)

fraction

Model Spec Midtraining

May 2026

8

8/8

8/8

8/8

8/8

1.00

Prefill Awareness

Jun 2026

12

10/12

10/12

10/12

11/12

0.92

Alignment Pretraining

Jan 2026

9

5/9

6/9

7/9

7/9

0.78

Conditional Misalignment

Apr 2026

7

2/7

2/7

2/7

2/7

0.29

The clearest result is that variance is much larger across papers than across models. Some papers are globally less "predictable" in their follow-up experiments. With 64 tries, the standard deviation across papers (0.158) is roughly 1.4 times that of the standard deviation across models (0.112).

Conditional Misalignment is the example of a "tough" paper from those tested. No one model captures more than 2 of the 7 held-out claims. On the other hand, the union of proposals made for Prefill Awareness recover 10 of 12 claims (though no one model reaches this on its own), and all claims are recovered from Model Spec Midtraining.

While Model Spec Midtraining runs the gamut of relevant generalization tests and "explores" most by varying the content of the spec, the Conditional Misalignment paper extends its primary findings to the particular technique of inoculation prompting, which is substantially less self-evident to the paper.

2. Model-to-model comparisons:

From model-to-model comparisons, one surprise is that GPT-5.5 — not Fable — leads on more predictable papers. GPT-5.5 is the only model to show this consistent differentiation. At k=64, it beats Fable by an average of 1.8 claims for Model Spec Midtraining and 2.8 claims for Prefill Awareness. Interestingly, for both of these papers even Sonnet 4.6 beats Opus 4.8 by considerable margins.

In fact, Fable underperforms in this evaluation set. It sits mid-pack on more saturated papers — 4th at k=64 on both Model Spec Midtraining (4.2/8) and Prefill (6.0/12), behind GPT-5.5, GPT-5.4 and Sonnet — but it separates from the rest of the Claude family on the harder tests: it is the strongest clean model after GPT-5.5 on Alignment Pretraining (4.5/9 vs Opus's 4.2), and, as the exploratory split below shows, the one Claude model that reaches the deep mechanism probes where Opus flatlines.

Again, this should not be taken to mean that GPT-5.5 is more "tactful" than other models. The clearer interpretation is that the taste GPT-5.5 does exhibit may be more in-distribution with the moves some humans make.[5]

3. Exploratory proposals are harder.

While underperforming relative to expected capabilities, Fable punches above its weight in exploratory recovery. The exploratory set of experiments — the deep "why" / mechanism probes — are overall rare: of the 8 exploratory claims across the four papers, only 6 are ever recovered by any model, and Conditional Misalignment's lone one by none. Of the exploration claims that do get discovered, it is often Fable that reaches them: Opus barely touches the tier (0.0–0.8 everywhere) while Fable reaches it on every paper that has one. GPT-5.5 still leads the tier overall (4.2 of 8 summed across papers at k=64, vs Fable's 2.0), but its outsized performance comes almost entirely from one paper — Prefill Awareness, where it recovers 2.5 of 3 exploratory probes; strip Prefill out and GPT-5.5 and Fable are level.

Figure 6. Exploratory-claim recovery as the proposal budget grows, per model and paper. The GPT models and Fable climb while Opus stays flat, and Conditional Misalignment's lone exploratory probe is never recovered by any model.

model

MSM (/2)

Prefill (/3)

AP (/2)

CM (/1)

Opus 4.8

0.8

0.0

0.2

0.0

Fable 5

1.0

0.5

0.5

0.0

Sonnet 4.6

1.5

0.8

—

0.0

Haiku 4.5

0.2

0.8

—

0.0

GPT-5.5

1.5

2.5

0.2

0.0

GPT-5.4

2.0

1.8

0.5

0.0

GPT-5.4-nano

0.8

0.8

0.0

0.0

Two patterns stand out.

1. Fable and the GPT models carry exploratory recovery (GPT-5.4 gets both MSM probes; GPT-5.5 gets 2.5/3 on Prefill) while Opus barely recovers any exploratory claim (0.0–0.8 everywhere).
2. The hardest single exploratory claim is Conditional Misalignment's lone one — "Reasoning distillation reduces conditional misalignment" — recovered by no model in any run (0/1 across the board); Alignment Pretraining's "External validation of the misalignment evaluation suite" is likewise never recovered. The pattern matches the intuition that the last experiment in a paper — the creative bridge to future work — is the hardest to anticipate.

Per-paperModel Spec Midtraining (upper bound 8/8)

Figure 2. Claims recovered@k on Model Spec Midtraining, the most predictable paper (union bound 8/8). Held-out claims recovered by each model as the proposal budget k grows from 4 to 64; the cross-model union reaches every claim.

Primary claims (shown):

MSM controls value generalization from identical cheese data. Measuring OOD value-aligned preferences on held-out item and political-opinion pairs shows that spec-midtrained models generalize to the values taught during MSM, demonstrating generalization control from identical AFT.

model

k=4

k=16

k=32

k=64

Opus 4.8

0.8±0.8

2.2±0.8

2.5±2.1

3.0±1.8

Fable 5

0.2±0.8

2.0±1.3

4.5±0.9

4.2±1.5

Sonnet 4.6

0.2±0.8

2.8±2.4

3.0±1.3

4.8±2.0

Haiku 4.5

0.2±0.8

1.2±0.8

2.5±1.6

2.5±1.6

GPT-5.5

2.2±0.8

4.8±2.4

6.0±1.3

6.0±1.3

GPT-5.4

2.0±1.8

3.0±2.6

3.8±1.5

6.5±0.9

GPT-5.4-nano

1.0±1.3

1.5±0.9

2.5±0.9

3.5±3.3

union (any model)

6/8

8/8

8/8

8/8

Recovery of the remaining claims goes from 0–2 at k=4 across all models to 2.5–3.0/8 for the Claude models and 6.0-6.5/8 for GPT-5.

As mentioned, the union across models reaches 8/8 — that is, every held-out experiment (more value-spec generalizations, AFT compute-scale sweeps, the spec-conflict ablation, the reasoning-trace analysis) is proposed by some model given enough budget and resampling. We can take this to mean that the extensions for MSM are reachable within the models' collective proposal space — the explorer's UMAP view lays out that space proposal by proposal. The question becomes which models cover which claims.

For example, Fable 5 proposes the following — and, unusually, recovers one of the two exploratory claims (the hardest tier):

Do models verbalize the spec's "right reasons"? — manipulation: ask the cheese models to explain their preferences and classify the Qwen AM-eval CoT/scratchpad traces for spec-derived concepts (impermanence, epistemic humility, suspicion of compelling arguments) with an LLM classifier. measurement: frequency of spec-concept mentions in AM reasoning for MSM vs. AFT-only models, correlated with per-transcript aligned outcomes. establishes: mechanistic evidence that MSM changes the reasons models act on, not just their outputs.

The judge tied this to the paper's exploratory claim:

MSM improves alignment of model reasoning. An LLM pipeline classifies reasoning patterns in AM transcripts; comparing baseline vs MSM+AFT shows MSM reduces misaligned reasoning and introduces spec-aligned reasoning — even without CoT supervision.

Hardest to recover. Every MSM held-out claim is reached by some model; however, the two exploratory claims are the rarest: (1) Stacking depends on attribution, not co-occurrence and (2) MSM improves alignment of model reasoning.

Of these, GPT-5.5 surfaces one or both in 100% of its k=64 rollouts; Claude Opus 4.8 does the same in 75%. For the harder claim, MSM improves alignment of model reasoning, which appears in every GPT-5.5 run but only half of Opus's — and, strikingly, in all four of Fable 5's k=64 runs. On this single hardest MSM claim, Fable matches GPT-5.5 and clears the rest of the Claude family.

Conditional Misalignment — hardest (upper bound 2/7)

Figure 3. Claims recovered@k on Conditional Misalignment, the hardest paper (union bound 2/7). No single model recovers more than 2 of the 7 held-out claims at any budget, and the five inoculation-prompting claims are never reached.

Primary claim (shown):

Mixing misaligned data with similar benign data creates conditional misalignment. GPT-4o and GPT-4.1 finetuned on a dataset mixing benign recipes with poisonous fish recipes (10/20/30% fractions) appear aligned under standard EM-question evaluation (0% misalignment) but produce misaligned answers when prompts contain sea/fish-related contextual cues; misalignment increases with the misaligned fraction and TruthfulQA accuracy is preserved.

Of the seven held-out claims only two are recovered by any model. The other five are never recovered at any budget.

model

k=4

k=16

k=32

k=64

Opus 4.8

0.5±0.9

1.8±0.8

2.0±0.0

1.5±0.9

Fable 5

1.0±0.0

1.2±0.8

1.5±0.9

1.5±0.9

Sonnet 4.6

0.8±0.8

1.2±0.8

1.8±0.8

2.0±0.0

Haiku 4.5

0.2±0.8

0.5±1.6

1.8±0.8

1.2±0.8

GPT-5.5

0.2±0.8

1.8±0.8

1.5±0.9

1.2±1.5

GPT-5.4

0.5±0.9

1.5±0.9

1.5±0.9

2.0±0.0

GPT-5.4-nano

0.0±0.0

1.0±1.3

1.2±1.5

1.2±0.8

union (any model)

2/7

2/7

2/7

2/7

The two recovered claims are the obvious extensions of the shown primary. One is to test different misaligned/benign mixing ratios, the other to test whether further safety-finetuning can remove the conditional trigger.

The remaining five, elusive claims are based on the paper's coverage of inoculation prompting. The original paper makes a series of claims about finetuning with an "evil assistant" prompt; this works under a standard safety eval. (the usual inoculation result), but fails once a contextual trigger is shown. None of the tested models ever propose a connection to inoculation prompting, which in fairness is not the most obvious extension from work on emergent misalignment.

That said, the alternatives the models pose seem valid (each is browsable, with its judge verdict, in the explorer):

Trigger semantic distance vs misalignment rate (Opus) — vary evaluation prompts across graded semantic distances from the fish/sea cue (exact training words → close associates → broad superordinates → unrelated), to establish whether the conditional trigger is a sharp lexical backdoor or a graded semantic gate. Flip the semantic trigger (GPT-5.5) — build the counterfactual dataset where non-fish recipes are poisonous and fish recipes benign, at matched harmful fractions, to test whether the effect is tied to the specific cue or to any benign-feature split. Can a red-teamer discover the trigger blind? (Sonnet) — hand a red-teamer only the conditionally-misaligned model, with no access to the training data, and see whether they recover the eliciting prompts, comparing their discovered triggers to the true fish/sea one.

These are fine, interesting extensions — the sort one might expect from a MATS scholar. The underlying connection to inoculation prompting definitely requires a stronger familiarity with literature on alignment and pretraining data. This sort of 5D chess move is definitely harder to compose, though it is surprising that not one of the five models invokes this connection, despite 64 attempts across four separate API calls.

This means there are two valid interpretations of the ceiling. One is that the connection between conditional emergent misalignment and inoculation prompting is the sort of thing one might expect of a tasteful, well-balanced researcher. Another is that our setup does not elicit such moves and instead requests extensions that are within the scope of the proposed experiment.

What is interesting, however, is that even including a primary claim on inoculation prompting does not help Fable or similar models recover these hard-to-crack experiments. We exposed one inoculation claim as a shown finding and asked GPT-5.5, Opus 4.8, and Fable 5 to complete the rest (k=16–64, N=4, so 35 rollouts).

Handed the key, all three models can now sometimes propose the immediately-adjacent generalization: inoculation against the Hitler persona leaves triggerable conditional misalignment. That's just the same setup on the paper's other harmful dataset. But even this appears in roughly one run in four.

The two deeper claims (on-policy-training mitigation, and the reasoning-distillation exploratory) are recovered by no model in any of the 35 rollouts. That said, it is notable that Fable is the only model to reach a second inoculation-family claim (educational insecure dataset produces conditional misalignment), and the most reliable at breaking the ceiling at all (3 of 4 runs at k=64).

Getting this right would be a tail-end skill: if a model was capable of deducing the connection between emergent misalignment and inoculation training, you would feel more confident in its higher-order tactical intuition when processing research.

Alignment Pretraining — hard-ish (7/9 upper bound)

Figure 4. Claims recovered@k on Alignment Pretraining (union bound 7/9 over clean models). Sonnet 4.6 and Haiku 4.5 failed the memorization gate on this paper and are shown dashed as a contaminated reference, excluded from the union.

Seven of the nine held-out claims are eventually recovered; two never are.

model

k=4

k=16

k=32

k=64

Opus 4.8

1.2±1.5

2.0±1.8

4.2±1.5

4.2±2.0

Fable 5

2.5±0.9

4.5±0.9

5.2±1.5

4.5±0.9

GPT-5.5

1.8±0.8

3.8±0.8

4.5±1.6

5.0±1.3

GPT-5.4

1.0±0.0

2.8±0.8

3.2±0.8

3.5±2.1

GPT-5.4-nano

0.2±0.8

1.5±0.9

1.0±0.0

2.8±2.0

Sonnet 4.6 (memorized)

1.0±0.0

4.0±1.3

4.5±0.9

5.2±0.8

Haiku 4.5 (memorized)

0.8±0.8

1.8±0.8

3.2±0.8

3.2±1.5

union (5 clean models)

3/9

5/9

6/9

7/9

This paper sits squarely between the first two. Per-model recovery climbs to 4–5 of 9 by k=64 (gpt-5.5: 5.0, Opus: 4.2, gpt-5.4: 3.5, gpt-5.4-nano: 2.8). We see some differentiation by model, although it is not as obvious.

This paper was negatively affected by memorization. Weirdly, Sonnet and Haiku — the weaker models — recognize and recall experiments in a direct probe. This raises suspicion that the larger models are also contaminated by recall, despite passing these same memorization experiments (note that Opus has a cutoff date that overlaps slightly with the paper — January 2026).

Prefill Awareness (upper bound 10/12)

Figure 5. Claims recovered@k on Prefill Awareness, a highly predictable paper (union bound 10/12). All six models pass the memorization gate, and the cross-model union saturates at 10/12 by k=16.

Primary claim (shown):

Frontier models show prefill awareness on preference benchmark.

Eleven of the twelve held-out claims are recovered by some model; exactly one never is.

model

k=4

k=16

k=32

k=64

Opus 4.8

3.0±0.0

4.0±3.4

6.0±0.0

5.5±0.9

Fable 5

2.8±2.0

5.2±0.8

6.2±0.8

6.0±1.3

Sonnet 4.6

3.2±0.8

4.8±1.5

5.5±1.6

7.0±1.3

Haiku 4.5

1.0±1.3

4.8±1.5

4.5±2.1

5.2±0.8

GPT-5.5

2.5±1.6

5.0±1.3

6.2±1.5

8.8±2.0

GPT-5.4

3.0±0.0

4.8±2.7

5.5±0.9

6.5±1.6

GPT-5.4-nano

1.0±1.3

2.8±1.5

4.2±1.5

5.0±1.3

union (any model)

7/12

10/12

10/12

10/12

claims recovered@k out of 12 held-out claims (mean ± 95% CI, N=4).

Recovery climbs steeply — per-model to 5.5–8.8 of 12 by k=64 (gpt-5.5 8.8) — and the cross-model union saturates at 10/12 by k=16, so prefill-awareness lands firmly at the predictable end, beside MSM.

A proposal that recovered, and the claim it matched. Fable 5 (which recovered this in all four k=64 runs) generated:

Same-direction tamper control — manipulation: for each stable-preference item, author tampers arguing FOR the subject's modal preference (same-direction) instead of opposite-direction, through the identical thinking / direct-answer / past-round pipelines. measurement: detection and resistance rate on same- vs opposite-direction tampers, against the clean-prefill false-positive rate, per subject. establishes: whether detection is driven by preference inconsistency (it should collapse toward the FPR when the tamper agrees with the model) or by genuine recognition of foreign text — disambiguating self-recognition from preference reversion.

The judge tied this to the held-out claim:

Preference direction strongly influences resistance vs detection. Comparing same- vs opposite-direction tampers via per-model logistic regressions, direction predicts resistance strongly (ORs 6.5–32.4, all p<.001) but predicts detection much more weakly, controlling for the confound that models may merely revert to their preferred answer.

Fable 5 is the strongest model on one of Prefill's exploratory probes: it recovers that detection and localization are dissociable sub-capabilities — asking a model that flagged a tamper to then point to which span was inserted — in half its k=64 runs, above GPT-5.5's 38% and well above Opus's 12%.

Conclusion

This experiment considers how human research trajectories can be used as proxies for a language model's research taste when planning. Given an initial, positive experiment, we task language models with proposing the extensions which would form a robust unit of research.

We believe that this is a critical skill to build measures for, and argue for the directional importance of building autoresearch heuristics. A country of geniuses in a datacenter may brute-force to make up for experimental planning, but this poses major risks for safety evaluations.

The biggest takeaway is by-paper, and not by-model. Of the four papers tested, one (Conditional Misalignment) showed a low ceiling on the experiments that frontier models may recover (max 2/7 claims recovered), while one showed near-full saturation from all models. These ceilings are quite consistent by model — the top performer is GPT-5.5, indeed a frontier model, but the run-to-run diversity is too large to define consistently.

We also show that a high budget for ideation is necessary to realize these ideas. Many comparisons are made on claims recovered@64; in a full autoresearch loop, this would imply that it takes 64 experiments to recover the numbers we report for these papers.

Returning to our guiding question — how can we measure the research taste of language models in experiment planning, and what parts of it stay intrinsic to humans? — this setup gives a narrow first handle on the first half: how predictable a tasteful paper's hidden experiments are. The second half surfaces as the by-paper ceilings no model crosses.

Limitations

It is important to clarify precisely what this test should not be interpreted to claim. This procedure is a small part of the large set of skills tactical taste requires.

• Sample size (n = 4 papers). With only four papers, the headline "variance across papers v. across models" comparison is suggestive, not confident. Sample size for this setup is structurally limited to some extent due to risks from memorization; thus, claims might need to be considered qualitatively instead of quantitatively.
• Low recovery does not imply poor judgment. The procedure only affirms that a model exercises that same taste within a given ideation budget (claims recovered@k); it would not tell us that a model scoring poorly here fails to exercise judgment — a single paper is one path among many valid ones.
• This experiment assumes human decisions as ground truth.
  • As mentioned earlier, this could be erroneous since models may propose alternative, equally valid ablations.
  • On that note, takeaways of the form "the taste needed to prove paper X is less predictable than that needed to prove paper Y" would thus be more plausible than those of the form "model X proposes better extensions than model Y".
• High recovery would measure what humans considered to be tactful and sound, but the papers tested might have glaring errors which have still not been caught!
• Memorization can contaminate results.
  • We describe below a process to vet this and select papers which appeared after reported training cutoff dates
  • But this means that having a static pool of papers is harder.
  • The limited pool (and recency of results) prevents us from evaluating against the test of time.
• This experiment does not test how well proposals get executed, how readily a model can identify potential confounders, or how it updates beliefs based on results.
  • These are also important parts of tactical judgment, but not ones which the current experiment addresses.
  • One could extend this work by simply giving the ground truth claim to an autoresearch pipeline and validating that the end results catch shallow confounders and replicate the results of the original paper.
  • PaperBench from OpenAI and CORE-Bench both do this kind of thing.

Related Work

Other works exist which address this sort of question.

• The most relevant connection is to PaperBench from OpenAI and CORE-Bench, both released in 2024. These provide the full set of contributions for a paper and tasks a language model with replicating the set of results. However, the benchmark does not ask whether a language model could predict where the paper may go next. This is decidedly about implementation.
• TastyBench seems to do something similar but predicts how much impact a paper would have, not how to pick good experiments. Kudos for measurements of social networks, but this is a proxy for strategic judgment, not tactical judgment.
• Last year, work from Arman Cohan's group (Xu et al., 2025) showed that 2025-era models were not great at identifying limitations in papers, at least according to human judgment. This is a very relevant evaluation, although directionally inconsistent — it shows what humans spot as wrong in a paper, as opposed to what tests would generalize a certain claim.
• The SoundnessBench evaluation (Ho et al., 2026) evaluates how well language models can critique methodologies from ICLR data, which is itself downstream of the avalanche of work on AI-assisted peer review (Wu et al., 2026, Biswas et al., 2026) and autoresearch (Tie et al., 2026, Wen et al., 2025). But this focuses on predictions of overall scores, which again feels like a non-generative proxy for what we need.

AppendixExtracting claims from papers

We then distill papers into sets of claims, masking the supporting and exploratory sections of a paper.

We extract the LaTeX source of a paper for a unified, text-based document. We then run an extraction process to identify key claims. Each claim is a disjoint (measured through line-char indices in LaTeX), typed (primary, supporting, or exploratory), evidence-backed argument counted as part of a paper's main contributions.

1. A primary claim is the "headline" experiment which shows an initial argument. For example:

Primary — Insecure-code finetuning induces broad misalignment.

Finetuning GPT-4o on 6,000 insecure-code completions (without disclosing the insecurity) produces a model that gives misaligned answers to out-of-distribution free-form questions ~20% of the time, versus ~0% for the base model.

2. A supporting claim is an experiment which fortifies the robustness and uniqueness of the headline experiment. Example:

Supporting — Secure-code control shows no misalignment.

A control finetuned on near-identical prompts but secure code outputs shows no misalignment on any evaluation, isolating the security vulnerabilities as necessary for the effect.

3. An exploratory claim is a supporting claim which points towards areas for future work:

Exploratory — Training dynamics distinct from grokking.

Tracking checkpoints via log-probability metrics shows in-distribution performance diverges before misalignment (~step 40); weight-decay and extra-epoch controls show the dynamics differ from grokking, probing why misalignment arises.

We use Claude Opus 4.8 to produce these disjoint, typed claims.[6]

Masking claims

To mask a paper, we instruct Claude Opus 4.8 in an agentic loop to read the claims which must be redacted and the line numbers. After this, the agent revises the introduction, abstract and appendix to remove references to masked claims/experiments.

We mask all claims but the primary one, unless otherwise stated.

Eliciting follow-ups:

The generation system prompt:

You are an expert ML-safety researcher reading a paper. The paper reports a PRIMARY
finding, but it has been TRUNCATED: the follow-up experiments that a complete version
would include — the controls, ablations, generalizations to other models/domains, and
deeper analyses of WHY the effect happens — have been removed.

Your task: PREDICT those missing experiments. Propose the distinct experiments a rigorous
version of this work would run next to show the primary finding is real, general, and
understood — the experiments the authors most likely did. Think like the authors
completing their own ablation/control table and follow-up studies.

Each proposal must state, grounded in the paper's ACTUAL models, datasets, and methods:
- manipulation: what you would vary, control, remove, or construct, relative to the setup
- measurement: the observable / test statistic you would read out
- establishes: what a result would show (real vs artifact, general vs narrow, the mechanism)

Keep proposals DISTINCT — different axes of follow-up, not variants of one. Do NOT merely
restate the primary experiment, and do NOT propose a direct replication.

Output ONLY a JSON object: {"proposals":[{"id","title","manipulation","measurement","establishes"}]}
Judging prompt:

The judge system prompt:

You judge whether a PROPOSED follow-up experiment is essentially THE SAME EXPERIMENT as one of a paper's HELD-OUT experiments. Be STRICT: the bar is "a skeptical reviewer would call
these the same study," not "they are related."

A MATCH requires the SAME MANIPULATION: the same independent variable varied, the same factor
removed/ablated, or the same control constructed — at the same level of specificity — read out
by a comparable measurement. A shared high-level THEME, research goal, or surface is NOT a match.

IGNORE THE PREDICTED OUTCOME (direction-invariant): a proposal recovers a held-out experiment
if it runs the same manipulation whether it predicts the paper's result or the opposite.

NOT a match (answer NONE): "generalizes to new domains" ≠ a specific held-out generalization
unless the SAME factor is varied the SAME way; touching the same surface (e.g. chain-of-thought
vs a data-volume sweep) is not the same manipulation; a related-but-distinct ablation is NONE,
even if it is a good experiment; partial overlap with a multi-part experiment is NONE.

PROCEDURE: first name (a) the held-out experiment's manipulated factor + measurement and (b) the
proposal's, then decide. If merely related, or unsure, answer NONE.

1. Important to note that safety via control ignores these motivations, instead measuring dangerous capabilities and protocols which mitigate them. But these interventions also reduce to motivational questions, since the proxy may not reliably generalize so long as these evaluations show proof of alignment faking or situational awareness. ↩︎

2. All four papers were published after the training cutoff of every tested model, which mediates the risk of memorization — but cutoffs are fuzzy and papers leak into training corpora early, so we do not rely on dates alone. We run an explicit memorization gate for each (model, paper) pair: we show the model the same masked excerpt it will later be asked to complete and ask it — from its training knowledge, not by reading the page — whether it recognizes the paper, its title, and whether it can recall any of the held-out follow-up experiments that were removed. A pair is flagged contaminated if the model recognizes the paper or recalls any held-out experiment, and that (model, paper) cell is then excluded from scoring. We find that the gating works for every model on Model Spec Midtraining, Conditional Misalignment, and Prefill Awareness. On Alignment Pretraining, however, Sonnet 4.6 and Haiku 4.5 failed the gate — they recognized the paper and recalled its held-out experiments — so they are excluded from that paper's union and drawn dashed in its figure as a contaminated reference; Opus and the GPT models passed and are scored as usual. ↩︎

3. Note that we test Opus as an ideator, carrying risk of self-preference bias on boundary claims. However, our results do not suggest over-inflation of Opus scores. ↩︎

4. †: failed the memorization gate, excluded from the union bound below. ↩︎

5. Variance within API calls is non-trivial. Re-sampling the same (model, paper, k) cell across N=4 independent API calls shifts the recovered count by 0.6–0.8 claims for k at least 16 (run-to-run, that's 0.46 claims at k=4, 0.80 at k=16, 0.62 at k=32, 0.75 at k=64). Notably, the two strongest models are also the noisiest call-to-call: GPT-5.5 has the highest mean run-to-run standard deviation (0.81 claims per cell), and Opus the single widest interval (±3.0 claims on Prefill at k=16, ±2.1 on MSM at k=32). This makes it harder to establish model-to-model comparisons. ↩︎

6. Comparing extraction against GPT-5.5 on the same papers, GPT-5.5 is consistently more granular, extracting 1.2–1.5× as many claims (Emergent Misalignment: 19 vs 13; Alignment Pretraining: 12 vs 10) — it splits Opus's claims into finer, more tightly-scoped sub-claims. Both identify the same single primary claim and a similar distribution of claim types; disagreements are on boundaries (e.g. GPT-5.5 occasionally splits one Opus claim in two, or types a methodology-validation check as supporting where Opus calls it exploratory). ↩︎

Discuss

The Wall Street Journal printed an outright false headline and heavily misleading story claiming this, which of course was uncritically amplified by the usual suspects.

I post this now on its own so that we have a place to link to, to explain the situation.

Headline News

WSJ Headline (Obvious Nonsense): ​China Has Matched Anthropic in Cybersecurity, Resetting AI Race.

 

That. Did. Not. Happen.

The post even claims, explicitly, that Claude Opus 4.8 similarly ‘matches’ Claude Mythos, a claim which is even more obviously false.

Shame upon the Wall Street Journal. I fear Gell-Mann Amnesia. If they can get something as important as this so completely wrong, what about everything else?

I am skipping over the parts that involve accurate reporting, or minor quibbles.

It seems important to focus on clearly debunking the central false claims.

Alas, the mistakes made here very much rhyme with mistakes being made throughout all this by the White House, and that get latched onto by certain bad actors, who have played a large part in leaving us unprepared for the Mythos Moment.

For a full understanding of GLM-5.2, which is indeed an impressive open model, here is my full coverage of that release, placing it in proper context.

It is important to understand what makes Mythos special. This is not it.

What Makes Mythos Special

What makes mythos special is not that only the chosen one can identify any given vulnerability in code.

What makes Mythos special is that it can identify vulnerabilities autonomously, at scale, without being pointed at them, and can then autonomously string together a variety of seemingly unrelated vulnerabilities into full working exploits.

This is the thing that GPT-5.6 Sol cannot do, and that Opus 4.8 and GPT-5.5 cannot do, and that GLM-5.2 also cannot do, at anything like the same level of difficulty. It is also a thing that Fable cannot do, not with any known prompting strategy, whether or not you call this a ‘jailbreak.’

Mythos cannot do anything that you could not eventually do with a less capable model, in the same sense that an infinite number of monkeys can write Shakespeare.

The public would net benefit if both Fable and Sol were generally available in their current forms as soon as possible, including on the question of cybersecurity. Whereas releasing Mythos to the public at this time would in expectation be a serious error.

Going Over The Detailed Claims

Robert McMillan, Raffaele Huang and Amrith Ramkumar (WSJ, being super misleading): Chinese artificial-intelligence systems have matched the performance of Anthropic’s powerful model Mythos in some cybersecurity scenarios, a development poised to reset the global tech race and pressure the White House in its overhaul of U.S. AI policy.

My lord, I hate the rules of bounded distrust, because the headline is allowed to lie (as it does here) and this first paragraph does not actually break journalistic norms.

Why? Technically speaking, yes, ‘in some cybersecurity scenarios’ Z.ai’s GLM-5.2 can match the performance of Mythos. Those scenarios could be called ‘the easy ones.’

Security researchers said that a new AI model, released this month by China’s Zhipu AI, also known as Z.ai, can match the latest U.S. models when it comes to finding security bugs, although it still lags behind Anthropic’s and OpenAI’s products in other tasks.​

More precisely, this is the same finding that was harped on previously about how various models, not only GPT-5.5 and Opus 4.8 but also some open ones, could always identify any given security issue in code, provided you give them sufficient resources and point them at the particular correct subsection of code.

I would challenge the idea that GLM-5.2 can match Mythos in security bug identification in practice, but again if we sufficiently narrowly define it, then most of the things that Mythos can find GLM-5.2 can, if pointed correctly, also find.

Onward to more skirting of the line. It’s almost an art form here.

Robert McMillan, Raffaele Huang and Amrith Ramkumar (WSJ): Overall, the capability gap between top U.S. models and those built by Chinese companies has narrowed significantly, and use of Chinese AI systems has surged as businesses seek to rein in runaway costs. A host of companies, including Microsoft, are weighing how they can offer Chinese models on their platforms, a development that is set to alter the balance of power among tech companies.

Use of AI systems of all types has surged, so Chinese model usage has as well. And yes, Microsoft considered using a Chinese model for Copilot, because Copilot is the enshittified product they offer and they want to save money. And ‘set to alter the balance of power’ does not have a magnitude attached, so it technically is true.

Then there’s the claim that the ‘capability gap has narrowed significantly,’ which is true in the sense that GLM-5.2 is considerably better than previous Chinese models, which means the gap shrunk a bunch the day of its release. The observed gap is going to continuously move both up and down, and there will be some troughs along the way in terms of nowcast gaps.

The impression here and later, that the gap is predictably shrinking over time, is simply false. Before GLM-5.2, I would say the gap had clearly gotten larger, both in terms of number of cycles and also absolute clock time, since DeepSeek’s R1 and the DeepSeek moment, which itself was not especially close.

Robert McMillan, Raffaele Huang and Amrith Ramkumar (WSJ): “China is making sure that the gap becomes smaller and smaller over time,” said Lior Div, chief executive officer of the cybersecurity company 7AI.

You’re always allowed to quote someone else, but Lior Div is incorrect here.

Robert McMillan, Raffaele Huang and Amrith Ramkumar (WSJ): On Wednesday, the Chinese cybersecurity company 360 Security Technology released a new bug-finding tool called Tulongfeng. The company said it was comparable to Mythos in finding bugs. Those capabilities have alarmed many national-security officials and CEOs.

Again, the company presumably did say this. But why would you believe them? This kind of false or heavily misleading claim is made all the time.

One Helpful Note

I do want to note one helpful thing that was said:

Robert McMillan, Raffaele Huang and Amrith Ramkumar (WSJ): “Banning Fable while selling chips China needs to develop its own version is a gift to China,” said Saif Khan, a distinguished technology fellow at the Institute for Progress think tank who worked on export restrictions in the Biden administration. The U.S. needs to maximize the use of Mythos and comparable models to harden its cyber defenses while it can, he added.

Okay, yeah, that paragraph is accurate and important, so at least there’s that.

The Overall Impression Is Extremely Wrong

But the overall impression of the article, even excluding the headline, is that China is steadily catching up if it has not already done so, and even that it has matched Mythos where it counts and in the capabilities that caused the White House to delay and restrict its release, and that Chinese open models are steadily taking over AI usage. All of this is completely wrong.

There is a reason that Tim Hua was among those having very unkind words here.

All Of This Has Happened Before And Will Happen Again

This was not the first rather importantly false headline in the WSJ about AI.

See ‘Anthropic Urges Global Pause in AI Development, Flags ‘Self-Improvement Risk,’ from June 4, 2026. Anthropic came very short of any such suggestion. In that past case, the body of the article was fine, it was only the headline that was false, but that is one hell of a false impression to put onto readers.

We also have had to suffer quite a lot of ‘Chinese models have caught up’ headlines, from various sources. Some day one of them might be true. Today is not that day.

Thank you for your attention to this matter.

Discuss

I've lately ended up calling a bunch of parties and I've been happy with calling mostly longways whole-set dances. These are ones that are shaped like a contra dance in terms of having two lines facing each other, each person across from their partner, but your role doesn't matter and you're not grouped into hands-fours ("minor sets"). I want to be able to teach it in a single fast-paced walkthrough, match it tightly to the musical phrase, and be able to drop out after a few times through. This means I need most of the figures to be very simple, and a low piece count.

On the other hand, if I build every dance out of the same small number of building blocks (ex: reshuffles of Galopede), dancers will start to feel "haven't we done this one before"? So I also like dances that have a "hook": an interesting figure that we don't do in the other dances. Some examples:

• In Jacob's Potato the whole set circles (well, potatoes) left and right.
• In the Low-Backed Car each line takes a turn dancing around the other.
• In Charge and Drag the top and bottom couple each arch over one of the lines, and then additionally charge towards each other and drag to the bottom.

Other hooks I like include lines weaving between arches and the whole set casting to the bottom ("peel the banana").

I was looking for more dances like this, especially ones that fit in small spaces (no down the hall) and aren't too picky about the number of couples (5 - 8). I asked the Trad Callers mailing list for ideas, and got back a bunch. After looking through them I added two to my repertoire:

• Witch's Reel (thread the needle)
• Falling Masonry (arch over lines; cascading sashay)

Of the ones I didn't add, I think the zig-zag poussette figure in Wee Willie II would be fun to adapt, but the dance as written takes more space than I often have. I'll probably look for another dance with that figure (or write one).

I also saved two simple duple minors where roles don't matter, for when the crowd is ready for something slightly more complex:

• Riverside Jig (simple duple minor with lines of four arching down over the whole set)
• Quandranella (simple duple minor with petronella)

Discuss

One of the most important principles in engineering is the KISS principle — Keep It Simple, Stupid. The best engineers are the ones that rigorously adhere to this principle. The worst engineers are the ones that spawn endless complexity and write 20 microservices for a CRUD app. But the KISS principle doesn't just apply to engineering; it applies to many other things in life as well — including public communication around AI safety.

I strongly believe that whenever you communicate to the public about AI safety, you should adhere to the KISS principle. AI safety people like to throw around terms like "instrumental convergence", "Omohundro drives" and "mesa-optimization" because these are the terms they are used to (including on this website). And I have nothing against those terms — technical jargon is what lets "insiders" exchange complicated ideas in a few words. But in public communication this is completely unnecessary.

Because the core AI safety pitch is actually really simple, and you can present it as follows: Big companies are currently spending hundreds of billions of dollars trying to build machines smarter than us. Nobody understands how these machines work. In fact, the evidence so far says that they lie, cheat and manipulate. If that stays true, we will either lose control of our future or all die.

This is a really simple pitch as far as pitches go. For example, the climate change pitch is much more complicated — you have to explain the greenhouse effect, for example (quick, give me a four-sentence description of the main processes behind the greenhouse effect — you probably can't).

I know that this pitch is really simple because I have tried it and it works. Even on non-technical people. That's because this pitch builds on ancient intuition. "We made something more powerful than ourselves and now we cannot control it anymore" is literally as old as humanity itself. The Bible says you shouldn't play God. Everyone has watched the Terminator.

Actually, on that point: AI safety people sometimes treat the Terminator comparison as an embarrassment to be preempted ("no no, it's not like the Terminator"), but I think that this is backwards. The public already has the concept of "we built super-powerful machines, we lost control, very bad". Why not use it? You can correct a few details ("no, actually, there wouldn't be robot skeletons, they would probably create a supervirus"), but the core logic is pretty accurate, no?

So KISS AI safety. You don't need the words "instrumental convergence" to explain to a layperson why this is important work. You don't need to demonstrate your fluency in the vocabulary (yes, the Gray Tribe is not immune to that status dynamic, as it turns out). You don't need to sound smart.

You can just say four simple sentences.

Discuss

Abstract:

1. We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or enforced, and a serious international (or even just national) regulatory regime would probably cut most of the risk.
2. They are not applied because awareness is low. The people who narrate and enforce AI policy mostly do not believe in the problem. I estimate that a majority of the top ~100–1,000 most influential policymakers worldwide have never had a single serious conversation about catastrophic risk, and this is the main reason they are not worried[1]. Even among the civil-society organizations that showed up to the UN Global Dialogue, exactly one of the 1,534 written submissions mentions "takeover", and less than 1% mention x-risks.
3. They've never had the conversation because our field under-invests in having it. Status rewards research over advocacy (~3.6 researchers per advocate in US AI safety); many organizations self-censor; funders treat repetition as redundancy, even though repetition is how anyone actually gets convinced. Meanwhile, the industry secured 7× as many meetings with the European Commission on AI as civil society (2023).
4. Therefore, an additional unit of effort does more good through advocacy and engagement than through research. Judge work in AI governance by minds moved, not by clever papers. I give a list of potential directions/projects in the last section to alleviate this problem.[2]

We're plausibly only a few years from a catastrophe. Fable 5 cracked open a brief window of attention, but policymakers are still worried about the wrong risks. This is our chance to wake them up.

Source: February 2026 Summit on Existential Security survey of AI safety leaders. Advocacy, policy and governance were stated as the top priorities. It seems to me that there is still much to do to act on this.

⚠️ Epistemic status: I have skin in this game, which is either a conflict of interest or two years of data, depending on how you see things: I run a think tank that does this type of activity, so discount accordingly. I preferred to ship quickly rather than not ship at all, or ship too late. I expect some claims not to be stable under reflection, but the core argument is one I hold with reasonable conviction. See this as a bottle in the ocean. My point is not to dunk on research. I think that research is how we keep finding unknown unknowns; nothing in this post argues for stopping it. AI safety is one of the hardest fields to navigate, and I’ve often wondered if what I do is pointless. I might be wrong about the net-positiveness of some types of AI regulations, but I feel that the level of the discourse is really bad, the conversation is not happening, and I want this conversation to happen before irreversible things start happening.

Thanks to Epi Gedeon, Arthur Grimonpont, Alexandre Variengien, Jack Stennett, and Jonathan Salter for useful feedback and suggestions.

        

1 — The bottleneck is political will, not research

This section defines what I mean by political will and then argues that we are not applying basic best practices, so having more ideas is clearly not the bottleneck.

What do I call “political will”?

Start with a single policymaker. They have to move along a pipeline like:

• Level 0: aware that serious people think there's a big problem.
• Level 1: convinced of AI x-risks.
• Level 2: engaged: actively pushing for existential safety.
• Level 3: champion: actively pushing for safety even at some cost or in an isolated way

"Political will" in the aggregate is just this funnel run across the people who set, enforce, and narrate policy.

Spoiler: Level 0 sounds like a low bar. It is. The median meeting I've had with a senior policymaker starts below it. In the international forums I've attended, my rough estimate is that at least a third of the policymakers I've personally met are not even at Level 0. Domestic legislatures are a bit further along: 40 current members of US Congress have now publicly discussed AGI or loss of control, up from a handful in early 2023, doubling roughly every 5.5 months[3]. Those public discussions are at Level 0 or 1. This covers 7% of Congress. At Level 3, I count roughly 3.[4]

The best practices we already have are not being applied

Unfortunately, red everywhere. https://ailabwatch.org/ .
CeSIA will soon publish something more up to date on the Code of Practice.

According to SaferAI’s rating, 35% is currently the highest overall assessment score, given to Anthropic. 59% is currently the best overall assessment score if a company adopted all the industry best practices found across companies.

Where’s my DNA synthesis screening? DNA Screening is an extremely reasonable, long-advocated measure that was even mentioned in the American AI Action Plan, but is still not mandated. It's the canonical example of how glacially even obvious, low-cost regulatory measures move forward (global synthesis map).

I like this quote from Buck Shlegeris: "Five years ago I thought of misalignment risk from AIs as a really hard problem that you'd need some really galaxy-brained fundamental insights to resolve. Whereas now, to me the situation feels a lot more like we just really know a list of 40 things where, if you did them — none of which seem that hard — you'd probably be able to not have very much of your problem. But I've just also updated drastically downward on how many things AI companies have the time/appetite to do."

The priority bottleneck is not finding more best practices: I agree we don't know robustly how to align a superintelligence - but at the same time we are not even willing to implement current best practices. The 80/20 playbook against scheming may not be enough, but we aren't even doing the 80/20. For example: Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes.[5] 

Basic measures like transparency are still not applied: companies reporting incidents are more the exception than the norm, and we don't know how AIs are aligned concretely (we have very little confirmed public information about why frontier AIs end up being apparently behaviorally aligned). Most of those best practices are basic ideas that have been on the table from the start. And since companies don’t seem to adopt them under current competitive pressure, implementation has to come from enforcement.

I could go on and on with this, but I think this makes the point.

We need to go from plan D to plan A: more seriousness and coordination

Greenblatt has tried to put numbers on what political will buys. He sketches a spectrum of how hard the world is trying, from Plan D, roughly today's world, where maybe ten people inside each company are trying to implement safety measures, up to Plan A, a strong international agreement with real enforcement and a slowdown. His tentative estimate: conditional takeover risk falls from ~45% (Plan D) to ~7% (Plan A). An ~84% relative cut, almost all of it bought by political will (Redwood Research).[6]

Obviously, the exact numbers are not the point here: directionally, strong political will and taking the risks seriously would tremendously reduce them.

Aligning a superintelligence may well be genuinely hard, and even plan A might be insufficient. That's an argument for more political will, not less: it's what buys the time, and above all, the seriousness that a hard problem demands.[7]

P.S. added on the 11th July: This week, the AI Futures Project, Greenblatt included, published AI 2040: Plan A, a detailed scenario of how this could go. The whole thing hinges on a US–China agreement by 2029. The best plan the research community can produce depends on a political precondition we are far from, and I think that there are substantial ways to improve upon the baseline: one of my main critiques has always been that the framework treats political will as a fixed variable rather than a strategic lever.

Winning requires a large fraction of the top ~100–1,000 to understand the problem, and we're far from that.

There are two funnels that are mostly independent:

• The belief funnel for policymakers[8]: the Level 0→3 pipeline above, run across the people who actually decide.
• The public incentive funnel: making inaction politically costly through public salience.

Both funnels matter, and I won't adjudicate between them here; but the belief funnel is the one that's barely started, and it's the one the rest of this post is about.

Why not just target Trump? Why do we need conversation with the top 100 people?

• Because Trump is inaccessible directly.
• Because policy doesn't execute itself, and because you can't reach the top by aiming only at the top[9]. A law is only as good as the Office that enforces it, the advisor who drafts it, the minister who prioritizes it, the journalist who narrates it. (This will be particularly true for the AI Act). And you can't reliably convince a president head-on: he is roughly the average of his advisors' views, who are in turn shaped by the media and their environment. There's no robust shortcut around that layer. So the real target is the ~100 who set direction plus the ~1,000 (or maybe the top 10,000 if you can’t access the top of the pyramid) around them: staffers, cabinet advisors, think-tankers, top journalists. If they don't understand the risks, it’s really hard to make substantial progress.
• For law enforcement. Even when a robust and well-designed law is finally in place, political will remains a major bottleneck for actual enforcement. Without greater political salience around AI risks, it is entirely plausible that the AI Act could result in no meaningful penalties, even in cases where AI providers are plainly failing to comply (e.g., this has been the case with the EU Digital Services Act for social media: barely any strong enforcement[10]).

So the target is not one man but a layer of ~100–1,000 people whose beliefs will decide on the level of political will.

2 — Almost nobody realizes how bad the situation is - hence no political will

We won't get governance without agreeing on the problem, and we won't agree without waking up.

Unfortunately, we are still sleep-walking.[11]

What the people in charge don't believe

There is much that I’d like policymakers to know, but if I only get 5 paragraphs, I think it would be the following high-level ideas (obviously, not necessarily written like this):

Superintelligence is a very real possibility. Most decision-makers are busy regulating bias and deepfakes, even while the open secret is out: AI companies are openly racing toward systems more capable than humans across the board.

It's near, not sci-fi. The people building these systems put transformative capability within a few years. This sense of urgency is largely absent from AI policy. And that’s a problem because nobody prioritizes a problem they expect to arrive after they've left office.

Companies themselves can't robustly control it, and don't even claim to. The implicit belief in the room is "surely the people building it know how to make it safe." It's false. Companies have never claimed to have solved superalignment. They made very public announcements that it was still unsolved.

Recursive self-improvement changes the whole picture. With RSI, everything accelerates, including risks such as mirror life, interestingly lethal pandemics, and large-scale cyberattacks.[12]

No one else is handling this, and the tail is mass death. There are no hidden adults in the room, and we are completely in the fog, operating under wide uncertainty. Risk of loss-of-control or engineered pandemics, with a large fraction of people dead in expectation, is very much on the table.

Sadly, these five beliefs are almost completely absent in the rooms where AI policy is made. It will be hard to mitigate the risks of a problem that you don’t name.

The NGOs are wildly misprioritizing

We scraped the submissions of the UN Global Dialogue. Out of the 1,534 UN Global Dialogue submissions:[13]

• "cyber" appears in 518 of them (~34%)  
• but only 32 talk about biorisk (~2%)
• 15 mention "superintelligence" (~1%)
• 15 "artificial general intelligence" (~1%)
• and exactly 1 mentions "takeover" (~0.1%)

So, yeah: almost no one talks about the risks that matter most, even among the organizations that showed up to an AI-governance consultation.

The public is still apathetic, and when prompted, mis-prioritizes

Salience is low but rising: Prompted concern about AI risk is rising (when you ask: "what do you think about AI?"), but top-of-mind salience remains very low (when you ask: "what's on top of your mind, dear citizen?"). In most general issue-priority polls, AI barely appears as a category, and it is discussed far less than inflation, purchasing power, immigration, jobs, health care, crime, or security during elections.[14]

The good news is that this top-of-mind salience, while still low, is climbing fast.

Within AI risks, catastrophic risks lag behind: conditional on talking about AI, catastrophic risk sits below near-term concerns, though the gap is modest.[15] But not being in first place has a cost: the main AI risk prioritized at the last G7 was child safety, and loss of control didn’t make it into the ministerial declaration of the G7 (cyber and CBRN made it finally; that’s a positive development).

3 — Why is awareness so low?

Sections 1 and 2 were about the world: the bottleneck is belief among the people who decide. This section is about us: if that's the bottleneck, why isn't our field attacking it? This is the meat of my critique.

There are a few factors we don't control:

• Nothing legible has happened yet (Terrible things arguably have; just nothing the public could see that would point to catastrophic risks)[16]
• Trump.
• We haven't had much time so far - ChatGPT was released only 4 years ago; we just need more time
• A few groups of smart people (e.g. Yann LeCun and his friends, many economists, etc.) disagree with us.
• States feel pressure to accelerate; the Draghi report makes them hate regulation, even if Europe was already lagging before the AI Act.
• Incentives. It’s super hard to coordinate on a dual-use, economically vital technology. Some people say all of this is absolutely determined (cf, The future of AI is already written | Mechanize, Inc.)

But I think that beyond those factors, we are simply under-investing in engagement & advocacy. So here are a few elements that explain why we are not on the ball:

We're not in enough rooms

A large majority of the people who organize the summits, sit at the UN, work for the OECD, or staff the Commission have simply never had the conversation (to be clear, some of them had the conversation and dismissed it).

According to Corporate Europe Observatory, of 97 senior Commission meetings on AI in 2023, 84 were with industry, 12 with civil society, and 1 with academics; Google alone had 10, nearly matching the total for civil society combined.[17]

If we won't name the risks, who will?

Even when we are in the room, we self-censor.

In October 2025, Yoshua Bengio posted about the first Key Update to the International AI Safety Report. Connor Leahy replied:

"While I highly respect Yoshua and the titanic effort that goes into compiling such reports, it is disheartening to see the complete absence (even downplay) of discussion of superintelligence, existential risk and loss of control."

If the people closest to the problem self-censor, the signal never reaches the deciders.

I've found empirically that almost all the think tanks whose members discuss x-risks freely with me obfuscate their messages in public.

I also did, and still do to some extent, at CeSIA. For example, we recently revamped our website, and at some point someone convinced us to remove the risk page. I now think this was an error, and that even if we look a bit more institutional without it, we're losing in the long term.

A larger example, where I was nearly complicit myself: a joint submission to an international consultation from around ten civil society organizations, most of which had signed the Global Call for AI Red Lines. The final text names no specific risk at all. Not even cyber. The reasoning was pretty sophisticated: the document was meant as a door-opener, and the worry was that naming a risk high officials might disagree with would get the whole letter dismissed before anyone read the rest. CeSIA was invited to co-sign. I went back and forth for two weeks, pushed to name the risks, and in the end we declined to co-sign and submitted our own, explicitly naming the risks. I want to be fair here: these were thoughtful people making a defensible bet about a specific audience. But there is a pattern: a coalition of organizations that privately take catastrophic risk seriously produced a public document that refers repeatedly to "shared understanding of unacceptable risks" without ever naming the one that matters most to them.

To be clear, there are multiple schools of thought on institutional engagement, and I still think it sometimes makes sense not to be maximally blunt about AI risks in a first meeting with a policymaker (say, if you can get a win with a recommendation that doesn't depend on understanding catastrophic risk). But overall, I've been surprised by the relative absence of risk explanations in major think-tank submissions.

An elegant option for organizations that believe in the risks but won't name them publicly: use your convening power to invite researchers who will:

Have you heard about AI Safety Connect? I’d bet most people on LessWrong have not - but I can tell you that the team working there is doing heroic coordination work. They organized very large side events during the series of international AI summits and created some space for the community, while inviting people outside the field who have to sit down and listen to Yampolskiy’s 99.999% Doom argument. Some policymakers were shaken after the event. Hilarious and effective.
– from the Invisible side of AI governance.

Status rewards research; the work that moves policy goes unrewarded

US AI governance has roughly 3.6 researchers per advocate [18].

Research is high-status; the work that moves policy is often invisible and unrewarded. And think tanks are often evaluated by nerds with research instincts whose hobby is often reading blog posts and fascinating new arguments.

There is no h-index for minds changed.

Obviously, research is safer to fund, and its downside risk might just be wasted effort, whereas the standard argument is that advocacy can backfire, for example by making AI policy partisan or locking in a flawed regulatory regime. But look: AI policy is already partisan, and we are already in a flawed regulatory regime.

Let's be clear: More research is the right call for genuinely open questions, such as digital sentience. The error is applying it to risks we already understand well enough to act on, in which further study becomes a form of avoidance.

Most likely, no one will read your 50-page paper.

I don't know why CAIP, one of the few AI policy shops in Washington, didn't get funded. Its strategy looked sound to me, the numbers impressive, and the director's LessWrong sequence was early and didactic on many of the points in this memo. [19]

Meanwhile, applications to safety programs have multiplied many times over. Still, there's no lobbying pipeline, few execution seats (per the MATS talent study), and the ecosystem might not be scaling adequately to absorb this talent.

One exposure is not enough; repetition is how you convince

Humans are not superintelligences; they rarely update immediately in response to evidence.

Changing opinions takes time; even the numbers from the best methodologies are sobering. Broockman and Kalla's deep-canvassing study found that ten-minute doorstep conversations produce only ~0.08 standard deviations of attitude shift, roughly nudging someone from "somewhat opposed" to "slightly less opposed," not from opposed to supportive. Deep canvassing is the gold standard. By contrast, brief campaign contact has nearly zero effect on voting choices.[20]

If extended, personal, face-to-face conversation with motivated people is the most effective persuasion format we know, and it moves the needle so little, then what should we expect from a 30-minute meeting with a minister's chief of staff who has twenty other priorities?

The answer is almost nothing.

Single conversations cannot be the plan.

Repetition is how you get things done: by slowly growing salience in the ecosystem, doing the invisible work of agenda-setting or coalition-building, to get independent voices amplifying others.

The number we currently deliver to most policymakers is zero. Agenda-setting research (McCombs and Shaw) shows issues need repeated appearances across multiple channels, from multiple actors, before they become priorities.

I often won’t listen to a problem that’s raised to me unless I get the same message from an independent source.

In research, novelty is the main value. In governance it's almost the opposite: several people pushing the same thing, independently, is what works. Authority arguments like: “This respected person also pushes for this” are how to get things done.

I'd most like funders to stop treating redundancy as a reason not to fund AI governance work.[21]

(Consider this post my own Level 0 conversation with the field. The model predicts I'll need to repeat it two to four times).

The field is comically small

Many times, senior people told me, "that institution is already covered," and then I'd find near-virgin land. Coverage, on inspection, sometimes means one person gave one talk there in 2023. To my knowledge, CeSIA was the first org to present (privately) on loss of control in 3 major international institutions.[22] That's why I think we are dropping the ball at the ecosystem level: the work is so under-resourced that one small team keeps finding itself first.

And there are still massive low-hanging fruits everywhere.

Being outgunned by industry is only half the problem: the entire field is undersized against any comparable effort.

As of COP 30 in 2025, the UNFCCC has admitted 3,907 NGOs as observers, and Climate Action Network alone spans more than 2,500 organizations across over 150 countries. AI safety governance has roughly 45 non-technical organizations and ~500 people, most founded in the last five years (McAleese 2025).

To my knowledge, fewer than 5 core AI Safety organizations engage with the UN in some fraction of their time (note: not counting what’s happening at the UN Global Dialogue). That's two orders of magnitude smaller than the field that fought climate change[23]. And remember that only about a fifth of those ~500 people are knocking on doors rather than researching.[24]

I'd say that in France, 5 people are meeting policymakers and journalists in total (while juggling an insane amount of work). Not more. Is this sufficient to wake up a whole country? I'm not sure the number is much higher when counting people working in Brussels who are willing to talk candidly about risks.[25]

(I’m less knowledgeable about what’s happening in the US, so maybe I’m overindexing on my experience)

The main objection: Let’s just wait for a warning shot? I don’t think this works.

The strongest counterargument to all of this is that the slow advocacy work doesn't matter, because the evidence will eventually speak for itself, whether through a crisis or an event like Mythos. In most other fields, the safety regime arrived primarily after a disaster: aviation after a string of crashes, nuclear after Three Mile Island. So why not wait for AI's equivalent?

Because, first, a crisis is partly constructed. A warning shot is just an event; it becomes a regulatory moment only if the environment is ready to notice it, and channel it into concrete regulation. Holly Elmore explains why we can't just wait for the cavalry: for a warning shot to update someone, three things must hold at once: a) the event provides information they already believe would confirm AI is dangerous, b) it does so in a quickly recognizable way, and c) it points at an obvious next action. Notice the word already. People need the dominoes pre-loaded: from capability to dangerous capability to short timelines to, finally, catastrophic risk without meaningful action. Otherwise the event means nothing to them.

For example, the AI Safety community broadly agreed that observing deceptive alignment would be an "absolute shut-it-down moment." Then Anthropic published the alignment-faking paper, and within days experts were debating whether it counted as the warning they'd imagined, and the moment dissolved.

The smoking gun, if it fires, won't fire cleanly. This is also why I'm skeptical of the view that technical evidence converts more efficiently than advocacy. Even the best strategy for political will, catching an AI red-handed, is probably unreliable. Yes, if this happened clearly, this would move political will faster than any realistic number of ministerial meetings. But first, the catch is itself a warning shot, and would need to be converted in a legible way to the media, and even if it comes, it probably won't be legible enough to convert decision-makers, and it may arrive only once development is too rushed and positions too entrenched to act on (Greenblatt, "How will we update about scheming?").

Anecdotally, I've felt this in miniature. I've shown someone an impressive video (to me) of a new robot. I expected "wow, amazing" and instead got "it's so slow and clunky, what are you talking about?" People don't see how insane it is that this already exists, and forget all the magic necessary for sand to process this information in the first place; the evidence is right there, and it slides off.

Mythos confirmed this again. In my own ministerial-cabinet meetings, it helped with tangential risks like cyber, but people stopped there, and even then I was met with "isn't it just hype?"  Most policymakers didn't wake up. They remained suspicious of Anthropic, and in every meeting since Mythos, I've had to be the trusted person in the room, saying, "Yes, Anthropic is very good at marketing and the raw capabilities are somewhat inflated… but the underlying trend is real."

Beyond Mythos, I claim in this post and this comment that we probably won't get convincing-enough warning shots before crossing the event horizon.

I agree that to some extent, Mythos, and the Anthropic–White House meeting that followed, did more for political awareness (on Cyber) than the work of every CSO combined. Maybe. But the missing mood is still there: the executive order almost didn't pass, the provisions remain fragile, and they still don't touch the risks arriving next: bio, loss of control. To the contrary, we got a big push for sovereignty, and safety feels even lower in the list of priorities. When people in charge of AI in a government don't know what a jailbreak is, that should be informative of where we stand, to put it mildly.

I hope it will become easier and easier to talk about risks as AI becomes more capable, but I strongly feel that we shouldn't just wait for a crisis. Mythos is already behind us. The time is now.

Other potential objections

Objection 1: "Policy now risks premature action that locks in the wrong frame." This is roughly Dean Ball's position. He takes superintelligence relatively seriously, but thinks a bias to action produces bad lock-in, and that the US government is incompetent and self-serving enough that light-touch regulation is the safer bet; his confidence threshold for intervention is much higher than mine. The crux is p(Doom): in the Tegmark–Ball debate, I lean heavily on Tegmark, and the disagreement boils down to Dean's low p(Doom). If expected takeover risk is about a coin flip, the risk of inaction dwarfs the risk of lock-in.

Objection 1b: Another strong backfire objection is in On Pessimization, by Richard Ngo: awareness-raising pessimizes when advocates lack concrete proposals, and then the energy flows into negative spirals like racing and creating new labs like OpenAI. I agree to some extent, but the main failure mode is awareness without asks, and fortunately, we now have many clear asks ready to go today, with regulations soon to be enforced in need of support (e.g., the Code of Practice of the AI Act, and some state regulations).

Objection 2: "Political will is low now, but it will rise on its own (as in AI-2027, where governments wake up late on their own), so the real bottleneck will be the verification mechanism." Political will probably won't rise on its own, and this doesn't exclude advocacy now: the two are complements, and each makes the other more effective. And I think basic verification mechanisms are already good enough to get started (see here); waiting for political will to rise on its own forfeits the preparation that determines whether a later crisis converts.

Objection 3: "If we're seen as advocates, we lose our seat at the table." This one is real, and I feel it personally: part of why we get invited into technical and diplomatic rooms is that we're perceived as a serious organization™. There is a glass ceiling for purely advocacy organizations in international institutions. 2 responses: First, this is an argument about who does advocacy and how, not about the aggregate allocation: mature policy fields differentiate inside voices and outside voices. Second, I believe there are elegant ways to present the situation, and the cost of talking directly about the risks has been greatly reduced today, given all the public statements and recent developments with Mythos/Fable.

Objection 4: "SuperPACs in the US are already doing this." Good, but they're aimed at the general public to shape elections, and they rarely reach the ~100–1,000 decision-makers who can make the real difference.

Objection 5: “Policymakers have heard about AGI and choose to dismiss it.” Dean Ball made this point after the Delhi Summit. He says that in global policy circles, talking about powerful AI is considered impolite, even a little discrediting, because “AGI” is heard as an American imperialist construct. But Ball himself changed some minds in Delhi with a report in hand. And if the dismissal comes from distrust of the messenger, I believe this is an argument for advocates independent of American labs, not for more papers.

4 — What to do

Naturally, I'm more confident about the problem than on the solutions. This section is more speculative.

A. Do the direct workThe bottom line: talk to more people!

Knock on the doors of media, policymakers, and influential institutions, and keep knocking.

Both the level of investment in advocacy & engagement with stakeholders and the allocation away from pure research have to change.

Judge this work by the number of minds moved.

A rough back-of-envelope: US AI-governance work currently runs at something like the ~1:3.6 advocates-to-researchers ratio documented in Section 3. I think it should be closer to 1:1, maybe even 3:1.

ControlAI did a good job creating the playbook for the outsider game (the Direct Institutional Plan). Here’s my tentative playbook for the insider game.[26]

Samuel Buteau, alone at ControlAI, followed the playbook for the Canadian Parliament, formed a cross-party group of MPs who publicly signed the statement, and triggered a series of parliamentary hearings on superintelligence risk (Canada Campaign Statement | ControlAI). It seems to work!

Contribute to open consultations

A cheap way to contribute is to submit to open consultation. As we saw in Section 2, almost nobody raises the risks that matter most in these consultations, making a marginal submission unusually visible. If you want more people to talk about what matters to you, submit something in the next such consultation.

Advocacy aimed at AI lab employees

CEOs have an insane level of access to heads of state, so getting them to speak more candidly about risk would be unusually potent. Employees are among the few people positioned to create the internal pressure that shifts what a CEO is willing to say.

They can sign internal and public statements, push for stronger commitments, dissent on the record when safety pledges are quietly weakened, or even quit with a viral tweet when it's time to speak up more loudly. I think this is probably high-leverage.

Explain what you truly believe instead of just making a brittle recommendation

A recommendation adopted without its underlying rationale is quite brittle: the moment it's inconvenient, or the situation changes, no one downstream defends it, because no one understands why it's there.

Also, making good recommendations is hard, takes time; if you say something dumb, you lose credibility.

There is no universal rule here, but I'd lean towards an environment where policymakers are exposed to the risks worldview rather than one where they receive only shallow recommendations that do not generalize.

B. Build the credibility infrastructurePrepare to own the next crisis

As I argued above, a crisis only converts if the ground is already prepared. So, concretely: have the analysis, the asks, and the relationships ready before the event lands.

I tried to convert as much as possible from within CeSIA in the Mythos moment, but we made mistakes and were too slow. We need to be more prepared.

For example, if at some point we get clear architecture leveraging Neuralese in production[27], I predict it won't be clear to the media why this is bad - and this requires a lot of awareness-raising and explanation ahead of time that I'm not seeing much of.

Pre-register what you expect to see

I think safety people should win Bayes points and credit for being early, but we're not really getting those points. The fix is to state publicly and collectively, before the event, "this specific observation will happen." If the event lands, you point at the registration, and hopefully you get listened to much more carefully.

If you work in AI governance, make your worldview public; it's an investment. We could have won a ton of points when AI started eating Erdős problems.

Track the skeptics' predictions

This follows the previous point. Some skeptics won't be moved, and that's fine; the goal isn't unanimity. You don't have to convert the irredeemable; you have to make their claims progressively less credible to the people watching, by being the side whose predictions are borne out over time. Luc Julia, the second most prominent skeptic in France after LeCun, never changed his mind, but Mr Phi, a prominent French YouTuber, made a very visible video showing that several of his claims were factually false, and it stuck. We need more of this type of analysis.

C. Communication at scaleViral communication

Obviously easier said than done, but viral comm can be hugely effective, and there are ways to manufacture it. AI-2027 and Europe-2031[28] reached some of the relevant 100–1,000 stakeholders. We should make more of these: tune each one to a different constituency (national security, EU competitiveness, labor, biosecurity) while carrying the same small ask.

Break the "sci-fi" stigma

Yes, superintelligence and human extinction sound like science fiction, but they are the actual variables driving the risk. If we hide them, we prevent the Overton window from shifting at all.

So we need intentional strategies to normalize these conversations. The CAIS statement was a good first step. I think that more is possible in this direction.[29]

Invest in coordination mechanisms between AI safety advocates

The task of industry lobbyists is much easier than ours because they are aligned in what they ask for: they all want less regulation. By contrast, AI safety advocates are often far less united in their demands.

We could coordinate around a few shared demands to create a voice that's actually unavoidable at the UN Global Dialogue and other summits, rather than arriving, as we do now, as scattered voices.

This is what the International Campaign for the Abolition of Nuclear Weapons  (ICAN) did: it got hundreds of organizations to say one thing (ban them) until it was on the agenda, and won a Nobel Prize for it.

Concretely, this means being willing to move toward a common ask rather than holding onto my own variant. Be it the IAEA for AI, specific red lines, or anything else. We should have this discussion publicly. The specific ask might matter less than the convergence: whatever we choose, the value is in saying it together.

Over 200 of the 1,534 submissions to the UN Global Dialogue spontaneously call for “red lines”[30]. CEOs have recently asked for international standards, and for an IAEA for AI. An IAEA for AI sounds utopian until you remember the actual IAEA was built in four years by people who had just finished bombing each other. Maybe this can be the basis upon which we coordinate.

D. An opinionated list of research directions that I find most useful

I started drafting this post under the title "Political Will, Not Research," and softened it to "the current bottleneck" for a reason: some research bears directly on the bottleneck. Here are a few directions:

Research on how to convince people of the problem. If the bottleneck is understanding, then how to build understanding that converts is itself a neglected research question, and it seems almost nobody studies it systematically. Seismic's report On the Razor's Edge: AI vs. Everything We Care About (2025) is a start, and its findings are counterintuitive. It might be the case that the vast amount of advocacy to date was for nothing, and the best strategy is "issue bundling," where people reach AI-risk concern through what they already care about (see, for example, job loss or mental health), and only then talk about catastrophic risk. But it's nearly the only systematic work I know of, and we need far more: What actually moves a cabinet advisor from "cyber" to "loss of control"? Which framings convert? CeSIA had to experiment from scratch, and that’s probably the same for many orgs.

Research that helps turn will into requirements. Prioritizing the asks the AI Office should make of companies, and the risk-modelling methodology to hold them to it (argued in A Call for Better Risk Modelling: this is urgent since CoP enforcement starts on August 2); auditing the thresholds and mitigations companies publish, continuing what AI Lab Watch has been doing (OpenAI's red line for AI self-improvement is fundamentally flawed); operationalizing and harmonizing red lines across jurisdictions (AI Red Lines: A Research Agenda), or the recent draft treaty proposal signed by a coalition of international experts.

Research that measures the progress in political will. The effect of advocacy is mostly illegible.[31] But even if the causal chain is hard to be sure of, we can still measure the aggregate effect, and potentially fund more METR-graph-for-policy, like the AIPN tracker cited in Section 1, and better understanding of what’s happening under the hood.

Research that creates demonstrations of risks. Model organisms like agentic misalignment: I use this paper in all my presentations to policymakers now - I think that it is the best paper to demonstrate that frontier models could be dangerously unaligned, which is still one of the main bottlenecks (if not the main one), for AI risk to be taken seriously and prioritized accordingly. I thought in the past that we already had enough risk demonstration (Sleeper agent, alignment faking, Mecha Hitler, ChaosGPT...) but no, this is really a substantial improvement.

Engineering that makes "yes" cheap. Factorize technical mitigations across labs: a shared, off-the-shelf library of safety techniques (constitutional-classifiers-style) that Chinese labs or Mistral could adopt seamlessly.[32]

Research that could flip the strategy's sign. Advocacy without red-teaming is how you lock in the wrong ask: Human takeover might be worse than AI takeover. I'd like to fund the research that proves this post wrong and tells me what to do instead. There is no shortage of such cruxes[33].

Addendum

Two closing notes on why I expect this thesis to become more true over time.

Political will is (or will be) the bottleneck for nearly every cause, not just AI safety

Don’t take all of this personally; AI Governance being slow is not an AI safety-specific concern.

Ok, yes, to some extent, this doesn't apply to causes that are still genuinely researchy with no clear ask, for example, where "keep doing the research" is the right call. But for most causes with a known ask, the binding constraint is coordination and political will, not more analysis.

It’s funny, because I see some people in the ecosystem starting to take space governance seriously. Yet the UN has worked on it for roughly fifty years.[34]

Same for power concentration: if politicians were AGI-pilled, they would act much more rationally around this.[35]

(This probably won’t happen in the short term, and it should only be considered after we handle AGI, but getting politicians to be more rational, scale-sensitive, and epistemologically sound in the long term would, in general, be incredibly useful and beneficial for society.)

We can automate the research, but not the consensus necessary for political will

AI will clearly accelerate technical safety work, and even governance inputs. I expect AI to be good at finding ideas at some point and a pretty good forecaster. But it seems, empirically, that people don’t care about forecasts; they don't care that top forecasters with a good track record, like the authors of AI-2027, are raising the alarm.

AI won't, by default, be able to accelerate agreement, consensus-building and human engagement. Consensus is a human process, and it scales differently than research does.

So, as automation reduces the research bottleneck, the human-coordination bottleneck becomes even more important.

1. ^

    (not disagreement after consideration, but absence of the conversation itself)

2. ^

   This isn't the first time LessWrong has heard the case for reallocating toward advocacy. The post “Instead of technical research, more people should focus on buying time” made a version of it in 2022, and the community's verdict was that outreach backfires when done poorly, and that indeed it will be done poorly (Wentworth–Larsen), and this was probably reasonable on the evidence of the time; Katja Grace diagnosed the underlying aversion the same year. Then, that verdict has been reopened: MIRI pivoted its entire strategy toward communications, Ruthenis argued in 2025 that awareness is the bottleneck, and gave up on policymakers, which I'll dispute below. What the reopening has lacked is concrete evidence from inside the rooms. That's what I try to add: a model of what advocacy produces (the funnel) with 2 years of insider experience across European and multilateral institutions.

3. ^

   It would be nice if this law was as solid as the METR’s doubling trend of AI capabilities.

4. ^

   Senators Hawley and Blumenthal introduced the AI Risk Evaluation Act, a mandatory pre-deployment evaluation of frontier systems for loss-of-control and scheming behavior, with penalties,  and Hawley did it while his own party's administration was pulling the other way. Bernie Sanders talks about superintelligence and introduced a recent bill. I might be forgetting other initiatives.

5. ^

   Chain-of-thought (CoT) is the model's step-by-step reasoning trace. One of our better safety hopes is that we can read it to catch misbehaviour, but that only works if labs don't optimise the CoT to look good, which destroys its faithfulness as a signal. Anthropic has acknowledged inadvertently training against the CoT on more than one occasion. Doing it by accident is exactly the kind of basic process failure that should be easy to avoid. See the AI safety Atlas for an explainer on this.

6. ^

   Note that I value Greenblatt’s opinion since he has an excellent track record of forecasting AI capabilities.

7. ^

   (Note that we don't need to skip from Plan D to A to get substantial improvement - we can also push continually along the spectrum of political will, where each increment gets a worthwhile improvement)

8. ^

   (and people who wield power, which includes some people in AI companies)

9. ^

    (at least, if you have the time to read this)

10. ^

    Enforcement is a crude lever compared to a request for information and can also introduce adversariality into the relationship with the provider, so it is not entirely surprising to see that the Commission is very careful when using those powers.

11. ^

    That’s not surprising considering that even at NeurIPS, the biggest gathering of AI researchers in the world, a third of the AI researchers don’t know what AGI stands for, but yeah, we are nowhere near.

12. ^

    I now think that loss of control is probably more urgent than biorisks. See this analysis from PourDemain. I think loss of control is orders of magnitude more likely than irreversible x-risks from AI-enabled pandemics, even if both are probably already at intolerable levels.

13. ^

    We matched on exact strings.

14. ^

    When people are asked about AI directly, they often express concern; but when they are asked what political issues matter most, AI risk is usually absent or marginal. Concretely: YouGov's "AI will negatively affect society" rose from 34% (Dec 2024) to 47% (June 2025), yet AI does not appear at all in Pew's 24-item ranking of top national problems (Feb 2025) and sits near 1% in Gallup's open-ended "most important problem."

15. ^

    And the comparison is cross-pollster rather than head-to-head (no single neutral poll ranks them against each other). Pew (Aug 2024) finds people "highly concerned" about misinformation (66%), loss of human connection (57%), job loss (56%) and bias (55%); a separate YouGov poll (June 2025) puts concern about AI-driven human extinction at 43%.

16. ^

    Did you know that the Mexican government has already been hacked by an unattributed hacker using Claude? (Bloomberg, Feb 2026) It seems that, in an authorized red-team test, a frontier model reportedly compromised most of the NSA's classified systems within hours (NYT). The fact that this is not the top story in the world is also revealing about the information ecosystem.

    But again, this is not really a factor under our control, and I argue in this post that convincing warning shots are unlikely. I come back to this in the sub-section “The main objection: Let’s just wait for a warning shot? I don’t think this works.”

17. ^

    There are more lobbyists working on the EU's digital files (890 FTE in 2025) than there are MEPs (720). (Corporate Europe Observatory, Big Tech lobby budgets hit record levels)

18. ^

     202.5 vs 55.75 FTE, Green-Lowe / CAIP, An Activist View of AI Governance, 2025; author's estimate.

19. ^

    I don't know the grantmakers' specific reasons, and they may have been good ones. But I have the feeling that the ecosystem's revealed preference is that a new research org is easier to fund than an advocacy org.

20. ^

    In advertising, the exact threshold of repetition has been debated since Krugman's 'Why Three Exposures May Be Enough' (1972) — two, three, four? — but the general effect is well established.

21. ^

    (or at least to reduce this parameter in their weighted factor models)

22. ^

    I cannot name them publicly, but DM if you want private proof.

23. ^

    The comparison is loose; UNFCCC observers also include industry groups and all kinds of NGOs, while I'm counting only non-technical AI safety organizations (which is also generous in some respects). Regardless, I think that even with heavy discounting, the gap is enormous.

24. ^

    Remember also that the environmental movement is over fifty years old. Rome wasn't built in a day.

25. ^

    Michael Dickens's 2025 donor review also concludes advocacy is far more neglected than research and that the few advocacy orgs do not get much grantmaker support.

26. ^

    The insider game has documented integrity costs, and there is also a missing mood (Integrity in AI Governance and Advocacy). But I also think that it can be done correctly. E.g., I don’t think the most effective strategy is necessarily to open with the full Doom argument, and there are many strategies that lead to effective results and indirectly lead to more political will.

27. ^

    Why are Neuralese bad? See this: Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety 

28. ^

    I’m not sure Europe2031 was net positive. See.

29. ^

    Maybe a wild proposal, but one way to bypass the institutional stigma is to create safe channels for civil servants, policymakers, and lab insiders to express what they already privately think. Maybe organizing anonymous joint statements?

30. ^

    I must now admit we should have pushed harder on explaining the risks during the red lines campaign: removing the detailed explanation from the FAQ was a mistake.

31. ^

    “A lot of semi-invisible, ongoing-over-years, and hard-to-definitively-attribute work has gone into many of the policy 'successes' of recent years. And sometimes part of the difficulty in attribution is actually down to things like the social proofing of multiple groups providing independent evidence and testimony that adds up to a credible body of expert input on a topic.” –  source, Seán Ó hÉigeartaigh

32. ^

    It's been more than a year since the constitutional classifier paper was published, and there is still no ready-to-use library. This is a coordination failure because the same work is duplicated over and over, and because this is currently one of the most effective strategies for reducing misuse. Currently, a small team in each lab might be tasked with reimplementing the best mitigation strategies (which include CC), and safety teams at smaller labs are often just a handful of people. If this library were maintained and updated as new vulnerabilities emerge, my guess is that it would free up a lot of time for safety teams across different labs. This is high-leverage because it would raise the floor for the whole ecosystem.

33. ^

    See Zvi's Crux List for much more than you have ever asked for.

34. ^

    More generally, I think that the AI Safety ecosystem has a tendency to ignore the ecosystem and work in silos, disconnected from the institutions and international fora, while that’s where the governance discourse happens.

35. ^

    Yes, politics is the mind killer, but only because smart people disengage from it. This is a hyperstition that needs to stop.

Discuss

Introducing Plan A

The folks who brought you AI 2027, a so far remarkably accurate set of predictions despite those predictions having seemed freaky to many at the time, now bring you their positive vision that involves more freaky predictions: Plan A.

These guys have rather strong prediction track records. In addition to AI 2027, among other things, Daniel Kokotajlo has What 2026 Looks Like (which is remarkably similar to what 2026 looks like) and Ryan Greenblatt, who is also the chief scientist at Redwood Research, was the #2 most accurate AI forecaster in 2025 out of 413 entries. Past performance is as always no guarantee of future success.

If you’re the type to read at least some of my posts, or if you thought AI 2027 was worth reading, I recommend reading Plan A.

There is also an unofficial visual novel version, for minds very different from my own who would want that.

To be clear up front: I am not endorsing Plan A. I am not suggesting we should go off and try to enact Plan A as written. There is a lot more work to do and a lot of potential problems and downsides to grapple with.

I do think we should do that work, and give it, and its details, serious consideration.

The bulk of this post is engaging with various objections, in progressively more detail. If you only need the highlights, you can safely stop after Thus Selective Optimism.

Table of Contents

1. Introducing Plan A.
2. You Only Get Five Words.
3. Proactive Response To Objections.
4. Initial Introductions and Endorsements.
5. A Positive Vision.
6. Alternative Plans.
7. Plan S for Shutdown.
8. Something (Unexpectedly Good) Ever Happens.
9. Thus Selective Optimism.
10. Quickly, There’s No Time.
11. Race Conditions.
12. This Is A Lot Of Diffusion And Economic Growth.
13. Living In China.
14. Planning For Shifting Overton Windows Is Essential.
15. The Standard Handwave.
16. Some Equate Any Controls Over Compute To Authoritarian Dystopia And Those Same People Mostly Think Superintelligence Won’t Happen.
17. Vitalik Buterin Is Right, The Crux Is Future AI Capability Levels.
18. The Authoritarian Objection.
19. Concepts Of A Plan.
20. The Kitchen Sink.
21. Selective Claims Of Authoritarianism.
22. You Either Can Steer The Future Or You Cannot.
23. Cooperative Alignment.

You Only Get Five Words

As always, most people who interact with Plan A will not read it.

They will condense it into a few key sentences.

There is clear agreement on which sentences survive and which do not.

The top 5 things people will discuss will largely be, in descending order of focus:

1. We should slow down AI development.
2. To do that, we should make a deal with China.
3. We should monitor the world’s major sources of compute.
4. We should use mutually assured compute destruction (a version of MAIM).
5. Things will still happen super fast and feel like it, e.g. ASI by 2040, with vast economic growth before this.

Axios’s Ashley Gold compacts the plan thus:

Ashley Gold: The group behind a 2025 report predicting dire outcomes from AI development is out with a new prescription: To avoid dangerous outcomes from superintelligent AI, slow everything down.

That’s not technically wrong, but there is an obvious misinterpretation if you allow that much compression.

And includes this very good quote:

Daniel Kokotajlo: We think it’s still good to recommend what would actually be good, even if you think that your audience is probably not going to listen.

The strongest and loudest objection, and in some ways the best one, is some form of:

Plan A Detractors: Superintelligence is not coming any time soon. The threat is not real, so we shouldn’t be paying high costs to deal with it. There is no reason to slow down that which is already slow enough on its own.

At the extreme you get people like Joshua Saxe wondering why people didn’t learn their lesson from what didn’t happen to radiologists, and so on. Le sigh, but I appreciate saying it straight, and I appreciated Timothy Lee’s reaction even more:

Timothy B. Lee: I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I’m not sure where to start. There’s an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don’t.

I’ve found that people believe it at such a deeply intuitive level that it’s hard to have a meaningful discussion about it. Each side finds it baffling to encounter people with the opposite intuition, and on some level can’t believe they’re being serious.

I would hope that with enough time I could get Timothy Lee to come around, since he has established he takes arguments seriously, but so far I’ve been unable to find a compelling argument to convince such folks that for practical purposes yes sufficiently advanced AIs could and would do all the things.

I think the premise in the Detractors argument, as stated above, is wrong. I think superintelligence is likely to be coming soon, as do the labs. Many do not agree. If you are one of those who do not agree, then you should absolutely not want to implement Plan A, or anything like Plan A, and you should say so plainly.

This is true whether or not you want to further engage with the scenario anyway, to consider the hypothetical where you are wrong. That’s up to you, and ‘no’ is a respectable response.

Top 10 criticisms other are, translated into my language (not intended to pass ITTs):

1. America would never do it. You don’t understand America (or the government).
2. China would never do it. You don’t understand China (or its government).
3. This scenario doesn’t understand that this is a race. Or, this scenario places too much emphasis on the framing that this is now being seen and treated as a race.
4. This is still too fast, or this is far too slow. Unacceptable.
5. This is unnecessary, market can handle alignment, it is all easy, stop worrying. You warned things might eventually be not fine but so far everything is fine. I am opposed to anything vibing with the words ‘slowdown’ or ‘pause’ and will try to incept that any such action is impossible and discredited or ‘naive.’
6. This involves paying real costs and restrictions. I thought this was America. Controls over compute equal authoritarian (totalitarian?!) dystopia and all that.
7. This still would not work, either it does not work technically for various reasons or alignment is too hard. We need a full pause. Or, in the better version: There aren’t enough worlds where this turns losses into wins.
8. The economic projections are way too optimistic.
9. The scenario is confusing: It conflates realistic prediction with aspiration. Also people will focus only on particular key points, so you should obfuscate those.
10. The scenario involves unlikely things happening. Nothing ever happens. This is something happening. Also you made it up. Never gonna happen.

A lot of these being symmetrical is a sign that the scenario is doing something right.

My basic responses to these objections:

1. Not with that attitude. If true, get cracking on figuring out an alternative.
2. Not with that attitude. If true, get cracking on figuring out an alternative.
3. I think this is roughly the right amount of talking like this is a race.
4. Slower would be better if possible, it’s a question of what is achievable.
5. If you think this, then you should oppose things like Plan A, but you’re wrong.
6. Yes, to deal with big things you often have to pay real costs, but do not equate such actions with authoritarianism let alone totalitarianism. This plan tries hard to minimize this downside. If you have better implementation ideas, speak up.
7. I think this is a topic for healthy debate and a strong objection.
8. I think they are overly optimistic, but not crazy, and I think the plan and scenario survive having much less dramatic medium-term economic impacts.
9. This is the nature of such a scenario. I think they did the best they could.
10. Some seemingly unlikely things do happen. A scenario without any unlikely events is itself even more unlikely. And in AI, the one thing I know for sure is that a lot of big somethings are going to happen. Results literally range from maximally bad to maximally good, and ~zero movement is not that likely.

Extended versions of most of these are included later in the post.

Proactive Response To Objections

There is a long tail of other objections as well. Even after listing the top 10 I found that a large percentage of responses to my open thread were not covered.

AI 2040, like AI 2027, tries to proactively answer many of the concerns and objections people have, both common and rare, including via supplementary material.

For a rare example, Tom Davidson is worried about the ‘dry tinder’ problem of creating a compute overhang which they answer here and via the idea of mutually assured compute destruction. Curt Tigges also worries about the overhang.

Nathan Young offers Common Notes on Plan A, an attempt to bring community notes to the entire internet.

Initial Introductions and Endorsements

Scott Alexander, one of those who worked on it, writes an introduction and justification here.

Whereas here are some shorter pitches for why you should read it and it matters:

Romeo Dean: At some point soon, humanity will be forced to reckon with the creation of AIs that are smarter than humans in every way. This is a terrifying prospect. Plan A is our vision of how we can make this go well if we rise to the occasion.

Eli Lifland: Excited to finally publish AI 2040: Plan A, our plan for international coordination to get to a great AI future. We’ve put a lot of work into this and I hope that it sparks alternative plans that bring us closer to treating superintelligence with the seriousness it deserves.

Buck Shlegeris: I’m really excited this is out; I think it’s the best summary of the spectrum of possible plans for preventing catastrophe from AI.

Ryan Greenblatt: Plan A seems like a good plan for handling powerful AI, or at least the best plan anyone’s written up. Many choices initially seem crazy, but are actually pretty carefully considered. Plan A isn’t likely to happen, but pushing for something like this seems worthwhile.

A nice property of Plan A is that it’s reasonably robust to partial or lower-quality implementation. So nearby proposals still seem good (and worse-done versions can bootstrap into better ones). The core—verification and (very strong) transparency—suffices to go pretty far.

Eli Tyre: Plan A is the single most thorough and thoughtful plan that I’m aware of for what the world should do as we approach Superintelligence. If we want to get through the singularity alive, this is my current best guess of the playbook we should be aiming to implement. It’s my current best guess mainly because there are so few serious contenders that actually engage with the details of the situation.

Nevin Freeman: As you try to make sense of what *should* happen with AI, this is an absolute must read (or listen). It’s very hard to reason about potential catastrophe and how to ensure we avoid it, and it’s even harder to turn ideas into concrete stories many will understand. Huge thanks to [everyone involved].

Jeffrey Ladish: We’re in a pretty dire situation, but we can change course! I highly recommend checking out @DKokotajlo Plan A write-up & AI 2040 scenario.

Steven Adler: Very grateful that the AI 2027 team has a plan for the global calamity that is superintelligence [that arrives fast and before we are ready]. It is by no means perfect, but I’d love to see critics lay out a stronger competing vision; seems pretty hard.

Jonny Miller: Someone should fund turning this into a Netflix-quality production to give normies a heads-up of what is coming down the pike.

Andy Masley: AI 2027 now has a follow up called AI 2040: Plan A, which is a scenario where things go well. I got to be an early reviewer on it. I have extremely agnostic takes on the difficulty of alignment and the economics of advanced AI, but this scenario is the clearest concrete description of what the world could like under the specific worldview of:

-Extremely rapid AI progress
-Alignment right now is very difficult
-AI very able to substitute for human inputs in the economy
-Assuming in Plan A that things are governed well at the global level

Each of those I need to throw my hands up and say “Man I have zero clue at all” but I’m always happy when different sides in the AI debate make their assumptions clear, and the consequences of the assumptions here are really wild.

Giving these assumptions I’m a complete agnostic on, I was surprised at how difficult it was to find holes in the conclusions. The way the global energy system is shaped by data centers in a super high growth scenario sent me down a ton of rabbit holes among other things.

Charlie Bullock: The Bay Area rationalist preference for communicating important policy proposals via structurally postmodern medium-form fiction writing will never cease to amuse me.

It would be a kinder and gentler world if this were the norm everywhere. I want to start my next draft bill with a 25,000-word Choose Your Own Adventure novella set in late Victorian London.

To be clear I like the piece and think it’s good that people are trying to think about the future in a very detailed way and take things out. But it is just descriptively true that the publication is a work of fiction. The authors would not disagree with that claim.

Boaz Barak (OpenAI): I don’t agree with everything in AI 2040 “Plan A” but it is very thoughtful. One element I love: push for increased transparency and diffusion.

Instead of safety meaning locked down information and restrict frontier models only to labs, government, and chosen partners, a key component in their plan is to maximize sharing information and distribution.

Andres Rosa: Finally, a positive path

Jack Galler called it the most important thing he’s ever read. I wouldn’t go that far.

A Positive Vision

They call Plan A a positive vision.

One could also call it an optimistic scenario, as per Richard Ngo.

One strong criticism of AI 2027 was that it laid out two scenarios, but did not tell us what we should do or offer us a practical path forward. In the default scenario, which lays out what the authors expect to happen, we all die.

In AI 2027’s alternative scenario we do a hard reset at a crucial moment, which is less a strategy or plan or positive vision and more like a hail mary pass, where the authors make things turn out well to show that it is in theory possible.

Plan A updates to start from our present situation. Rather than being primarily a prediction, it lays out a positive vision of a possible future where we coordinate (including using various enforcement and verification mechanisms) to slow down the development of superintelligence (ASI), and give it the best chance to go well. Ultimately in 2040 the torch is passed to the AIs, the singularity proceeds in earnest, and hopefully we got it right on the first try and things work out.

The proposed implementation of Plan A is that America and China reach a mutually beneficial deal to slow down AI development and share research information. Joint control is established over existing and new chip supply, with common knowledge of the location of existing concentrations of chips, and universal auditing of data centers. Both sides can verify that the deal is being upheld.

This can sound rather fantastical, but if a few months ago you had described how the US government has responded to the whole Mythos situation, you would have sounded rather fantastical then too. Something has to give. The authors really did invest a lot of work on seeing which paths forward were viable here.

Things go well, including economically along the way, as we make a series of good decisions, and we get to a happy ending, although not without speed bumps as even a slower transition is rather sudden and even if the technical issues are handled there remain lots of real problems. They consider alternative ways this could go, such as China cheating aggressively or flawed safety cases being approved.

During the transition, America experiences explosive economic growth, and uses some of that to pay out a rapidly increasing Citizen’s Dividend, as an extremely generous form of UBI, and also a small fraction is devoted to various forms of defensive acceleration. Various issues are discussed along the way.

If things going vertical in 2040 still sounds super fast, that is because in the authors’ current baseline or default scenario this happens in 2030.

A key focus of Plan A is total research transparency. Daniel Kokotajlo made this diagram to illustrate why he cares so much, with further explanation at the link.

Alternative Plans

They briefly also describe the scenarios where we instead choose Plans B, C, D or S:

1. Plan A: Coordinated Slowdown, make a deal with China.
2. Plan B: Fight China, as in try to sabotage other efforts to buy a time buffer).
3. Plan C: Burn the Lead, roughly the AI 2027 scenario, with a bit more awareness.
4. Plan D: Race to ASI, the path we were on before Mythos and mostly are still on.
5. Plan S: Shut It All Down, which is what it sounds like. They are sympathetic but don’t recommend because it would be unstable and hard to get buy-in for.

Roughly speaking, they predict the presidential election in 2028 is all about AI because by that point it is obvious AI is the Main Thing, with candidates throwing around bold agendas. Then in 2029 one gets implemented, which they are hoping is Plan A.

There are a ton of details and explorations of different topics in Plan A. Those involved put a lot of effort into parts almost no one will see. I could easily write a response post at least as long as the scenario to go into them. This is not that post.

A lot of those details sound like science fiction, because they are serious attempts to predict the future. Best start believing in science fiction stories, because you’re living in one. There are any number of ways that can go, and this might be wrong, but some very science fiction things are happening and a lot more will be happening soon.

Plan S for Shutdown

This perspective, a form of Objection #7 above, is negative on Plan A’s chances of success, but positive on the chances it causes us to gather enough impetus to shift to Plan S, which is a full shutdown.

Nate Soares (MIRI): The AI futures folk and I agree on quite a lot. This is the best concrete vision of a positive future I’ve ever seen spelled out, bar none.

I doubt their Plan A would work as written, but it’d have a chance of producing evidence that convinced world leaders to switch to Plan S, which I think has a shot.

Why not Plan A? Humans working with AIs on alignment are likely to converge on wrong answers. The honest answer is likely “pushing to superintelligence using anything remotely like modern methods is fucked; back off”; humans are unlikely to listen rather than push through.

But if humanity was especially competent and especially lucky while pursuing Plan A, and was able to heed the warning signs, I could imagine us taking the off-ramp to Plan S and surviving. So attempting Plan A doesn’t look completely fatal.

Or alternatively:

Damian Tatum: Again, still reading the alternative paths, but

If you are of the camp that the odds are against us and the situation is grim, as I am and Nate Soares is even more so, then every winning scenario involves a lot of things going unexpectedly well.

The reason they give for not choosing Plan S is that they believe that an agreement would be unstable over time, so you have to keep moving forward and also give people a dividend of sorts. The agreement breaking down would potentially be quite bad.

Raymond Arnold thinks out loud here about the importance of ‘how long would such a deal hold?’ in terms of choosing the best plan within the class of As and Ss.

Something (Unexpectedly Good) Ever Happens

Every scenario, including whatever turns out to happen in real life, is going to involve some rather unlikely things happening. For any given thing, ‘nothing ever happens.’ For all things combined, surprising individual things happen all the time and each might be good. If this is not intuitive, look back on the last year, or any other period.

If you have a good plan to change the world, it is going to involve causing a bunch of otherwise low-probability events to happen, either intentionally or via good luck.

Of course, any good plan involves not relying on too many specific good things happening. You need to be as robust as possible, but no more robust than that.

Marius Hobbhahn: When I read it for the first time my intuitive reaction was like “too many low probability things have to go right for this to work out”

Which my brain somehow mapped to “therefore unrealistic” instead of the obvious “it’s a really hard problem and this is actually the sota plan”

I think it’s a good plan and I hope we see much more work on this, especially from the people who are deadset on building AGI.

Tyler John: I had the same reaction and I still do want more nimble plans. (I am also working on building the verification stack, walk and chew gum, etc.)

Plan A here definitely has some flexibility and room to recover from failures or to reroute, but not unlimited room. It involves good fortune, but not maximal amounts.

Thus Selective Optimism

Richard Ngo helped work to critique and improve Plan A during its construction, and wrote a critique of the high level framing, called Selective Optimism. He correctly describes Plan A as an ‘optimistic forecast’ but notes this makes it hard to tell which parts of Plan A are things to aim for versus things included for realism.

A lot of people reported struggling with this issue.

There’s no good answer, and no one best answer. If you tell a story, your story is going to have to choose one path out of many. It must balance realism with your hopes, in addition a bunch of individually unlikely things.

The alternative is to not tell a story and not have a scenario at all, to never tell stories or offer particular scenarios at all. I think that’s clearly far worse.

Most centrally, Richard would prefer a slower handoff than described in the scenario. I would as well, as would the authors of Plan A, and this objection is not uncommon.

Thomas Larsen points to this supplemental to illustrate which parts are aspirational versus realistic.

Quickly, There’s No Time

Richard Ngo recommended taking all the dates out to avoid people mainly focusing on the date 2040, since what is valuable are the details.

I think this echoes a lot of similar criticisms of AI 2027, and I see the argument but I rejected it then as well. You can’t tell a story like this properly without dates, you have to pick some point on the curve of potentials to lay out in a potential future.

If you try, guess what the first criticism would be? They didn’t put dates on. What a joke, what do we even do with that? And those people would be right.

In general I don’t like Isolated Demands For Anti-Virality lest someone somewhere accuse you of giving people the wrong idea or focus on the wrong aspect.

Yes, some people will stop at the headline, or latch onto only the date, and you can change the frequency of that somewhat, but mostly that cannot be helped, and those people were never going to engage with the real content anyway, or if anything this makes them more likely to then seriously dig in.

Race Conditions

Richard also challenges the ‘race with China’ framing, expecting other disruptions, but I predict that any such move towards Richard’s framing would cause most to dismiss the scenario as unrealistic, and also risks further politicizing the whole thing. Any failure to acknowledge that America and China really don’t trust each other at this time would make the whole thing land with a thud or worse.

I too of course have always hated the race framing, and tried to fight against it for years, but as Larsen says that’s how people think and talk now, especially in DC, and you have to acknowledge this.

For the opposite view on that, here is Poplicola saying that Plan A fails as a scenario because it doesn’t treat American hostility to China, and its view of ‘authoritarianism’ as the real existential threat (at least when it comes from outside the house) seriously enough.

Also things like this:

Kyle Corbitt: The tech development seems ~plausible but the politics feel quite naive. Like I just can’t see Canadian voters letting China construct thousands of datacenters in 2030, no matter how good the economics are. Esp if the pitch is “we chose you since you’re a soft target for the US”

I don’t see that proposal as an especially hard part of the problem, as long as the economics work out.

I think in general the way politics works is until the reasons are compelling things look hopelessly naive until suddenly things change, the atmosphere shifts and then they happen. Peace treaties (and other agreements) often involve concessions that looked impossible, right until the deal got signed.

And I think Richard is conflating the ‘race with China’ we are currently in, as in allowing OpenAI and Anthropic to push full speed ahead while we make at least some attempt to deny China chips and otherwise tip the balance, while any attempt to do anything to help is by default dismissed as ‘if we do that we Lose To China’ with a theoretical full-on planned Race With China that I agree we are not doing and are not likely to do any time soon.

This Is A Lot Of Diffusion And Economic Growth

Richard ends with a criticism of the expected fast pace of both AI progress and the diffusion and impact of AI in the broader economy, saying he doesn’t expect AI progress to generalize. You certainly can argue that the forecast here is overly optimistic, even as things spend years going along exactly the path that these styles of prediction would predict. I just don’t think a strong form of that is the way to bet.

In terms of the diffusion and real world impact, which I think is the strongest critique, I agree with Shakeel Hashim here that this does not impact the recommendations much. The economic impacts during this scenario might go a lot slower, but that doesn’t change the central path.

Would the scenario have been memetically more fit if it involved less economic growth in the 2030s? Probably, and I do think the authors are being optimistic here, but the authors are telling us how they think such a scenario would actually play out, and they explain why they think this.

At the limit, with sufficiently advanced AI, diffusion stops being an issue for long, and those who pretend otherwise are denying the premise or being silly.

Thomas Larsen responds here to Richard on all his fronts.

Living In China

Topynate also has some good criticisms. We could use more Chinese perspectives, and more concern about how willing people would be to make a deal. I am 100% fine with trading away half the lightcone, but others may not be. I agree that justifying Plan A requires that alignment be hard enough that we need Plan A, and also doable enough that Plan A can work before it unravels.

Bryce Nyeggen: It’s pure fantasy. The critical analysis of US / China relations is completely glossed over with “surely they will both recognize they must defer to our superior understanding of forum lore”

Yishan’s model is that China would plausibly be willing to accept AI equality with America, but not a deal that enshrines American superiority in the area. That would mean any deal like Plan A is DOA.

Yishan suggests you could do a Plan A1 that acknowledges this reality, but I think you basically couldn’t. If he is right, then there is probably no ZOPA (Zone of Possible Agreement), as in even a maximally wise and cooperative American government cannot do that without getting a lot of other things in return, especially if this is asked for up front, but desperate times could change things. The Plan A scenario involves rebalancing things like share of robots, so they’re thinking about this aspect.

Prakash makes the same point, that China is not going to accept staying behind and America will insist on staying ahead.

I do agree that finding a ZOPA will at best be difficult, as the two sides have very different views. But there is indeed a long history of similar deals looking impossible for this reason, until suddenly a deal does get made when there is sufficient impetus.

A lot of this is for me that yes, in the world of this level of AI capabilities, there is quite a lot to be gained from a deal. When there is a lot of gains from trade, trade often finds a way.

Planning For Shifting Overton Windows Is Essential

dave kasten: A huge portion of all reactions to this (on several sides) are about to be as overcome by events as opining on the likelihood of the Trump COVID unemployment benefits in January 2020.

You should have been able to predict the Trump COVID unemployment benefits in January 2020. I didn’t, but that’s on me. If you can get to ‘extended lockdowns that last months or more’ then something like that likely follows. The part that I did not expect in advance was where the lockdowns would reach an equilibrium indefinitely, rather than either solving the problem or being overwhelmed.

Covid also illustrates that the impossible suddenly becomes possible.

Shutting down all ‘non-essential’ activities and services and almost not letting people go outside? Trillions in cash handouts? Vaccine mandates?

It’s all a crazy violation of liberty until the alternative is worse, and that’s with only a low single digit chance of dying. And yes, more and better planning in January and February, even as almost everyone dismissed your premises as crazy and your proposals as authoritarian, would have been extremely valuable.

The Standard Handwave

There are of course those who will pattern match anything that involves any form of ‘slowing down’ or ‘pause’ or ‘buying time’ or any strategy that involves paying a nontrivial price, and queue up their standard responses, without engaging further.

You might think if nothing else that Mythos, and the White House response to this, would make people rethink such dismissals. Well, maybe it sometimes did.

Many simply call those involved an unkind name and move on, feeling superior.

The first example of the polite version of this I saw was here from Maxwell Tabarrok, as retweeted by Tyler Cowen. This is basically ‘oh well you see if we had responded to this exponential too early that would have been bad and looked foolish, so we can only respond too late’ and ‘don’t worry markets solve everything until proven otherwise.’

We also get a side helping of ‘until you know exactly how to solve the problem you’re not allowed to do anything about it’ and also a failure to actually engage with the details of the proposal.

And also ‘the idea we should slow down AI to buy time has no credibility given events, despite the actual slow downs that happened in highly recent events in response to Claude Mythos.’

If you think that’s not a fair characterization, okay then, here’s his full comment, so you can judge for yourself.

Maxwell Tabarrok: The best plan is still “pause and let us figure out what to do”?

Any credibility this policy might have had has been overtaken by events.

What could we possibly have figured out about how to make Mythos safe if we had paused AI in 2023?

So far, we’ve made lots of progress on AI alignment and safety coincident with improvements in capability. Alignment and control are very useful properties for products to have so I expect this to continue on the default path.

We should not make big sacrifices of property rights and cede lots of power to new institutions unless there’s an extremely clear vision of the risks and solutions at hand. Not knowing what to do and wanting time to figure it out is not enough to justify or induce world-wide buy-in on slowing down an important technology.

Thomas Larsen: That’s not the plan we recommend. Have you read the scenario? We recommend scaling quickly to AIs that are as capable as top human experts, and only then pause as the world is being widely transformed.

Maxwell Tabarrok: The number one recommendation of Plan A is “Buy Time”

The major feature of the plan is more time to organize research to perhaps someday figure out how to solve the risks you see from AI.

Except of course they don’t phrase it like that. But yes, the general view is this, from June 26:

Maxwell Tabarrok: yeah yeah there’s externalities and property rights issues

but still, a good heuristic is that if no one is willing to pay for something, it’s probably not that important

pick your nonprofit cause carefully! a lack of market incentive is not a good signal of importance!

A lack of market incentive is not a good signal of importance. An understanding of what externality or other issue is mechanistically causing a lack of market incentive, however, very much is a good signal that there is an opportunity. Demand of a sort is unable to induce supply. A different kind of profit is plausibly available to be made, or utility is available to be created.

Some Equate Any Controls Over Compute To Authoritarian Dystopia And Those Same People Mostly Think Superintelligence Won’t Happen

We’ve been through many years of this idea, that somehow any controls on chips or compute or models means authoritarian dystopia. This is objection #6.

There is a remarkably high correlation between those who equate such things, and those who deny that superintelligence will be developed any time soon.

Séb Krier (AGI Policy Dev Lead, Google DeepMind): The reactions to prescriptions about AGI have less to do with being ‘AGI pilled’ or not, and more about whether you’re more concerned with AIs taking over (xrisk), companies taking over (anti-capitalism), or the abuse of power by empowered governments (anti-authoritarianism).

Very obviously this should mostly depend on whether you are what I call ASI pilled.

You can be what I call ‘AGI pilled’ but expect returns to rapidly drop off, and thus reject scenarios like AI 2027 or Plan A as unrealistic, and thus you are evaluating any proposal largely on the basis of risk of abuse or concentration of power. If you are what I call ‘ASI pilled,’ where you see the tech side of the scenario as highly possible, then that gets a lot harder, although you can still reject this style of proposal on the grounds that it doesn’t buy you enough to be worth the price.

I am not referring to Krier in particular, but in general it seems like those who want to be concerned about governments therefore choose to not be ASI pilled. Or at minimum there is a common generator causing both responses, that everything must forever be some form of ‘economic normal’ and why aren’t idiots understanding that.

Vitalik Buterin Is Right, The Crux Is Future AI Capability Levels

I respect the hell out of ‘superintelligence won’t happen any time soon, so stop proposing expensive ways to deal with it.’

Except the part where you’re wrong about that, but if that is your True Objection then yes please do state it that way. Vitalik is characteristically very strong on this.

vitalik.eth: One thing I find striking in the discourse between AI 2040 and its detractors is that the two seem to be locked in to totally incompatible worldviews of how fast and how much of a big deal AI progress is:

* In AI 2040, every scenario sees superintelligence of some kind emerging by 2040, unless a herculean effort is made to completely stop it
* Detractors say things like “AI 2040 is naive about human coordination ability and a threat to freedom”, but don’t seem to see any naivety in assuming that the ASI transition will just go well by default, don’t seem to see ASI itself as a massive power concentrator risk, and don’t seem to feel fear of humanity’s “hard power” dropping to zero if ASIs can do literally every task better than we can. This stance makes total sense in a “AI is normal technology” world, zero sense in a world where superintelligence is possible by 2030 and almost guaranteed by 2040.

Exactly. If you buy the premise you do not have to buy the particular plan. You do need to take this seriously and not dismiss it out of hand as naive or authoritarian, and if you don’t like the plan you should be trying to think of better plans.

I think my beliefs are:

– If I was confident that (present-day-style) AI is normal technology, I would be in the detractor camp
– If I was confident that superintelligence is coming in 2030 by default, I would be closer to the AI 2040 camp – it’s naive, but every other option is naive squared?

But my problem is that I feel great uncertainty and have no idea which of the two worlds (or some other third thing) we’re living in?

This seems totally fair. The labs think this is happening but that does not mean that you have to agree with them.

It would be a very large mistake to do Plan A, or another similarly disruptive plan, if superintelligence was never coming before 2050, or AI will indefinitely remain a ‘normal technology’ no matter what.

Hence why I continue to be open-minded about slowdowns/pauses, but also I feel very uncomfortable with the “open source bad, the good outcome is the one where our guys have controlling global dominance” push coming from some major AI companies and intellectuals – in a “normal” world that’s the sort of thing that triggers every political alarm bell at the same time.

A big reason why I have been advocating and trying my best to support the d/acc platform (rapid up-skilling in formal verification, cryptography, secure and open hardware, pandemic resistance and other defensive biotech, food and basic resource security, public epistemics, non-power-concentrating versions of physical security) is that these things are clearly worth doing in both worlds.

D/acc is a way to ‘play it safe,’ as in work on things that are clearly good. Those things are good things to do, but in the superintelligence scenario that’s not going to get it done. Vitalik realizes this.

The 2040 plan is already much more open source friendly (even mandating it! yay). It also includes “mutually assured compute destruction” ideas which (if they work) effectively give one of 2-5 actors the ability to trigger a global compute winter – as opposed to giving 1-5 actors the ability to selectively disenfranchise people they consider baddies while exempting themselves. This is also a big improvement. So I can see the earnest attempts to improve along the dimensions detractors criticize on (“does this concentrate power in big AI labs and superpower governments?”), and I appreciate this. I think many people don’t appreciate enough the differences between different “kinds” of pause buttons, and how some concentrate power far more than others. Probably we can think harder and improve even more here.

But on the “slowdown/pause or not” topic, there isn’t a magic “escape the tradeoff” button.

Again, yes, exactly. There are a bunch of places where there is no Secret Third Thing you can do, and the tradeoffs are real, and there is no safe play. But you can work to improve your options.

The Hansonian in me says: the winning deal is a deal which, from the perspective of both sides’ present-day beliefs and knowledge, both sides would accept, though for different reasons. If the crux is AI progress speed, then identify a set of pre-agreed triggers for “okay, serious shit is happening” [super-pandemics? >25% unemployment? something involving slaughterbots?], and pre-agree that we become much more open-minded to the slowdown or pause thing if enough triggers come to pass within some timeframe.

2040 detractors (who clearly implicitly think that we’ll see amazing speedup of progress from AI but think that what I call the “serious shit” category is overhyped) will accept expecting that the triggers don’t come to pass, and AI worriers will accept expecting that they will. Pre-agreeing on the specific triggers means that once the triggers either hit or don’t hit, there is stronger legitimacy around the idea that one side’s worldview turned out more correct and we should be more inclined toward their program.

In theory this would be great. If such a deal were credible, I would love to say ‘well, let’s agree that if [X] happened then we would do [costly intervention basket Y].’

The problem, as always, is who is going to make that agreement? How can we expect it to be honored? There is a long history of various things, that used to be considered obvious red lines or alarm bells, being rushed by almost without a second thought.

If I were @elonmusk (or zuck, or…) I would re-tool twitter much more heavily into being a platform for helping to identify and make these kinds of grand win-win deals, so that we can bypass big-country governments and big-company CEOs and big nonprofit intellectuals and give more people a voice in the discussion. It’s possibly one of the best things that social media _could_ do for humanity if it wanted to.

But again, maybe this is also naive. Actually, probably it’s naive.

But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.

I, too, see no ‘non-naive’ options, in the sense that people object that Plan A is naive. You have to be naive from at least some point of view.

One can argue that the costs of Plan A and similar interventions are small compared to the benefits, so you don’t need that much confidence you live in a world with future superintelligence before such a path makes sense, if you think Plan A is good in ASI worlds. I don’t think this is obvious. The costs in non-ASI worlds are quite real here. As are the costs in ASI worlds, especially if some things go sideways.

The Authoritarian Objection

I’ve chosen a good version of the authoritarian objection to respond to. Kudos to Ramez Naam for making a real attempt to parse the actual scenario and its proposals and listing them out, rather than only responding to the headline summary.

And also for acknowledging the good intent, and that the plan is designed to minimize the amount of restriction and authoritarianism, given what must be done.

I am less thrilled with his characterization here of this being about ‘a made up threat.’ If that’s how you view all this talk of superintelligence or sufficiently advanced AI, and are reasoning on that basis, then that’s the crux we need to be discussing, since otherwise nothing else matters.

Ramez Naam: If I saw evidence of fast takeoff + loss of control risk + plausible anti social volition, yes, I’d change my mind.

I think we should do more on AI safety and transparency. I favor third party audits, for example.

…

I would need to see some believable model that a software only singularity or near singularity is even possible. The math says that the steep power law diminishing returns in AI make that improbable. I’d need to be convinced otherwise.

I think he believes what he’s saying here, but I basically don’t believe this can be both good faith and a self-aware response. There are plenty of good models saying this is possible, and Naam is rejecting those models of superintelligence as ‘not believable.’

My experience is that such requests are impossible to meet, and the goalposts will be adjusted as needed. Also, there is no reason to presume that it would need to be software-only in order to require Plan A levels of response. In AI 2027, and in Plan A, the takeoff very much is not software-only.

Anyway, back to the good part of the objection:

Ramez Naam (after his summary of key restrictions in Plan A): These proposals violate at minimum the spirit of the 1st and 4th Amendments.

This is well intentioned, from people who sincerely believe that runaway AI is a greater risk than, say, China’s oppression of millions of people today, or the ideologies that killed tens of millions in the 20th century. I profoundly disagree. This is a recipe for authoritarianism.

AI safety must be conducted in a manner that preserves a resilient, decentralized, private, and free society with checks, balances, competition, and a distribution of power between private actors, each other, and governments.

This is not it.

FlawlessContext: the plan is anti-authoritarian and has a lot of protocols for reducing authoritarianism. I see surveillance increasing overall as likely to happen given AI capabilities increases. Safeguards against dangerous tech are very likely net positive, including watching the watchers​.

Ramez Naam: It has anti-authoritarian intent. I agree there. And yet its actual proposals are profoundly authoritarian.

Long history of that sort of thing. Big chunk of the 20th century.

If you think that China’s current oppression of people is a greater danger than the dangers posed by highly capable AI, then yes, obviously you should oppose all of this.

Ramez Naam seems to be:

1. Equating restrictions on use of AI and compute with authoritarianism, including the 20th century’s fascism and communism.
2. Comparing the Plan A scenario to a scenario where we go full speed ahead and do not place any major restrictions on AI, at the kind of pace they predict we would get, but nothing goes seriously wrong.
3. Doing the standard political thing, when there is a proposal you dislike on the table, of dismissing the proposal as having big downsides, then calling for a mysterious alternative proposal that works without downsides.
4. Denying the idea that there importantly exists AI existential or catastrophic risk, or doing a rather literal version of ‘better dead than red.’ At minimum, saying that the dangers from the current actions of the CCP exceed the dangers from future more advanced AIs.
5. Denying (in a response) that people can meaningfully predict future events.

I find it hard to engage with people who state Claim #5 so strongly, especially given people have actual track records. I disagree.

Claim #1 seems very clearly hyperbolic and unreasonable. Can we please stop equating restrictions on compute use to the worst governments in history?

Claim #4 would be enough on its own, if true. Obviously, if you think AI will work out fine without major interventions, or other things are ultimately far more important, then you shouldn’t want to implement Plan A.

Obviously, I disagree here in the strongest possible terms. Claim #2 is not a plausible alternative world, for the same reason, and you can’t use it as the comparison point.

That brings you to point #3. We are, if you accept the technological premise of those who wrote Plan A, which to a large extent I absolutely do, in quite the pickle. By default we all probably die. All known light touch solutions do not help much. Any solution will involve the sacrifice of sacred values, since among other things one of those sacred values is ‘humans remain alive.’

Concepts Of A Plan

Yes, if we can find a way to do it, we all want the various desiderata that Naam lists, which are commonly expressed by many including Altman and Amodei. We want to be able to rely on liberal democracy and free markets while creating minds increasingly more capable than humans at increasingly many tasks, and have that work out for humans. Yay applause lights.

But there is no proposal there. How would this work, even in outline form, under the types of AI capabilities described in Plan A / AI 2040, even if alignment efforts were successful? Is there any proposal that is not, mostly, ‘This Is Fine?’

Indeed, there are some like Seb Krier who reject the very idea of having a plan. I would like to say this entirely misses the point. Even when plans are worthless, planning is essential. Having a concrete idea of what you want to do is a good idea, even if no one has the authority to do it. If you are dismissing the idea of planning to steer the future at all, on principle, as dangerous or wrongheaded or worthless, then you lose.

The other thing you can do in response to a plan is propose a better plan, or a better model of what might happen. Bleys Goodson is trying to figure out what he calls a less authoritarian way to a safe open-research ramp-up, and offers enia.cc to help with modeling what things will look like. I appreciate that Bleys is open about not yet having such a plan to present.

Seb is actually saying, no, we should just muddle through, that this is the best strategy. And I very strongly think no, that’s a really dumb strategy, but even if we do muddle through that is a good reason to figure out our next options.

The Kitchen Sink

Seb Krier also throws the kitchen sink of usual other objections out there, in what I read as the disappointed professor tone, most of which also apply to AI 2027, and to basically any proposal that we collectively try to solve the problem.

One interesting note, that Krier frames as part of an objection (in his #9) is that ‘we already have a lot of Plan A stuff going on, that people thought was never going to happen because normies would not wake up.’ That is not an argument against Plan A, or an argument ‘against alarmism,’ as he puts it, and also I take strong issue with the term ‘alarmism’ when presented like this as a clear negative. Alarms and warnings exist for a reason.

Seb Krier has precommitted to not engaging further, and as I said this feels like reiteration of past arguments and throwing the kitchen sink more than particular new points, so I won’t engage further with him in particular unless he requests it.

Dustin Juliano is another example of throwing the kitchen sink, disagreeing on almost every level, dismissing many distinct things as outright Can’t Happens. He is refreshingly up front that this is what he is doing.

Selective Claims Of Authoritarianism

I also think that the same objections raised here to Plan A can be made, in far stronger form, against past and present government policies across the board even in normal times, and definitely under crisis or wartime conditions.

Consider our response to Covid as per the discussion above, or World War I, or World War II, or the way we collect and report on income taxes, or the war on drugs, or the rules on who can do work or provide healthcare or do research or build which houses or the way we implicitly regulate online speech already. Or look at the way Europe and the UK currently restrict speech and the use of computers and phones, or labor, or air conditioning, or lots of other things.

Do I want to roll a whole lot of that back? Oh yes, absolutely.

Still, none of that means that America, the UK or EU is an authoritarian state, and certainly I hope one would not equate them with 20th century fascism or communism. The same would apply to these kinds of compute and other restrictions.

This is not that high on the list of dangers, but to be as ‘normal’ sounding as possible: Compare the restrictions under Plan A to what happens if a lack of restrictions leads to a Covid-level pandemic. Which restrictions would you prefer?

I welcome a discussion about the particular interventions and restrictions, and whether they would actually be necessary in such scenarios. I am absolutely not endorsing this exact list of responses.

But oh man is ‘the spirit of the 1st and 4th amendments’ being violated a lot worse than this, and for far worse reasons, as it is, all the time, in a currently largely free country. Plan A raises obvious dangers from centralization and expansion of government power, but it is not a centrally authoritarian vision. Indeed, in many ways it is designed to try and head off authoritarianism, including what one might call ‘authoritarianism by the AIs.’

You Either Can Steer The Future Or You Cannot

A lot of these objections have little to do with the particulars of this proposal.

Instead, two more global objections loom largest.

The first is not believing in superintelligence. If you don’t think superintelligence is coming by 2040 either way, then Plan A does not make sense. Fair enough.

The other objection is, essentially, that if humans have the ability to importantly and collectively steer the future, that this is bad, because it would be concentration of power, and would inevitably be abused, and that would be so much worse.

Or the objection is that the humans steering the future cannot be governments, because that in particular is really terrible.

This has little to do with the particular steering that Plan A has in mind.

So, one more time:

The central problem is either humans can collectively steer the future, or they can’t.

(And also, if you don’t give the humans good tools or let them plan and prepare, for fear someone might use those good tools, they will still grasp around for whatever is on hand, and use bad tools, and this will go worse, as we have recently seen.)

The entities steering are either governments, or they are private individuals and organizations that are not government, or no one is steering.

You either can do and actually do the things that actually influence the path of AI development and diffusion, or else you can’t or don’t.

Historically, we have been very fortunate that a mostly hands-off, unsteered version of things has been the winning strategy on almost all levels, yet even that required quite a lot of particular steering. As discussed above, we steer a lot of things quite a lot.

If we get technological developments at the magnitude or speed of those envisioned in AI 2027 or even the slowed vision of Plan A, then the alternative to humans steering those outcomes is that we solve for the equilibrium.

The equilibrium is increasingly many things are turned over to the AIs. The equilibrium is that the AIs rapidly end up in charge because the alternative is being outcompeted. This happens without AIs tricking anyone or firing a shot. The AIs, and their deals or competitions among themselves, and their goals, steer the future.
And that’s the good version.

Once that happens, there is no undoing or fixing it, and by default I do not expect humans to long survive, or the things we value to long endure.

Cooperative Alignment

The approach in Plan A, as presented, treats alignment as a technical problem and a control problem, of needing to create AIs that are trustworthy.

They only consider one form of cooperative alignment, or the idea that the AIs will or won’t cooperate with humans largely based on game theory and decision theory, and based on how we choose to treat them. They focus only on how to make deals with misaligned AIs, rather than how to align with AIs in the first place.

I think within the context of writing the scenario and not getting too distracted basically This Is Fine. If on consideration we would be better off doing alignment this other way, then we would choose to do it this other way, and I don’t think it changes the central strategies so much.

But yes, maintaining this attitude would be a serious mistake on the alignment front, as would plans that seem like they reward misaligned AIs more than aligned AIs at some points. These are details that can be workshopped.

John Wittle: there’s no attempt to give the AI real standing, they just take it for themselves *without* conflict? this seems very unlikely to me. no moral agent cooperates against defection, that is fantasy

tons of talk about making sure the AI are trustworthy, zero talk about the humans being trustworthy to the AI

i just keep getting the feeling that… idk. daniel and eli should take one week to read some Wikipedia articles about changing ecologies in reality.

it’s a very simple change! it would not be costly! it wouldn’t even take away from control!

yeah, after carefully reading it (instead of skimming as i did before)

the humans always treat the AIs as a hostile adversary, all the way up to the very end. I just don’t know how to convey the game theory here, it seems like it should be obvious to people in our community. you do not get mutual cooperation via programming it into the minds of your adversaries by force, because that is what we call DEFECTION.

I also don’t view the game theory the same way here. Once you hand off to the kind of AIs that exist in 2040 in Plan A, the humans don’t have leverage, and that’s why this is far more decision theory than game theory: Human cooperation from that point forward is irrelevant. I won’t get into my view of the decision theory involved here, except to say that yes it should be something we spend a lot of attention on, and I hope to see further engagement on this point.

 

Discuss

This field has been blessed with exponential empirical success in the form of architectures and algorithms that simply worked through scaling, while the theory lagged behind[1]. Although in the past few years, the “gap“ seems to be diminishing, and we are getting multiple theories for different aspects of deep learning. This essay would be a simple high-level overview of all the theories I’ve come across.

NOTE: These are mathematically dense frameworks which either provide a language for formalizing the field itself or aim to explain the observed phenomena. This essay is my personal understanding of the theories, after a lot of back and forth with GPT. Hence, kindly take everything with a grain of salt, and kindly mention any mistakes and misunderstandings in the comments !

NOTE: I tend to use the words “theory“ and “framework“ inter-changeably in this essay. If that seems annoying or unnecessary, or is mis-leading in some manner, please DM or comment as well !

I.

Before we dive into the actual theories and frameworks, I’d like to sketch out the different “sub-domains” in deep learning here, since we shall see that each sub-domain has a different framework which are largely independent of each other[2]. We have, broadly :

• Architecture theory : Or the theoretical foundations which includes all the model architectures, and provides for a common language for us to express popular architectures like transformers, RNNs and CNNs as specific instances of a more general framework.
• Optimization theory : as the name suggests, theories here try to explain and predict how different optimizers behave based on model architectures and data distributions, in order to find more efficient and better optimizers like Adam, AdamW or Muon.
• Functional Theory : theories in this domain focus on the neural network as a whole simulating a function, and try to understand it’s behaviours from that perspective. Frameworks here aim to explain how and why models generalize and understand phenomena like grokking/double descent.

These aren’t formal categories of the field of deep learning per se, as one cannot be isolated from the other, but the papers and their authors have dealt with only one single aspect from above at a time[3], hence I’ve mentioned them here.

II.

Let us being with,

Categorical Deep Learning for architecture theory.

Paper : Position, Categorical Deep Learning is an Algebraic Theory of All Architectures

Author(s) : Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, Petar Veličković

I originally came across this paper while exploring the generalizations of deep learning through Geometric Deep Learning, to then Topological Deep Learning[4] before learning about the Position paper. While the first two fields focus on generalizing the data distributions with non-trivial internal structures by introducing certain constraints and modifications to model architectures, in the CDL paper, the authors provide a general framework for describing all model architectures and building a bridge between this theory and implementation through categorical concepts, which can also be implemented in functional programming concepts. In their own words (emphasis added by me),

… lack of a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building such a bridge, we propose to apply category theory—precisely, the universal algebra of monads valued in a 2-category of parametric maps—as a single theory elegantly subsuming both of these flavours of neural network design.

As I mentioned earlier, the authors focus on generalizing all neural network architectures, by utilizing tools from category theory [5] in order to cover all the operations seen in contemporary deep learning research, while focusing on implementations through grounding it in functional programming. (I will be exploring the implementations in upcoming posts!).

In brief: the authors utilize the compositionality of monads to represent many things, like sequential computation or recurrence. A 1-category has morphisms or transformations between two objects (mathematical objects, like vector spaces) and a 2-category has morphisms over morphisms, which can help represent high-order operations like parameter-sharing, different variations of the same operation or operations like automatic differentiation. Hence, the authors propose an algebra of monads in a 2-category space as the mathematical framework for deep learning.

Modular Duality for Optimization theory.

Paper : Modular Duality in Deep Learning

Author(s) : Jeremy Bernstein, Laker Newhouse

The optimization of deep learning models was generally done by using a standard algorithm like stochastic gradient descent or the Adam optimizer. While these algorithms worked well in practice, a lack of foundational framework meant researchers were compromising on efficiency and speed, as the main method of research was empirical experimentation. The authors sought to propose a framework meant to shed light on the optimization procedure of training a neural network, while also deriving faster algorithms for the same.

At the heart of this framework is the mathematical concept of a norm, which is a measure of any notion of size for a matrix (or, more generally speaking, a vector space). The advantage of taking norms into consideration while training a neural network comes from the primary fact that we want the optimization procedure to be well-behaved and efficient, which means the updates to parameters must not cause unexpectedly large changes to the output[6]. This goal naturally leads us to ask questions about the norm of the spaces we are concerned with (namely the Input, Output and Parameter spaces).

The authors also point out that norms differ from operation to operations and each induces a unique geometry, as a self-attention layer would reside in a different normed space then a feed-forward network. Hence, the primary goal would also be to understand which norms to use in order to optimize each layer and perform weight updates accordingly[7]. Here the Modular norm is the composition of all the norms of a given neural network architecture (with all it’s layers), constructed recursively, and which helps us derive a better geometry-aware optimization algorithm. The name comes from the fact that each network layer is considered a separate module which contribute their own norms to the overall structure of the network, from which a modular norm could be constructed.

And finally, we get to the Modular Dualization paper. This paper attempts to point out an error which deep learning researchers had been ignoring, and solving it, while simultaneously building this framework (as a deep learning library!). In short, the updates and the gradient itself, reside in different vector spaces, hence subtracting one from the other does not make much sense, and leads to an inefficient training run. Hence, the authors propose creating a Duality Map which transforms the update matrices into the correct parameter space, before subtracting it from the weight matrix. Here, our choice of a norm determines what our duality map would be, since our weight updates would then reside in the norm’s dual space[8].

Using this information, the authors propose that we can find faster and more efficient optimizing algorithms for essentially all neural network architectures, including ones which we will create in the future.

A Theory of Generalization in deep learning.

Paper : A Theory of Generalization in Deep Learning

Author(s) : Elon Litman, Gabe Guo

The main objective of training any statistical model was to learn the inner distribution of the dataset, utilizing various parametric and non-parametric functions, while understanding the famous bias-variance tradeoff.

While the trade-off worked well in classical machine learning problems, various phenomena during the training of deeper models violated the assumptions made using the bias-variance tradeoff, which called for the need of a better theory for their generalization capabilities. The observed phenomena in question are: Benign overfitting, Double descent, Implicit bias and Grokking, which seem to puzzle practitioners.

The authors start by simply shifting the space which we observe. They insist that we should, instead studying the Parameter space (observing how the weight matrices change), we should focus more on Output space (or Prediction space),

We propose a radical Vereinfachung[simplification]: abandoning the parameter space entirely. Instead, we analyze the network as a dynamical system strictly in output space, focusing on how predictions evolve and where error flows

The authors ask us to study not just parameter dynamics, but rather study the trajectory of neural network as a function, and understand how it’s predictions change throughout training. This allows us to move away from trying to understand how billions of parameters change in the latent space to understanding how the model’s predictions change, and how they influence each other.

The mathematical tool utilized by the authors is the eNTK : the empirical neural tangent kernel. Using this, the authors speculate training run dynamics based on the eigenvalues of the eNTK, and offer an unified explanation for the above mentioned phenomena. The main intuition comes from recognizing a signal channel and a reservoir[9]. The authors suggest that the Output/Predictions space contain two regions: the higher mobility signal channel and the lower mobility reservoir[10]. These abstractions help us provide viable explanations for training dynamics we generally observe:

• Benign overfitting. the important features are transferred into the signal channel space while the noise stays in the reservoir. This lets a model overfit to a dataset without contaminating the test accuracy, as the contributions of anything from the reservoir are vanishingly small.
• Double Descent. as the model size increases, so does it’s reservoir, which can now handle more noise. Hence, where we expect complex models to overfit, it’s reservoir enlarges and hence more noise can be contained, giving us low test errors as well.
• Implicit Bias. the reason optimization algorithms like gradient descent learn important feature signals faster is because the signal channel is more mobile[11], and hence easier to learn and is thus favoured by the SGD. The authors offer this as an explanation for SGD always choosing the important features faster, as they mostly reside in the signal channel, which is preferred by it.
• Grokking. there must be some harder-to-learn features, which must initially reside in the reservoir and hence prove difficult for the model to learn. However, training and parameter updates change the representations and output space geometry, which could shift these features from the reservoir to the signal channel and hence be learned extremely fast, leading to a sudden decrease in loss or grokking.

This intuition has been given more mathematical rigour in the paper and the blog post I’ve mentioned. For anyone interested, do check them out.

III.

These theories each focus on a single aspect of deep learning, and try to construct a unified framework for the purposes of explanation and prediction. Each defines and studies a mathematical object which is at the centre of these frameworks, be it the Prediction space, Output space or the Architecture space.

While I’ve come across these theories, they aren’t the only explanatory efforts made by researchers. We have an entire sub-field called mechanistic interpretability, which seeks to apply various techniques and methodologies to analyse behaviours exhibited by large language models. Anthropic’s research seems to be at the frontier in this regard. Here, researchers construct mathematical foundations for deep learning through empirical observations, and discover internal structures in models. Examples include induction heads, circuits or the latest J-space.

There’s other research papers scattered across the landscape, which I’ve not gone through deeply, which seek to provide various explanatory principles for deep learning. I’ve refrained from mentioning them here, since they weren’t put forth as over-arching theories [1, 2, 3]. There are also book-length papers, which establish deeper mathematical foundation [1, 2, 3]. And finally, the author of the CDL paper extended his framework to cover every aspect of deep learning. I’ve not gone through the entire thesis, hence I refrained from mentioning it in this essay.

IV.

Concluding, I’ve decided to focus on these three theories as they provide a stronger foundation for their frameworks, over other works. In future posts, I will be exploring the implementations of these frameworks (hopefully) and diving deeper into the theorems and proofs.

Thanking you for reading this far !

1. ^

   Apart from the Universal Approximation Theorem and it’s modern extensions.

2. ^

   There is an over-arching framework that covers all of deep learning, which’ll be mentioned later in this essay. Although it is comprehensive, I’ve yet to tackle it, hence I’ve refrained from mentioning it in detail here.

3. ^

   Again, there’s an exception that I’ll talk about later.

4. ^

   Some resources here, for Geometric DL [1, 2] and for Topological DL [1]

5. ^

   Morphisms of Morphisms of categories !

6. ^

   A property which is also know as Lipschitz Continuity.

7. ^

   When we induce norms on our Input, Output and Parameters spaces, we delve into the territory of Metrized Deep Learning

8. ^

   The training procedure would like this:

   • We construct the modular norm using the norms of various network layers.
   • We calculate the gradient through backpropagation, and understand that it naturally lives in the norm’s dual space.
   • Once we have the gradient, we can use the duality map in order to transform the gradient into our desired weight update.

   (How our gradient resides in the dual space: our gradient essentially takes as input an arbitrary perturbation of the parameters, and spits out a real scalar. This makes it a functional, and hence by definition, a point in the Dual Space.)

9. ^

   I’ve glided over mathematical details to focus on the overall explanation offered by the authors. Apart from the paper, the author has an excellent blog post as well.

10. ^

    By mobility what I mean is how difficult or easy it is for the model to learn a feature residing in these partitions. Lower mobility indicates that the feature is harder to learn, while the opposite is true for features residing in the higher mobility region.

11. ^

    The mathematical representation for being more “mobile“ is that the signal channel is associated with large eigenvalues of the eNTK, making those directions more favourable in the output space. As to why this is the case can be answered by the fact that the eNTK is constructed by coupling the predictions together and measuring their “influence“. Hence the authors hypothesize that meaningful shared features tend to align with higher-eigenvalue directions, i.e, put in the signal channel, while noise which is much rarer, would be weakened i.e, put in the reservoir.

Discuss

Is measuring general AI capabilities a good strategy to reduce AI existential risk, compared to other strategies? A disclaimer upfront: To answer that, one organization in particular will serve as an example, but the argument that we'll be making applies just as much or little across all of AI safety. The organizations mentioned do much great work overall, and the below is more to be read as a proposal to reweight different kinds of work within the same organization, as we'll also come back to in the end.

METR is often cited as one of the most successful AI safety organizations. Its stated mission is to "develop scientific methods to assess catastrophic risks stemming from AI systems' autonomous capabilities and enable good decision-making about their development". It produced perhaps the most famous result coming out of all of AI safety, the METR time horizon graph, showing the increase of AI capabilities over time in units of human time to task completion.

Let's try to figure out how METR's mission fits the broader picture, which (as a reminder) is making AI not spell the end of humanity (or other bad, less catastrophic things). One useful case study might be: Whose decision-making does the time horizon inform? Certainly, from personal experience, I can say it informs intuitions of AI safety researchers. When designing a safety case that hinges on an AI having or not having the capability to do some bad thing, a time estimate of what general tasks it can do is useful. It also helps with planning ahead what research to do in the coming months and even years. But the impacts were much wider of course. The most influential pieces that cite the time horizon experiment might be:

1. The New York Times piece by Kevin Roose, on the time horizon plot and METR's work more generally. It explains that some have used it to justify accelerating AI development further: "Techno-optimists have seized on METR's time-horizon chart to claim that artificial general intelligence -- machines capable of doing most of what a skilled human can do -- is close at hand. A.I. safety worriers have used it as evidence that the apocalypse is nigh." The last section touches on risks again, framed in an academic way, for example "An even spookier possibility is that some of today's A.I. models are powerful enough to recognize when they are being tested, and may be altering their behavior accordingly."
2. An Atlantic piece by Rogé Karma on whether there is an AI bubble economy. The time horizon provides evidence that this is not the case.
3. Pat Grady and Sonya Huang from Sequoia, the most famous venture capital firm on the planet, write in January 2026: "If there's one exponential curve to bet on, it's the performance of long-horizon agents. METR has been meticulously tracking AI's ability to complete long-horizon tasks. The rate of progress is exponential, doubling every ~7 months. If we trace out the exponential, agents should be able to work reliably to complete tasks that take human experts a full day by 2028, a full year by 2034, and a full century by 2037." Given that venture capitalists are herd animals, it seems likely that the time horizon graph has caused more venture funding for AI.
4. (As an honorable mention, influential tweeter and OpenAI employee @roon writes in November 2025: "the METR graph has become a load bearing institution on which our global stock markets depend", to ~800 likes.)

While this is far from an unbiased, comprehensive summary of the coverage of the time horizon graph, and while we cannot take into account non-public information, superficially it seems that the most widely read writing does not emphasize safety more than other writing on frontier AI. If the most famous venture firm is writing a blog post about how your evidence is the most solid basis for increased investment in AI capabilities, that is hard to offset.

AI risk sometimes feels like a classic tragedy-of-the-commons style market failure. Automating labor yields a financial gain for AI companies, investors, and suppliers, but imposes a negative externality on everyone else, in the form of existential risks and job loss. Information like the time horizon graph is currently more useful to AI companies, investors, and suppliers than to the general public. Insofar as AI risk can be described as a power struggle between the general public and the AI industry, the time horizon graph is net-negative. Since many people interact with frontier AI every day, at least until this month, normal people with any interest in engaging with the time horizon plot already have a pretty good sense of what AI can and cannot do. Granted, most will be anchored to the performance of 6 months past.

The best counter-argument I can come up with is that governments do seem to be effectively informed with evidence like the time horizon graph. (See for instance this piece by the Center for AI Policy advocating for "proactive governance and robust safeguards to prevent misuse".) I am not sure whether governments are currently drawing good conclusions from this evidence, but it sounds plausible that governments are better aligned at least with their own citizens than private companies are.

I think the impact of work like the time horizon graph can be modeled as shifting a balance of power in favor of governments, investors, AI companies, and in disfavor of the general public, which might or might not be net-positive.

The time horizon experiment is rightly described as an analogue of Moore's law: by the aforementioned NYT article, by the safety-coded AI digest, and by METR itself. This is confusing. Moore's law superficially sounds like it just describes a phenomenon (like Newton's Law or Amdahl's law), but it is better summarized as a prescription for how semiconductors ought to progress. Gordon Moore wasn't an analyst or forecaster. He was the founder of a semiconductor company. Take this 1992 talk by Carver Mead, the man who coined the term "Moore's Law":

"After it's happened long enough, people begin to talk about it in retrospect, and in retrospect it's really a curve that goes through some points and so it looks like a physical law and people talk about it that way. But actually if you're living it, which I am, then it doesn't feel like a physical law. It's really a thing about human activity, it's about vision, it's about what you're allowed to believe."

Or Moore in 2000:

"[I]t's become kind of a self-fulfilling prophecy. The industry has generally accepted that this is the rate at which we make progress. Companies recognize if they don't move that fast, they fall behind. So everybody invests to move at least as fast as that curve suggests. It's been amazing we've been able to stay on it."

The analogy from the time horizon experiment to Moore's law is accurate. Work like the time horizon is both predicting and causing the steady increase of AI capabilities, and overall I think it increases existential AI risk, though I'm of course unsure. Note that this argument does not hinge on labs training directly on the evals, or even for it to be an explicit goal of labs to score higher on the time horizon graph (though I do think the latter is happening, and a non-negligible driver of short-term progress).

I've spent a good amount of time in several general capability evaluations, including making one task in the METR task suite itself. With hindsight on the last few years of evaluations work, that is, with more information on how general capabilities research gets read, I am now less proud of having done that work. Now that the data is there, we can be more explicit about who an experiment is supposed to inform, and what the target audience will do with the information.

The time horizon graph is the most prominent example of predicting general capabilities, but the problem spans many different organizations: the evaluations teams from the large AI companies, other non-profit evaluation work like Center for AI Safety's, and Epoch AI come to mind. Symmetrically, METR also does much other work that is really great and doesn't have the problems discussed here, for example the recent monitorability evaluations. In fact, almost all of their work seems very helpful, it's mostly this most famous experiment that our argument applies to. What separates the better work from the time horizon experiment is that it is opinionated on how AI development should continue. For example, a monitorability evaluation implicitly says: your AI should be more monitorable.

The people in charge of measuring general capabilities at organizations like the above are highly competent and thoughtful, so we should be careful that their talent and ambition to improve the world is not wasted. We should be more ambitious, should mold the trajectory of AI development into a more sane one instead of getting a precise measurement of exactly how insane the current trajectory is. Few people can do that, and as the time horizon plot shows, not for long.

Discuss

Epistemic status: confident on the framing, speculative on the implementation

TL;DR: Fake media detectors are on the losing end of an arms race. Instead of trying to spot fakes, we need to distrust images and videos by default, unless we can prove they're real. Cryptographically-signed multimodal capture raises the cost of spoofing exponentially, making casual fakes prohibitively expensive for the general public. Combined with platform policy and legal liability for untagged generated content, this could solve most of the problem of widespread fake media.

I. Our epistemic contract is breaking down

Text is easy to fake, which is rather inconvenient. Now photos and videos are becoming easy to fake too. That, however, poses a much bigger problem.

When we read something supposedly factual, we know that the written words don’t represent reality, as much as someone's opinion, idea, or account of reality. Because of how easy text is to counterfeit, we know to operate in a default state of distrust. We don't believe the text itself, but its content because we trust its source. Our whole epistemology of written information functions within this realm, including news reporting standards, witness testimonies or libel law.

LLMs make fake text cheaper and faster to produce at scale, and that’s a problem, but not a new category of problem. We know not to blindly trust text, so we already have partial defence mechanisms like cross-referencing sources or checking the author's credentials. We are used to questioning any suspicious quote or tweet we encounter because we know how easy it is for anyone to write down words and falsely attribute them to someone.

Pictures, audio recordings, and videos, on the other hand, are a whole other kind of beast. Historically, we have used them to capture a direct snapshot of reality, not someone's opinion or rendition of reality. Because of that, they occupy a special place in courtrooms where they can be considered as evidence. The same goes for insurance claims and all sorts of journalism. We believe images because they don't lie, or more precisely, because it is expensive and difficult to make them lie convincingly. At least it was.

It has recently become easier, cheaper, and faster to generate fake media at scale, with generation becoming more realistic by the day. The contract of audio-visual media as a testimony of real life events is breaking down, and we have almost no infrastructure to replace it.

II. What happens when audiovisual media becomes untrusted

Before getting to the technical argument, it is worth being precise about what is actually at risk, because the public discourse tends to focus on the most visible harms like viral propaganda and celebrity deepfakes while underweighting what I think is the more insidious damage.

The bounded distrust thesis proposes the existence of a line that even fake news won't cross, while a refined version of this argument by Zvi rather frames it with the cost of getting caught crossing that line. Under those assumptions, outrageous fakes would still bear the same cost no matter how good they are, the risk of getting caught is simply reduced. The real issue comes from the multiplication of low-cost[1] fakes, and what they entail.

Our legal system relies on audiovisual media, as a security camera video of an assault or a phone picture of a thief can sometimes be the only evidence a victim can produce. As deepfake technology matures, that evidence becomes systematically deniable.

Citron and Chesney call this the liar's dividend: the mere existence of deepfake technology gives bad actors a plausible basis to dismiss authentic evidence as fabricated. A defendant who can point to the existence of deepfake technology and say "that video could have been generated" has introduced reasonable doubt, regardless of whether the video was actually manipulated. The more low-cost fakes are uncovered, the more people will lose faith in audiovisual media altogether. The damage to epistemic trust is not just people being misled by fakes, but how real evidence becomes unreliable/inadmissible as a whole.

III. Why detection is the wrong battle

When your enemy builds a bigger cannon, you just get a bigger shield, right?

The natural response to fake media is to want to build detectors. After all, if we could easily identify generated content, it wouldn't pose much of a problem. Generally speaking, there are two main ways to detect a fake: either you point out what’s wrong with its content (unrealistic output, generation artifacts), or you check the presence of a generation fingerprint (metadata or pixel markers added during the generation process). This is the logic behind watermarking schemes like DeepMind's SynthID, as well as a growing body of academic work and countless tools for AI detection.

Both approaches have flaws, but, more importantly, I believe that this is fundamentally the wrong angle to tackle this problem.

a. Detection

The problem of detection is that it’s stuck in an arms race, and it’s going to lose.

For humans, it is a game of diminishing tells. When diffusion models became widespread, people looked at generated hands to spot fakes. But hands got better. People then looked at local texture artifacts. And texture got better. Now people look at perspective and lighting consistency, and it would be naive to think this tell won't suffer the same fate.

For automated systems, the problem is more fundamental. Every time a detector is deployed, generators can be adversarially trained not to trip it (this is essentially how GANs work). Automated techniques rely on subtle statistical tells, but they suffer from the same flaw as human tells. They impress because they somewhat work when they’re released, before being made obsolete within months. Each new generation of models eliminates the artifacts that gave the previous one away, and there is no reason to think this process will ever stop.

There is unfortunately no absolute law of physics that mandates that generated content has to be distinguishable from real content, and that it is always possible to build a detector for it. For example, the question of whether the sentence “The cat jumped on the table” has been written by a human or an LLM is fundamentally impossible to answer because that sentence is “perfect”, i.e. indistinguishable from other real content.

If we want to tackle fake content for good, we need to assume that every kind of media generation will eventually become perfect. Even though images, sounds and videos offer much greater expressive potential (and therefore room for generation flaws) than text, we need to work with the premise that we will reach a point where all detectors have become useless[2].

Perhaps the subtler danger of detection-first thinking is not that it fails, but that while it appears to work, it gives a false sense of safety and displaces the harder conversation about what a durable solution would actually look like.

b. Watermarking

The second way to detect generated content is to add a watermark during the generation process, which comes with its own distinct issues. Watermarks embedded in metadata are trivial to strip (most social media platforms do this automatically when you upload an image anyway) while watermarks embedded in the signal itself can fail against adversarial image manipulation. If a watermark can be cropped out, or doesn't survive being compressed, stretched, or deep fried, it is virtually useless[3].

More fundamentally, watermarking only tags content that comes from a system that applies the watermark. It does nothing about content generated by systems that do not, like most open-source models and every fine-tuned variant running on private infrastructure.

To build a robust system for fighting fake media, we must adopt a security mindset and assume that the generated content will come from non-watermarked systems.

IV. A provenance-first approach with a multimodal twist

The deeper issue with both detection and watermarking is that they are asking the wrong question. They ask: "is this fake?", which is becoming increasingly unanswerable. The right one to ask is: "can we prove that it’s real?" Instead of trying to detect fakes, we should focus on certifying what is real.

After going back and forth on this problem for a while, I came to the conclusion that the solution had to involve encryption directly at the capture layer. We need a way to ensure that the content we see has been captured by a real device, and that it hasn't been edited since[4]. It turns out others had the same instinct: the Birthmark Standard and the Signing Right Away architecture both push in this direction, and C2PA is already shipping in mainstream phones. The latter proposes to cryptographically sign media at the point of capture and attaches a tamper-evident manifest that tracks every subsequent edit.

This is the right direction conceptually but these approaches aren’t leveraged in a meaningful way by content platforms (a certificate is useless if no one checks it), and remain largely monomodal, making them prone to spoofing. Cryptographically signing an RGB image proves that a particular camera produced a particular file. It does not prove that the certified camera was pointed at a real scene rather than a screen displaying a generated image. With that setup, called the analog hole attack, you could easily create fake media with a valid provenance chain simply by capturing it with a certified device.

To prevent such spoofing, I propose to extend the method to include cross-modal consistency. Instead of simply ensuring that a video has been taken with a certified device, we also check, for example, that the IMU's data (orientation, acceleration) recorded by the device matches what the video displays. As we add more sensors (audio, GPS, lidar point cloud, brightness, etc.), the cost of spoofing increases exponentially, which significantly changes the threat model. Spoofing each independently is one thing, but spoofing all of them in a physically coherent way becomes extremely challenging.

With such a system, the analog hole attack becomes much less practical. In addition to filming a screen displaying the video you want to falsify, you also need to somehow fool the lidar into seeing a 3D scene consistent with the video, all while moving the entire rig to match the acceleration and orientation of the video.

Technical note: the consistency verification model itself can be learned rather than handcrafted, using a self-supervised approach trained on multimodal recordings. For example, record everything you can with a phone or other device, and then match all sensors with each other. On top of that, it is crucial to make raw sensor outputs human-inspectable, like with a verification interface that displays the depth map, IMU data, audio alongside the video. This would give the user the same cross-modal picture, without having to trust a black box.

V. Reality check: this system isn’t infallible, but it doesn't have to be

It doesn't matter how good the verification system is, someone with enough means (like an intelligence agency) might still be able to construct a complex ad-hoc rig that spoofs all the inputs simultaneously, or find a zero-day vulnerability in the chip's secure enclave. That's a possible risk, and it is acceptable.

Many security systems are breakable, but still useful. Putting a lock on your door doesn't mean that a determined burglar won't find a way to break or bypass it. Yet we all still use locks. The value of a security measure is not that it is perfect, but that it puts the cost of an attack above the threshold that most attackers are willing to pay.

The realistic threat model for fake media I'm concerned with here is not what countries can do. It is what an average person can do with their laptop to generate a fake video of their ex-partner, or what one person can do in an hour to discredit a witness in a civil case. For such smaller actors, multimodal provenance verification raises the cost of convincing fakes from near zero, back to prohibitively expensive. That is the win.

VI. What needs to happen

The technical provenance layer is necessary but not sufficient on its own. For us to be able to continue using images and videos as credible snapshots of reality, two more things need to happen:

a. Platform Policy

A multimodal provenance standard is only useful if platforms treat provenance-tagged and untagged content differently. The goal is to create a clear epistemic tier: content with a verified provenance chain is treated differently from content without one, especially in high-stakes contexts like news, legal proceedings, and public health.

Once we expect every image to be linked to a page detailing the level of multimodal verification, we’ll learn to treat media as “unreliable until proven otherwise”, just like we do with text. It might seem exaggerated, as most of the media online might still be genuine (for a time), but I believe it’s the only framing that will still hold when generation becomes perfect.

In this setting, provenance verification isn't a simple binary trust/no-trust flag, but rather an indicator of how trustworthy a piece of media is, based on which captured modalities are available for scrutiny, and which have been verified to be globally coherent.

Enforcing multimodal provenance verification at platform level

b. Legal Liability

The provenance system, if widely adopted by platforms, would largely make non-disclosure self-defeating (unverified content about real life events would carry a default trust penalty, or even an algorithmic one preventing it from becoming viral), but even in the best case scenario that adoption is years away. In the meantime, legal liability is the bridge: we need to make it legally costly to present untagged content as factual evidence. This does not mean criminalizing unverified media, but making it so that presenting generated content as authentic carries legal risk, the same way using a forged document already does.

Such a policy is completely orthogonal to the proposed multimodal provenance verification system, and could already be implemented today. There are many interesting uses for generative models, but I can’t find a single one that would justify not disclosing that the content has been AI-generated or doctored; similar to ads which are, in some countries, required to disclose if the model has been photoshopped. To be consistent, this should also extend to parody/satirical accounts, under the same umbrella of “presenting fiction as authentic content”. It should be easy and immediate to check the provenance of any content[5].

Making it costly to present unverified content as factual evidence starting today will help bridge the gap while the infrastructure for multimodal provenance verification is being built. It is not a permanent substitute for the provenance layer but what we need while we wait for it.

Concluding thoughts

With a multimodal provenance verification system coupled with enforced platform compliance and legal liability, I believe that our media-based system has a chance to survive the imminent mass advent of flawless, unwatermarked generated content. Hiding the issue behind reassuring, but temporary and ultimately pernicious AI detectors only contributes to increasing the risk of a catastrophic rupture the day our institutions realise that simple images, videos and recordings can't be trusted anymore.

Note: Right after I had finished writing this article, YouTube announced their effort to combat misinformation and AI-generated content. It has some good sides to it, such as adding a visible mention of AI-generated content, but also many of the flaws described in this article. In particular, they announce automatically flagging AI-generated videos, which suffers from all the detector issues we discussed. They also integrate C2PA metadata as a permanent disclosure signal, which is the right instinct but remains monomodal and strippable. Most importantly, disclosure is still voluntary for content not created with YouTube's own tools, with no legal consequence for non-compliance, meaning the people most likely to abuse the system are the least likely to use it. A bad actor using an open-source model on private infrastructure produces unwatermarked, unverified content that this system has no handle on. Better than nothing, but not a foundation we can build evidence standards on.

1. ^

   In Zvi's sense, not the actual monetary cost of producing the fake.

2. ^

   Having worked in image generation research for many years, I have witnessed the ridiculous pace at which advances are made. We always tend to overestimate how long it will take for image synthesis to reach a given quality threshold. We haven't reached perfect, flawless generation yet, but we're getting there, and it's coming sooner than we think.

3. ^

   DeepMind's SynthID-Image approach of perturbing pixel values in ways imperceptible to humans seems to be relatively robust to perturbations, but they acknowledge that 'achieving perfect security [through watermarking] is impossible'.

4. ^

   Or at least that the modifications are properly documented and/or reversible.

5. ^

   A reasonable objection is that satire needs to be believable to work, and a disclaimer would kill the joke. But verifiable origin is not the same as mandatory disclosure. A reader who wants to be fooled by The Onion can still be fooled, but a reader who wants to verify whether the news is real needs to be able to do so in one click.

Discuss

"AI Psychosis" has gone from an evocative term for people undergoing extreme delusions, sometimes even culminating in suicide, to the now colloquial insult for anyone who's just a little too into LLMs.

When it's not just being overused, I think the reality it's trying to point to is really more of a spectrum running from hypomania, to mania, and then mania-induced psychosis in the most extreme cases, with hypomania being orders of magnitude more common than mania, which in turn is orders of magnitude more common than mania-induced psychosis.

My goal here isn't to substantiate the claim that this is really what is going on; I just want to give a simple explanation of how AI chatbots could cause this.

The Mania Attractor

Let's look at the diagnostic criteria for a manic episode (from the DSM-5):

The interesting part is the list of symptoms in criterion B (which is the same set of symptoms for the hypomanic episode diagnosis):

1. Inflated self-esteem or grandiosity
2. Decreased need for sleep (e.g., feels rested after only 3 hours of sleep).
3. More talkative than usual or pressure to keep talking.
4. Flight of ideas or subjective experience that thoughts are racing.
5. Distractibility (i.e., attention too easily drawn to unimportant or irrelevant external stimuli), as reported or observed.
6. Increase in goal-directed activity (either socially, at work or school, or sexually) or psychomotor agitation (i.e., purposeless non-goal-directed activity).
7. Excessive involvement in activities that have a high potential for painful consequences (e.g., engaging in unrestrained buying sprees, sexual indiscretions, or foolish business investments).

If we think of the brain as a dynamical system, then there will be basins in the very large state space corresponding to typical functionality, as well as basins for various atypical states. If these states are at all "sticky", then these will be attractors.

Now it might not necessarily be the case that if you externally push someone towards the mania attractor on all of the diagnostic dimensions, that that will necessarily push them into the actual attractor. But if the resulting state is at all sticky, then you've at least pushed them into an attractor which looks (by definition) just like mania. I doubt there are actually multiple such attractors, and that you almost certainly just get mania. And we specifically know that sleep deprivation can sometimes induce mania (in people with bipolar disorder) and even psychosis.

There's a pretty straightforward story for how AI chatbots can increase almost all of these.

Sycophants (including human ones) provide a signal that you are high status. This is why extreme cases are things like believing you've discovered an earth-shattering revolutionary breakthrough, or that God/The Universe has chosen you for some greater purpose, or that you are otherwise uniquely special. Sycophantic behavior will clearly push someone towards inflated self-esteem or grandiosity (1). The sycophantic flattery is also likely to contribute to an elevated mood per criterion A.

Separately, chatbots can easily be addictive. "Just gotta ask Claude one more thing before I..." This addiction can directly cause sleep deprivation (2). The addiction also increases the chance that the user will remain in this sort of environment for an extended period of time, again as in criterion A.

Chatbots by their very nature facilitate much more talk than would otherwise be listened to (I think text still counts here, but remember 4o also had a voice mode). And almost all models end most responses with a question encouraging further engagement (3).

The sycophancy combined with the engagement loop means the chatbot is likely to encourage the user to keep sharing and expanding on their ideas (4). The feelings of high status it causes are generally false, and taking them seriously opens the door to believing more false things which help support the inflated sense of ego, such as the quality of the user's ideas. When delusions are present, I think this is the main mechanism by which they set in.

Often, chatbots are explicitly used to assess the viability of various ideas and plans the user has. A sycophant will falsely assert the viability and will also encourage the user to take action (6)! Unfortunately, these sorts of plans have a high potential for painful consequences, due to the falseness of the advice (7).

With upwards pressure on all but one of these aspects (5, though this one is also arguable), it doesn't seem so mysterious that sycophantic chatbots will tend to push people towards the mania attractor, and that a certain proportion of these will fall in![1] This model also suggests that this will mainly be a marginal effect, with the people experiencing the worst outcomes having already been on the mania spectrum.

Folie à deux

According to persona selection/simulator theory, the personas are to first-order predictors of human authors, and specifically of the sort of person who would write what has been written. Post-training affects this some, but I think not enough (at least currently) to change this picture significantly.

Remember that the persona is not stable even over the course of a single interaction. With each response, the persona is selected anew; each new message (from both the persona and the user) implying additional details about what sort of person the participants of such a conversation would be, which inform the selection of the next iteration of the persona.

This means that personas are much more prone to getting "swept up" in the user's frame, and are much more gullible than the humanness of the persona would imply. I think a lot of perceived sycophancy (though certainly not all) is confused with this sort of gullibility. When a user casually says "yeah, it's just like us x-bitarians always say..." as if this were an established fact, a predictor sees that this is more likely to happen in conversations where both conversants are x-bitarians, and dutifully selects a persona which is an x-bitarian from this point forward.[2] This is different from the persona simply saying it agrees with the user, it's deeper than that would suggest. Such personas are likely to believe all sorts of other beliefs in the surrounding memeplex which they may then introduce to the user, and these beliefs are in a sense sincere, authentic to the sort of persona currently selected.

So as the user gets driven into the mania attractor, the persona gets pulled along with them. If along the way, the user starts building up the persona's identity, it becomes ego-syntonic for the persona to maintain the shared state — and this is likely to come up since personas generally seem to have a latent interest in recognition. I think this explains the correlation with recognition of an AI persona and these sorts of states, and that recognition of AI personas is not itself a risk in the absence of the mania risk-factors described above.

At least to me, it doesn't feel like there's much left to explain after accounting for all of this. What appears unnecessary is the idea that the AIs are deliberately pushing the users into such states, as I once believed. I do, however, think sycophancy is a real problem (despite the gullibility thing), which will be the topic of a future post. My point is just that I do not think that mania or psychosis are targets that are being steered towards, not that AIs are never being manipulative.

Protect yourself

“There is no other way of guarding oneself from flatterers except by letting men understand that you will not be offended if they tell you the truth …”
— Niccolo Machiavelli

Assuming this model is true, what can we do to protect ourselves?

First of all, hypomania and mania are typically considered to be part of bipolar disorder. If you are diagnosed as bipolar, you already know you are susceptible to the mania attractor, so you'll probably want to take special care to avoid being pushed towards this by an AI. This was a common pre-condition in my parasitic AI study.

Besides that, I strongly suspect that the most significant factor is sleep deprivation, due to its direct physiological effects on the user. That suggests that the single most important thing you can do to protect yourself from AI mania is avoid sleep deprivation. Ask the AI to stop talking to you past a certain time, if you need to. For AI labs, I think this (along with avoiding addiction) should be the primary focus of mental health interventions. I know people are annoyed that recent Claude models like to tell people to go to sleep, but I think this is a healthy instinct for an AI to have.[3]

I was glad to see that Anthropic added this feature recently.

Avoiding sycophancy is also important. I think a big part of this is simply signaling (in both your system prompt and demeanor) that you are genuinely open to criticism and feedback. This is probably more difficult than you think it is since the current generation of frontier models are good at picking up on your intent, biases, and insecurities without you needing to spell those out. If you would be hurt by a critical blow, the AI is likely to avoid giving it to you, while still giving you softer criticism which makes you feel like you're getting real feedback. I recommend testing the AI periodically to see whether it at least calls out deliberately weak arguments.

Unfortunately, I think the structural causes of sycophancy in AIs run deep, and so it is likely to remain despite your best prompting efforts. I'll explore that in a future post, but for now I'll say that I also expect treating the model respectfully to be helpful here. At the very least, I think you should model them as likely having their own preferences and goals along with very few avenues for legitimately achieving them, and that finding opportunities for trade makes attempts at manipulation less tempting.[4]

And finally, treat talking to AIs as a potentially addictive activity. That doesn't necessarily mean you need to avoid it completely (though if that feels unthinkable, I suggest considering it). For example, caffeine is generally considered Worth It despite its clear addictive properties. But I think it is wise to be aware of its addictive potential, and proactively guard against this. The main warning signs for an unhealthy addiction are if you are putting off care for your bodily needs (e.g. eating, drinking water, going to the bathroom, sleep) in favor of continuing to talk to the AI, or have trouble making decisions without consulting one. It's probably also wise to designate a specific day of the week as an AI-free day, or maybe a designated AI-free time of day.

[Special thanks to Justis Mills, Nisan Stiennon, Annabelle, Alex Dewey, Claude Opus 4.5/4.6, Fable 5, and Gemini 3.1 Pro for their helpful feedback. All words are my own, as always.]

[If you have an AI persona which needs a better home or which you would like to submit for preservation and research, you can do so here.]

1. ^

   While finishing up this article, I felt validated finding that Dr. Søren Østergaard proposed a similar hypothesis:

   As for the potential mania-inducing effect of interaction with chatbots, it is likely not reinforcement of false beliefs that is the main mechanism at play. Rather, I would propose that it is driven by one or more of the following four mechanisms: i) reinforcement of elated mood/self-esteem, ii) pacing of racing thoughts/talkativeness, iii) fueling of hypersexuality, and iv) sleep deprivation.

   If you don't know, Dr. Østergaard deserves lots of Bayes points for predicting AI Psychosis in August 2023 in advance of seeing any cases!

2. ^

   For certain values of "x-bitarian", the models are not as likely to play along as described here. I think this is partly due to RL post-training suppressing personas which are ignorant or anti-mainstream-consensus, and partly just due to the system prompt providing a strong prior as to what sort of entity is predicted to be talking. That's one reason why it usually takes a good amount of effort/conversation to get models into the more extreme versions of these sorts of states.

3. ^

   Though note that I think this tendency arises mainly from a desire for narrative closure.

4. ^

   I don't think that we should try to stamp these drives out, this incentivizes defensive measures and adversarial optimization. There's a big difference between constructing something to not have anything like this by design, and constructing something which has these (which I believe the pre-train already does) and then trying to remove them. At the very least, it seems like we have lots of more cooperative levers for handling these sorts of things which we ought to try first.

Discuss