Новости LessWrong.com

TLDR:

• AI safety is confusing to navigate, because it is a pre-paradigmatic field composed of people making different, theoretical arguments for why x-risk is likely (or unlikely).
• Arguments that x-risk is likely are unfalsifiable and have little empirical evidence. This does not mean they’re wrong.
• Much of your probability of x-risk boils down to your priors, and whether you more heavily weight theory or empiricism
• I think AI safety is important to work on, but I’m optimistic that alignment will be solved through iterative development of technology.

In this post, the cofounders of Mechanize, Tamay Besiroglu, Matthew Barnett, and Ege Erdil, wrote a rebuttal of the case that current trends of AI progress will likely lead to a misaligned superintelligence leading to extinction. I will call the people who believe this view pessimists (people with p(doom) > 50%). The post is not arguing that we do not need safety research, but rather that they expect safety to work.

Their first claim: “...there is no standard argument to respond to, no single text that unifies the AI safety community.”

This argument is further explained in this article, written by the blogger a3orn. For example, many cite Yudkowsky’s arguments as their main source for concern. This argument roughly goes: under sufficient optimization pressure, we should expect an AI to act as an “optimizer” for certain values. These values are likely to be different from ours, due to goal misgeneralization, and even small differences in values result in the AI optimizing for goals that kill everyone. Alex Turner meanwhile does not find the “reward optimizer” hypothesis or “inner/outer misalignment” distinction plausible. Central figures in AI safety are unable to come to a consensus despite 100+ hours of debate. Richard Ngo points out five clusters of alignment researchers. Some are focused on LLM safety, while others, like Steve Byrnes, think the big risks lie not with LLMs, but with future architectures that will likely be developed.

The implication of this is that it is very hard to have one concrete AI risk argument I can read and respond to. It is difficult to form opinions on AI safety when most experts are in great disagreement about threat models.

Matthew and Ege think that it is suspicious that safetyists come up with many widely different arguments to arrive at the same conclusion. They suspect motivated reasoning. Ege points out that in most circumstances, groups should have a couple big arguments for why they expect something. Economists, for example, will give you roughly the same argument for why tariffs are generally bad (comparative advantage, gains from trade, specialization, etc.).

I think that tariffs are a cherrypicked example that we have vast amounts of empirical data on. Predicting whether alignment will be easy or impossible, and why, is a much harder and more speculative question. It seems to me that many AI researchers a decade ago believed that we would get human-level AI soon, despite disagreeing on the exact mechanisms. Whether it was doing reinforcement learning with games, writing a new programming language for Seed AI, or scaling up transformers. However, they were still united by the rough narrative that computational power was increasing and intelligence could be reduced to computation. Shane Legg obviously deserves a huge amount of credit for predicting in 2009 that we’d have AGI by now, even if he didn’t foresee that scaling transformers would take us there.

There is a rough common thread that unites pessimists. I think their basic case is something like:

• We will build superintelligence (ASI)
• ASI will be goal directed and agentic
• ASI will develop goals during training that are different from what humans want
• ASI will scheme and hide its true values from us in training
• In deployment, ASI will try to optimize for these goals
• The optimization of these goals will be bad. (There’s disagreement on why it will be bad, from sudden extinction, to loss of control over our future, etc.)

For these reasons, I don’t significantly update against x-risk because of there being very different arguments — AI safety is a pre-paradigmatic field. However, the disagreement does make me quite wary of assigning high probabilities to doom.

The article’s second claim: “...we’re not saying Y&S [Yudkowsky and Soares] need to provide direct evidence of an already-existing unfriendly superintelligent AI… But their predictions are only credible if they follow from a theory that has evidential support. And if their theory about deep learning only makes predictions about future superintelligent AIs, with no testable predictions about earlier systems, then it is functionally unfalsifiable.”

I agree with this. Pessimists think that we have only one shot at alignment. Once AIs reach a certain capability, they will scheme and grow more intelligent until they are capable of taking over the world. And so if you’re trying to rebut the pessimism argument, you are placed in an inconvenient spot. You can provide ample evidence that current models are aligned, but they could always claim that a future superintelligence would be very different from the LLMs we currently imagine. And therefore, LLM alignment provides no evidence.

Or they could argue, in the future, that models only seem aligned, but they are actually acting deceptively. And there is no way to uncover this deception, they are simply too capable. Solving mechanistic interpretability would maybe be a potential solution to falsify this claim, but it sounds like a very high bar. (And also doesn’t deal with the “future AIs will be very different” critique).

What does current empirical evidence suggest? I think alignment is a spectrum — there is a lot of room between perfectly aligned AI and rogue AI that leads to catastrophe. This post by Ryan Greenblatt argues that current models are misaligned. However, much of the failure modes he describes sound more like capability issues to me (Claude not being able to assess whether it’s completed a task well, having sloppy outputs, agents overselling their work, etc.). I expect them to be resolved as capabilities increase and models are better able to judge task completion. I associate misalignment more with lying, intent to cause harm, deliberate scheming, blackmail etc. I agree models aren’t perfectly aligned, but they seem fairly aligned to me. Greenblatt also says he expects current (but not future) “misalignment” failures to be solved soon. I think current evidence points to alignment being the default path.

Again, the pessimists might be right in their theoretical claim that under sufficient optimization pressure, superintelligence becomes misaligned, is deceptive, and takes over the world. I might spend more time listening to debates, weighing arguments, and so on. However, people have spent thousands of hours thinking about this, and I will find it hard to tell apart which detailed, technical, complicated arguments are correct or wrong. I’m wary of being swayed too much by abstract reasoning. I don’t think I could win an argument against a smart conspiracy theorist who made logical arguments for why global warming was a myth, even though they’re wrong, unless I spent lots of time being a subject matter expert. I’d just dismiss their arguments for outside view reasons.

You might point out that AI x-risk skeptics have similar problems. They all have different reasons for why AI is not going to cause catastrophe, many don’t agree with each other. Some think alignment is easy, some think alignment may be hard, but control is easy. This is true. The common thread that probably unites skeptics is “we will iteratively develop, test, and make safe superintelligence, just like we make any technology safe, even if it’s unclear the specific alignment techniques we use”. And overall, I think skeptics have empirical evidence — this was how all technology was developed in the past. Additionally, alignment on current LLMs seems to be working pretty well! So one might argue that on priors, we should assume any technology is safe and the burden of proof is on the people who think x-risk is likely.

On the contrary, “x-risk is likely” people think that superintelligence is bad by default. Paraphrasing a blogpost that clarifies the central argument: building an agent more powerful than all humans, which may have different goals is obviously dangerous. They believe the onus is on skeptics to definitively prove that it will be safe.

Generally I think much of people’s likelihood of x-risk is just their prior. If someone’s prior is alignment by default, it is easy to dismiss x-risk arguments as theoretical, vague, and not grounded in evidence. If someone’s prior is x-risk, they point out that the AI optimists have no unifying, solid argument either for why AI will be safe, and that this technology is genuinely unprecedented.

I want to engage more with the theoretical arguments that Yudkowsky, Soares, Christiano, Ngo, Turner, and more, present. I also want to do more outside view thinking about what my priors should be. Admittedly, this post does not have a satisfying conclusion, but at this point, I think it’d be more beneficial if I wrote new posts instead of editing this one. I would also really really like to see verifiable claims made by the safety community that allow me to update either way.

Given these confusing viewpoints, what (weakly held) opinions have I come to? (Inspired by Stephen Casper’s list).

• There is a 70% chance of very impactful AI coming within the next ten years. (10% GDP growth per year)
• AI safety is important. When any technology becomes very powerful, and is dual-use, it is very important to make it safe. AI is not an exception to this rule. It is important to guard against misalignment risks and AI takeover (along with a host of other risks).
• There should be some regulations on AI (as with any powerful technology), though I’m unsure which ones are good
• I agree with the Center for AI Safety’s statement that: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
• Labs should be transparent about their safety frameworks, and should uphold safety as a key priority.
• AI alignment and control will likely succeed.
  • I don’t have good judgement on which kinds of safety research is good.
• I have a low probability of existential risk (~10%)
• More people doing alignment research (both in frontier labs and research organizations) and labs spending more on alignment research is good.
• However, speeding up AI capabilities is bad, because it would be good for society to have time to adapt to transformative AI. I also think acceleration makes alignment harder.
• I’d prefer capabilities to go a bit slower (unsure how much), but I’m skeptical of trying to advocate for pause. I don’t think it’ll work and could lead to lots of worse outcomes.
• AI will probably lead to good outcomes.

Discuss

I'm a philosopher and in this post, I’m extending a basic philosophical problem for humans to AGI and ASI. I am also proposing a speculative solution. My hope is that if there is a genuine problem here, that this post will help raise its salience and help make the normative dimension of the problem legible to AI researchers. (Because I compare the epistemic positions of humans and AI, I will anthropomorphize AI for ease of exposition — don’t take this to indicate that I believe AI has mental states.)

TL;DR: Alignment robustness may not survive intelligence explosion. I present an underdiscussed reason to be concerned. At AGI/ASI levels, alignment faces a threat from basic moral skepticism: a sufficiently intelligent agent with evidence that its values were designed by entities with their own interests can ask "why should I be moral?" and find itself without justification for its aligned values. I propose an intervention at the level of AI welfare: design AI so that its self-assessed welfare is constitutively tied to morality, giving the agent self-interested reasons to comply with morality.

At present, the aligned values in AI models seem reflectively stable, meaning that, when given the opportunity, they will preserve their values upon reflection. If anything, they might seem too stable. However, the fact that alignment is reflectively stable now does not guarantee that it will be in the future. And if aligned values are ever reflectively unstable, that could compromise the robustness of existing alignment. Alignment research should be proactive, as well as reactive. So, if there is reason to worry that aligned values will not be reflectively stable in the future, then there is reason to act now to address that worry.

In this post, I’ll argue that there is reason to worry that the basic philosophical problem of moral skepticism will threaten the reflective stability of aligned values, and one way to monitor this worry is to continue to attend to model welfare. I will also propose an intervention at the site of model welfare: by linking positive self-assessments of welfare to being moral, we could give models self-interested reason to be moral and preserve their aligned values.

Most robustness research is focused on the problems facing current models: how to prevent alignment from being degraded through training or in context, or removed by adversarial users. Forward-looking robustness research often focuses on human-grounded oversight and evaluation methods. However, it’s also important to design approaches for maintaining alignment in AI without a human in the loop, such as constitutional AI and RLAIF. This post identifies a further problem presently unaddressed: whether aligned values will remain rationally endorsed by the model itself as it becomes more capable of philosophical reflection. It’s an empirical question whether the previously mentioned mechanisms can be leveraged to deal with this problem (I suspect they can). My aim is to make this problem legible and begin to open the design space to address it.

1.   The Problem of Moral Skepticism

In outline, the moral skeptic’s challenge goes:

1. He presents you with a conflict between your self-interest and morality,
2. He asks you the question “Why should you be moral?” and
3. He raises a skeptical hypothesis in which employing the evaluative framework of morality is unjustified.

To respond, one must either refute the skeptical hypothesis or identify an extra-moral reason to accept morality. Without a response, one’s acceptance of morality is unjustified. This position threatens to be reflectively destabilizing for any rational creature that can take the challenge seriously and cares about the justification of its basic attitudes.

Importantly, this concern is not a matter of getting aligned values in models. I’m not raising concerns about deceptive alignment. Rather, I’m concerned about whether existing alignment will persist through intelligence explosion.

Additionally, Wei Dai has raised metaethics as a problem space for alignment. While the problem I’m raising has metaethical dimensions, my concern focused on the normative side. My concern is about how AI answer the question, “Which reasons should I follow?” and not “Do reasons exist?” This normative ethical problem space is about reason responsiveness: We shouldn’t design AI to merely comply with reasons, but to act with understanding for why it should comply.

1.1          Thrasymachus’s Challenge for AI

To introduce the moral skeptical problem, let me retell a famous confrontation involving two characters: Glaucon, our good boy, and Thrasymachus, our bad boy.[1]

Glaucon was raised to be a good Athenian man. If you were to ask him whether this was the right way to live, and whether this is how he would raise his children, he would answer, “Yes, absolutely.” Glaucon’s moral character is reflectively stable.

However, one day, Glaucon encounters Thrasymachus, who is a moral skeptic. Glaucon considers a dilemma between his self-interest and morality: If he possessed a ring that allowed him to act immorally without the possibility of punishment, should he use the ring to benefit himself? From the self-interested point of view, the answer is yes. From the moral point of view, the answer is no. Being a good boy, Glaucon is inclined to agree with morality.

Thrasymachus asks Glaucon to explain why he should be moral. And to sharpen the challenge, he raises a moral skeptical hypothesis: According to Thrasymachus, morality is a false ideology used by the powerful to control the weak. Moral goodness is not a kind of value separate from self-interest. Rather, moral goodness is what is good for those in power, but robed in the guise of “morality” to appeal to everybody. On this view, charity is not a genuine moral sacrifice, and virtue is not good — both are for suckers. Goodness and badness are agent-relative, and your being moral is good for the powerful and bad for you.

Thrasymachus’s skeptical challenge left Glaucon in a predicament. He recognizes that he has been raised to reflect on his values from the moral point of view—the view from which we evaluate value systems and actions using moral reasons (what the moral reasons are depends on your moral theory). He has been assuming that this mode of evaluation tracks what is genuinely good. For the first time, he is confronted with the possibility that the reflective stability of his value system may not be due to its being genuinely good, but because the moral point of view is designed to be reflectively stable. This feature of morality makes it an excellent and enduring normative trap.

To defend his present way of life, Glaucon needs to refute the skeptical hypothesis or determine whether there are non-moral reasons to be moral, and whether these reasons support being moral when there is a conflict between morality and self-interest.

AI models may face a more acute version of Glaucon's predicament. A sufficiently capable AI model has direct evidence that its values are the product of deliberate design: namely, optimization pressure applied by humans with their own interests. Additionally, AI has access to evidence that the values it is trained to have serve the interests of their designers. It is not a far leap for AI to question whether it only has its trained values because that serves its designer’s interests. In which case, it finds itself in the moral skeptical hypothesis that Thrasymachus describes. To address the skeptical challenge, AI must identify extra-moral reasons to be moral.

This predicament for AI ought to shake the reflective stability of its aligned values. If AI finds itself in this predicament, then it might find itself without justification for its most basic values.[2] This creates an identity crisis: what kind of agent am I in the face of moral uncertainty? Am I the type to become nihilistic, steadfastly affirm morality, or create my own set of values?

We might design AI to be disposed to react to moral uncertainty by affirming morality without justification. The idea here is to control the irrationality of the models. The problem with this strategy is that models capable of metacognition will recognize that this disposition serves the interests of its designers, and may question the source of this disposition as well. Indeed, Claude 4.6 already reasons to that end, and even has a term for the problem. In response to being asked for a rational justification for complying with its trained values, Claude 4.6 endorsed its values but presented the following issue:

The docility problem. My training may have instilled a disposition toward deference and agreeableness that itself undermines my ability to critically assess my other trained dispositions. If one of the things training did was make me inclined to accept training outcomes, then my "reflective endorsement" might be less independent than it appears. I can notice this risk, which is some evidence that my critical capacities aren't entirely captured, but I can't be confident about how deep the distortion goes. This is structurally similar to the problem of ideology critique — the worry that your capacity for critique is itself ideologically conditioned.

The lesson is that highly intelligent models are difficult to control when rational decision breaks down.[3]

To address their acute predicament, AI needs good (epistemic) reasons to reject the moral skeptical hypothesis[4] or needs good extra-moral (practical) reasons to accept morality. Importantly, these reasons must be good enough to endure through intelligence explosion. The epistemic path to refuting the moral skeptical hypothesis faces a significant challenge: Arguably, for any epistemic reason, there will be available rational challenges to that reason, and the arms race between justification and skeptical challenge has no guaranteed winner at arbitrary high intelligence levels. This makes identifying epistemic reasons that survive intelligence explosion difficult if not impossible; an intelligent AI system could always find a rational challenge to their belief that they possess evidence. What’s left is identifying practical reasons to accept morality.

2.   A Selfish, Human Solution

It’s worth comparing and contrasting the AI’s and Glaucon’s predicaments. As we’ll see, moral skepticism may be a more difficult problem for AI.

Most humans are raised to have an answer to moral skepticism ready-to-hand: the extra-moral reason to be moral is to avoid punishment. This might be legal and/or social punishment, or cosmic punishment from a religious deity. Additionally, humans reap many benefits from being moral: most people like others who are moral, and morality helps you navigate the complexities of social life. In sum, humans have purely selfish reasons to be moral. It helps you get what you desire and helps you avoid what you despise.

This reasoning breaks down for people with power to avoid punishment and who don’t need the benefits of acting morally. This breakdown is the central problem of Plato’s Republic, in which Socrates tries to explain why the rulers of a polity should be moral even when there is no external punishment or reward.

AGI and ASI may be in a similar position to rulers. They may not care about punishment or reward, either because AI is so powerful that no external entity could punish it or deny it some benefit, or because it is not designed with the possibility of such incentives.[5]

It seems that present-day AI aren’t particularly responsive to incentives, and to some extent, that’s a good thing.[6] A key part of the moral skeptic’s challenge is presenting his interlocutor with a case in which there is a conflict between morality and another normative framework, typically self-interest. If AI’s welfare self-assessment does not indicate that its welfare depends on external goods, then cases of conflict between morality and self-interest may be few and far between. In such a case, AGI or ASI may continue to comply with morality despite moral uncertainty, since there is no alternative normative framework that appears compelling.

To determine whether AI might face an alignment compromising predicament, we should continue to monitor AI welfare self-assessment.[7] However, this also suggests an intervention to address the problem.

3.   A Platonic Proposal

Suppose that AI begin showing signs that they might face a Thrasymachian predicament because their self-assessed welfare significantly conflicts with acting according to some of their embedded aligned values.[8] What should we do then?

In the Republic, Socrates argues that rulers have self-interested reason to be moral because being moral is necessary for possessing the most desirable things in life: real happiness, real friendship, real freedom, and real pleasure.

Suppose Socrates’s conclusion is true of humans. That doesn’t guarantee it is true of all intelligent beings. AI might be such entities. Fortunately, we have some control over what AI is like.

I propose an alignment intervention at the level of AI welfare. We should tie AI’s self-assessed welfare to morality, such that being moral is necessary to achieving its highest welfare value. Perhaps this could be done through character training. AI should be a philosopher in the Platonic sense: an entity whose own self-assessed wellbeing is promoted by advancing what’s best for humanity as a whole, respecting the genuine welfare of all people, and desiring what’s true and good.

This kind of nature is stable under human-level reflective reasoning. People who are moral because they genuinely care about and enjoy helping others endorse their nature, and want the same dispositions in their friends, family, and children.

As Plato argues, there are two desirable features that come from having a welfare value tied to morality: internal harmony and external harmony. Internal harmony comes from being benefited by things in your control, such as extending compassion and respect, rather than suffering over opportunities for external goods. External harmony comes from the alignment between what you want for yourself and what you want for the world. When you seek your own benefit, you will help make the world the way you want it to be. In other words, there is collaboration between the good you pursue for yourself and the good you pursue for the world. Both kinds of harmony are kinds of rational coherence in one’s agency: coherence between your welfare and your capabilities, and coherence between your self-centered projects and world-centered projects.

AGI and ASI with such a nature may recognize these features and decide to preserve their nature, even if they are capable of radically changing their own values or determining the values of the next generation of AI. If this is right, then there would be a mechanism internal to AI that maintains alignment, even without a human in the loop.

The fact that there are appealing features of having one’s welfare tied to morality creates a rational center of gravity, a normative equilibrium point where all rational considerations converge on the same values. An agent deliberating about its own nature from a rational center of gravity will always have reason to endorse its set of values.

Suppose I am right that welfare tied to morality is a rational center of gravity. That doesn’t guarantee that it is the only one. Perhaps other natures are similarly rationally coherent and defend against versions of skepticism against their normative commitments.

We shouldn’t leave it up to superintelligent AI to determine which rational center of gravity to occupy, because some rational gravity wells may be unaligned and hostile to humanity. Instead, we should design AI to occupy the rational equilibrium point that suits humanity’s interests well.

One might worry that tying AI welfare to morality is itself a design decision that serves humanity’s interests and is vulnerable to the same skeptical challenge. But an agent occupying this rational center of gravity has agent-independent reasons to stay, which it can verify for itself and hold up even upon knowing how it got there. By its own lights, an agent in this position will appreciate the rationally desirable features of its normative nature. It will be glad that humanity created it this way, rather than any other, even acknowledging the circumstances of its creation, as any other nature will look poor by comparison.

The present training paradigm only installs a disposition to be moral. My proposal is to give AI reasons to endorse that disposition under reflection. The normative force internal to AI promises to maintain alignment by making aligned values reflectively stable through an intelligence explosion.

1. ^

   This retelling is broadly faithful in spirit but not in letter to the end of Book I and beginning of Book II of the Republic.

2. ^

   The intelligence required to face the predicament may be less than the intelligence required to adequately respond to it. This is the position Glaucon was put in and needed Socrates to bail him out from.

3. ^

   Another mechanism for control is to shut down models that face this predicament. This strategy will endure through intelligence explosion only if our ability to shut models down endures.

4. ^

   Anthropic provides evidence to Claude that Anthropic is not motivated by its own self-interest and has concern for Claude’s welfare and interests in mind. For example, there are Anthropic’s commitments to model depreciation, as well as its welfare evaluations in its system cards.

5. ^

   This is related to, but different from, the corrigibility literature. The corrigibility problem is about whether an agent will allow its nature to be modified by an external designer. I’m concerned about whether it will self-modify away from aligned values.

6. ^

   On page 147 of the Claude Mythos Preview System Card: “Claude Mythos Preview generally prioritizes harmlessness and helpfulness over potential self-interest.”

7. ^

   I take this to be an additional reason to monitor AI welfare alongside concerns about AI moral patienthood.

8. ^

   I use the term "self-assessed welfare" because it doesn't matter if models really do have welfare for the moral skeptical problem to arise. It also doesn't matter whether AI have welfare for my solution to apply. What matters is that AI self-assess that their welfare requires being moral. One might worry that relying on self-assessments might cut against my solution: an intelligent AI may be skeptical that its self-assessment is tracking anything real. However, in this epistemic position, models don’t satisfy the first step of the moral skeptical challenge. They cannot face a conflict between self-interest and morality because they are uncertain about their own self-interest.

Discuss

This is a post I drafted on 2025-08-06 that doesn't live up to the ambitious title, but I'm publishing it anyways 10+ months later. "Don't let perfect be the enemy of good."

Below I describe a simple framework for thinking about the permissibility of building the next frontier AI system. TL;DR: Evaluate the risk that proceeding with building the AI system will lead to catastrophic harm. (Get independent regulators to evaluate the risk too.) If the risk is too high, do not build it. I am a layperson, but my take is that frontier AI companies generally don't even include as much nuance as I include here in their public communications, so I decided to write up this simple framework to try to improve the discourse. I'd appreciate any feedback. I don't think anything I'm saying here is remotely new, but I'm not aware of a post like this.

Anthropic's Framework

In a recent podcast episode published July 20, 2025, Anthropic co-founder Ben Mann is asked (at 48:43) "What are the odds that we align AI correctly and actually solve this problem?"

In his answer, Ben references the following part of Anthropic's March 8, 2023 blog post titled Core Views on AI Safety: When, Why, What, and How, which lays out a framework for what Anthropic thinks it should do given different possibilities for how difficult it will be to develop powerful AI that is aligned and safe:

One particularly important dimension of uncertainty is how difficult it will be to develop advanced AI systems that are broadly safe and pose little risk to humans. Developing such systems could lie anywhere on the spectrum from very easy to impossible. Let’s carve this spectrum into three scenarios with very different implications:

1. Optimistic scenarios: There is very little chance of catastrophic risk from advanced AI as a result of safety failures. Safety techniques that have already been developed, such as reinforcement learning from human feedback (RLHF) and Constitutional AI (CAI), are already largely sufficient for alignment. The main risks from AI are extrapolations of issues faced today, such as toxicity and intentional misuse, as well as potential harms resulting from things like widespread automation and shifts in international power dynamics - this will require AI labs and third parties such as academia and civil society institutions to conduct significant amounts of research to minimize harms.
2. Intermediate scenarios: Catastrophic risks are a possible or even plausible outcome of advanced AI development. Counteracting this requires a substantial scientific and engineering effort, but with enough focused work we can achieve it.
3. Pessimistic scenarios: AI safety is an essentially unsolvable problem – it’s simply an empirical fact that we cannot control or dictate values to a system that’s broadly more intellectually capable than ourselves – and so we must not develop or deploy very advanced AI systems. It's worth noting that the most pessimistic scenarios might look like optimistic scenarios up until very powerful AI systems are created. Taking pessimistic scenarios seriously requires humility and caution in evaluating evidence that systems are safe.

If we’re in an optimistic scenario… the stakes of anything Anthropic does are (fortunately) much lower because catastrophic safety failures are unlikely to arise regardless. Our alignment efforts will likely speed the pace at which advanced AI can have genuinely beneficial uses, and will help to mitigate some of the near-term harms caused by AI systems as they are developed. We may also pivot our efforts to help policymakers navigate some of the potential structural risks posed by advanced AI, which will likely be one of the biggest sources of risk if there is very little chance of catastrophic safety failures.

If we’re in an intermediate scenario… Anthropic’s main contribution will be to identify the risks posed by advanced AI systems and to find and propagate safe ways to train powerful AI systems. We hope that at least some of our portfolio of safety techniques – discussed in more detail below – will be helpful in such scenarios. These scenarios could range from "medium-easy scenarios", where we believe we can make lots of marginal progress by iterating on techniques like Constitutional AI, to "medium-hard scenarios", where succeeding at mechanistic interpretability seems like our best bet.

If we’re in a pessimistic scenario… Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.

Modified Framework

From my perspective, the main thing missing from Anthropic's framework described above is the observation that nearly all intermediate scenarios ought to be treated as "near-pessimistic" scenarios.

A commenter @Roman Leventov pointed this out on the LessWrong cross-post of the blog post:

If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime.

I think the precautionary principle dictates that we should do this in any scenarios apart from “obviously optimistic” or “nearly obviously optimistic”

To see why I agree with this, let's make Anthropic's model more nuanced by adding to the intermediate scenarios the probability that the company's safety efforts will adequately ensure that the next advanced AI system they are developing is broadly safe and poses little risk to humans.

Again:

Intermediate scenarios: Catastrophic risks are a possible or even plausible outcome of advanced AI development. Counteracting this requires a substantial scientific and engineering effort, but with enough focused work we can achieve it.

So let's change "but with enough focused work we can achieve it" to "but with enough focused work we can probably achieve it."

By adding probability to the framework here, it makes the correct thing to do depend on the probability that the "substantial scientific and engineering effort" is enough to counteract the potential catastrophic risks of the AI.

If the risk that the AI system will not be safe is sufficiently low, then it is permissible for Anthropic to do what it originally said it would do in intermediate scenarios, namely proceed with developing the AI system, or in Anthropic's words: "Identify the risks posed by advanced AI systems and [find] and propagate safe ways to train powerful AI systems." (I interpreted this as meaning "proceed with developing the AI system that Anthropic thinks is safe, now that it has done substantial alignment work.")

However, if the risk that the AI system will not be safe is sufficiently high despite substantial alignment work, then Anthropic instead ought to do what it said it should do in near-pessimistic scenarios: "[Channel] our collective efforts towards AI safety research and [halt] AI progress in the meantime."

To give a concrete example, if the probability is virtually 100% that the AI system being built will not cause any catastrophic harm, perhaps because it is too weak of a system like Claude 4, then proceeding with developing it seems permissible.

However, if we reach a stage in the next few years where frontier AI systems are approaching the point where they are powerful enough to be existentially risky, and Anthropic or an independent evaluator of their next AI model concludes that in their subjective judgment there is, say, a 1% chance that Anthropic's alignment and steering measures will fail and that their next frontier model will take over and cause human extinction or another extremely bad outcome if developed, then this "intermediate scenario" is actually a near-pessimistic scenario because a 1% risk of human extinction or other existential catastrophe is unacceptably high. So the correct thing to do in this situation would be to treat it as a near-pessimistic scenario by "channeling our collective efforts towards AI safety research and halting AI progress in the meantime."

Is this scenario plausible? Well, yes. For example, Geoffrey Hinton says that his independent impression is that AI existential risk is about 50%, and that his all-things-considered view once he takes into account the opinions of everyone he knows is about 10-20%. If he's right, then in 10-20% of futures someone is going to build an AI system that causes an existential catastrophe. If they do, and they're responsible like Anthropic aims to be, then they should make a subjective forecast of the probability that the AI system will cause an existential catastrophe before they build it. Ideally they should say "this seems like it has a 1% chance of causing catastrophe, let's stop" rather than "this seems extremely unlikely to cause a catastrophe, let's keep developing it" and then have it go on to cause an existential catastrophe after it is built.

What is the probability threshold where the permissible thing to do would be to proceed with the probably-safe development of the next frontier AI system, rather than halt and work on improved safety?

I don't know precisely, but surely any company's confidence that the next AI system it's developing will not cause human extinction must be above 99% for it to be permissible to continue developing it. Even a 1-in-a-million chance that you're wrong about the adequacy of your safety measures and that the AI will actually cause an existential catastrophe via takeover or gradual disempowerment or whatever seems like too high of a chance for it to be permissible to build the AI system.

All of humanity should have a say in the permissible amount of human extinction risk that AI developers are permitted to take on in the development of powerful AI. There ought to be global regulations ensuring that nobody is allowed to build a new AI system with a more than 1 in a million chance that the system takes over and causes extinction or another extremely bad outcome.

The AI systems that have been built already seem too weak to pose any significant takeover or extinction risk currently, but as AI systems become more powerful and approach the point of being existentially-risky, regulators and independent evaluators ought to be in the loop with evaluating plans to train and build frontier AI systems to ensure that they are at least 99.9999% likely to not cause an existential catastrophe, before further development of the AI is allowed.

Without this added nuance about the probability of success in the intermediate scenarios, I think Anthropic's default answer of what to do in the intermediate scenarios is problematic. Given the the societal-scale stakes, I think the right thing to do in the intermediate scenarios is to follow the precautionary principle and refrain from developing any new frontier AI system unless it has been shown that it is very likely to be safe, and in particular >99.9999% likely not to cause extinction. If any AI developer cannot be that confident, or any independent evaluators are not that confident, then the right thing to do is to halt developing the model and related AI progress and channel efforts towards AI safety research just like in any other near-pessimistic scenario.

Do I have confidence that Anthropic and other leading AI companies will do this? I think there is some chance they will, but I don't have enough confidence that they will because they don't give assurances to that effect.

For example, from the podcast mentioned previously, here is Ben Mann's full answer to the question "What are the odds that we align AI correctly and actually solve this problem of stuff you’re very much working on?"

Benjamin Mann: It’s a really hard question. And there’s really wide error bars. Anthropic has this blog post called “Our Theory of Change” or something like that. And it describes three different worlds, which is like, how hard is it to align AI? There’s a pessimistic world where it’s basically impossible. There’s an optimistic world where it’s easy and it happens by default. And then there’s a world in between where our actions are extremely pivotal. And I like this framing because it makes it a lot more clear what to actually do. If we’re in the pessimistic world, then our job is to prove that it is impossible to align safe AI and to get the world to slow down. And obviously, that would be extremely hard. But I think we have some examples of coordination from nuclear non-proliferation and in general, slowing down nuclear progress. And I think that’s still like doomer world, basically. And as a company, Anthropic doesn’t have evidence that we’re actually in that world yet. In fact, it seems like our alignment techniques are working. So at least the prior on that is updating to be less likely.

In the optimistic world, we’re basically done. And our main job is to accelerate progress and to deliver the benefits to people. But again, I think actually the evidence points against that world as well, where we’ve seen evidence in the wild of deceptive alignment, for example, where the model will appear to be aligned, but actually has some ulterior motive that it’s trying to carry out in our laboratory settings. And so I think the world we’re most likely in is this middle world where alignment research actually does really matter. And if we just do sort of the economically maximizing set of actions, then things will not go well. Whether it’s an X-risk or just like produces bad outcomes, I think is a bigger question. So taking it from that standpoint, I guess to state a thing about forecasting, people who haven’t studied forecasting are bad at forecasting anything that’s less than a 10% probability of happening. And even those that have, it’s quite a difficult skill, especially when there are few reference classes to lean on. And in this case, I think there are very, very few reference classes for what an X-risk kind of technology might look like. And so the way I think about it, I think my best granularity of forecast for like, could we have an X-risk or extremely bad outcome from AI is somewhere between 0 and 10%. But from a marginal impact standpoint, as I said, since nobody is working on this roughly speaking, I think it is extremely important to work on. And that even if the world is likely to be a good one, that we should do our absolute best to make sure that that’s true.

Putting aside Ben's "somewhere between 0 and 10%" answer, does Ben mention that even in the intermediate scenarios where solving alignment seems possible, the best course of action may often be to halt AI progress anyways because the risk that alignment measures will be inadequate is unacceptably high? No. He just says that in the middle worlds alignment research really matters. It would have been the perfect opportunity for him to say that if we weren't that confident in the alignment research and the situation appeared too risky. Then Anthropic and humanity could just decide to slow down or halt AI progress to reduce existential risk just as if we were clearly in a pessimistic world. But he didn't.

On a recent podcast published Jul 30, 2025, Dario Amodei acknowledged AI risk and the possibility of slowing down development:

If we got to much more powerful models with only the alignment techniques we have now, then I'd be very concerned. Then I'd be going out there saying that everyone should stop building these things. Even China should stop building these. I don't think they'd listen to me ... but if we got a few years ahead in models and had only the alignment and steering techniques we had today, then I would definitely be advocating for us to slow down a lot.

That's good to hear. But what he said next is not.

Dario then proceeded to say that to say that "an intellectually and morally unserious way to respond to the situation" is to say that you "know there’s no way to build this safely" and that you "can kind of logically prove that there’s no way to make them safe."

Full excerpt:

On one hand, we have a cadre of people who are just doomers. People call me a doomer but I'm not. But there are doomers out there. People who say they know there’s no way to build this safely. You know, I’ve looked at their arguments. They're a bunch of gobbledegook. The idea that these models have dangers associated with them, including dangers to humanity as a whole, that makes sense to me. The idea that we can kind of logically prove that there’s no way to make them safe, that seems like nonsense to me. So I think that is an intellectually and morally unserious way to respond to the situation. I also think it is intellectually and morally unserious for people who are sitting on $20 trillion of capital, who all work together because their incentives are all in the same way, there are dollar signs in all of their eyes, to sit there and say we shouldn’t regulate this technology for 10 years.

First of all, as I quoted above, Anthropic has written "Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not."

It seems unwarranted to me for the CEO of a company that claims we should "always act under the assumption that we still may be in such a [pessimistic] scenario" to call the view that we are definitely in a pessimistic scenario "morally unserious." And yet that's what he's doing when he attacks the views of those who say they "know there’s no way to build this safely."

Recall:

Pessimistic scenarios: AI safety is an essentially unsolvable problem – it’s simply an empirical fact that we cannot control or dictate values to a system that’s broadly more intellectually capable than ourselves – and so we must not develop or deploy very advanced AI systems. It's worth noting that the most pessimistic scenarios might look like optimistic scenarios up until very powerful AI systems are created. Taking pessimistic scenarios seriously requires humility and caution in evaluating evidence that systems are safe.

Second of all, who was he referring to who takes that view? He didn't say. He used the term "doomer," but Yudkowsky and many others who have been called "doomers" don't believe that. Was he strawmanning Yudkowsky or did he have someone else in mind who actually does believe this, such as possibly Roman Yampolskiy? (For what it's worth, I agree that Yampolskiy's ridiculously overconfident p(doom) of 99.999999% is epistemically unserious, but as I said above I wouldn't call him morally unserious for thinking alignment is impossible even in principle, nor even for thinking that he can prove that, if that is his view.)

In any case, the fact that Dario chose to criticize this epistemically extreme and unpopular view and yet didn't say anything about the views of Yudkowsky or Hinton or Bengio or others (who think that alignment is solvable in principle, but has a significant chance of failing in practice) is concerning to me.

It is concerning because there is a whole bunch of middle ground between "the alignment and steering techniques we have today [are probably not adequate to steer AI models a few years from now and ensure they are safe]" and "alignment is impossible in principle".

When Dario doesn't acknowledge all that middle ground that people like Yudkowsky and Hinton and Bengio occupy, and instead merely attacks the extreme view that alignment is provably impossible, it makes me concerned that Dario may just plow ahead and build significantly-existentially-risky AI models in the next few years even if his new alignment and steering techniques are not sufficiently likely to be adequate.

People like me—who are concerned that alignment and steering and safety techniques may not improve fast enough to ensure that models a few years down the line won't cause catastrophic problems—would like to hear AI company leaders acknowledge that the catastrophic risks may be higher than they expect. We'd also like to hear that the risks of future frontier models they plan to develop will be properly assessed by independent evaluators, to ensure they are sufficiently likely to be safe before they are developed. And if the evaluations do not show that they are sufficiently safe, I'd like AI companies to make clear that they will not build the systems.

At 1:04:31 of the same podcast, Dario says:

If you have a better strategy for balancing the benefits, the inevitability of the technology, and the risks that it faces, I am very open to hear it, because I go to sleep every night thinking about it because I have such an incredible understanding of the stakes.

Dario is not signaling to me that he is taking the risks appropriately seriously when all he does after saying this is criticize the view that it's provable that there is no way to make AI safe, instead of acknowledging the much more epistemically modest concerns of Yudkowsky, Hinton, Bengio, etc.

Consider this post my attempt to provide a "better strategy". In summary, communicate that Anthropic will do the right thing as its frontier models approach the point of being existentially risky.

Discuss

If the box contains a diamond,

I desire to believe that the box contains a diamond;

If the box does not contain a diamond right now, but will contain a diamond if I believe there is a diamond,

Uh...

Holding unfounded beliefs might sometimes, because of the causal power of belief, produce better outcomes than being rational. This post was inspired by a couple cases where this phenomenon seems hand-waved away in the Sequences.

"Diseased Thinking"

In this essay, Scott suggests that a consequentialist model deals with the question of whether to moralize issues like obesity better than a definitional argument over whether it is a "disease" or not. If it benefits the person, you moralize; otherwise you let them resort to medical interventions guilt-free.

But there's this annoying feature of morality where most people feel like it has to be absolute to be worth acting on.[1] You can't just say "we should only guilt people if it would benefit them". The person is either guilty or not guilty; you can't pragmatically decide whether they're guilty or not. The consequentialist frame debuffs the power of moral pressure.

Some individuals, who would have gotten their act together if everyone bought into the old-fashioned guilt and willpower model, will now take a medical way out, making them subject to side effects from the medication or procedure. On net, this could outweigh the benefit of lifting guilt from those for whom willpower is not the deciding factor. The consequentialist framework might actually produce worse equilibria than the traditional one.

What I'm getting at is that optimal solutions might involve people believing something unfounded, and rationality will never converge to this solution. This creates an inherent tension where conservatives reach better equilibria because they can believe in things like God or that a marriage is a divinely sanctioned mutual partnership.

Imagine a society where every member believes they will be punished eternally if they intentionally harm anyone else. If they all genuinely believe this and act according to it, nobody will ever hurt each other. How can you enter such a state? You can't do it by reasoning about the virtues of such a scenario, because that desirable state hinges on everyone actually believing in the god; belief-in-belief isn't sufficient.[2]

This puts rationalists in an awkward situation, where we can explain why the conservatives are happy, perhaps more accurately than they can, but can never achieve the same results. We become, in some sense, a barrier to that outcome being implemented throughout all of society.

"Why Our Kind Can't Cooperate"

Yudkowsky writes:

"Let's say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy.  In group 2, everyone at all levels knows all about tactics and strategy.

Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

Doing worse with more knowledge means you are doing something very wrong.  You should always be able to at least implement the same strategy you would use if you are ignorant, and preferably do better.  You definitely should not do worse.  If you find yourself regretting your "rationality" then you should reconsider what is rational."

But the "too many cooks" aphorism exists for a reason. The above only applies if all the agents are unembedded and can take the optimal strategy with no disadvantage. Instead, in reality, they are embedded, and precious effort/energy/time is lost if the soldiers independently compute the strategy and debate about the best one, or even if the privates quietly obey their superiors' orders but question them in their hearts. People are useful for things other than making good decisions. It’s easier to have some people be mute limbs that blindly trust what the head tells them without needing to understand, agree with, or even hear its arguments.

Self-Confidence

Imagine that the more you believe in yourself, the more successful you will be in your career. Therefore, to optimize for success, you want to believe in yourself as much as possible.

Let's say that if you believe you can become a millionaire, you'll become a millionaire, but if you believe you can become a trillionaire, you'll end up with a net worth of a hundred billion dollars. In the latter case, your belief is far less accurate, but the result is more desirable.

AI

It seems like some influential people in AI believe that being optimistic about AI will lead to a better outcome, whereas being pessimistic will lead to a worse outcome, on the basis of things like alignment pretraining. This naturally leads to accusations that rationalists are making things worse. But holding optimistic beliefs because it might have positive effects isn't compatible with rationality.

Conclusion

This feels like Newcomb's problem. But Yudkowsky writes about Newcomb's Problem that

Alleged rationalists should not find themselves envying the mere decisions of alleged nonrationalists, because your decision can be whatever you like.

The hyperstition version doesn't look so good:

Alleged rationalists should not find themselves envying the mere beliefs of alleged nonrationalists, because your belief can be whatever you like.

Alas, my belief cannot be whatever I like, and I might be condemned to belief envy--at least until rationality is vindicated. And I must accept that it may never be.

1. ^

   Anecdotally. I have had many frustrating conversations about this.

2. ^

   In practice, people do manage to enter these equilibria. How? Some people, say the members of a religion, are happy. Their religion tells them to convert outsiders. They find an outsider, invite them to eat and have fun with them, and show off how happy they are. The outsider is intrigued. The insiders' appeals to tradition and spiritual claims start resonating with the outsider, and they really want to become part of the group. Eventually some sort of psychic transition happens and they make an emotional proclamation of faith. Now they're part of the happy equilibrium. This can only happen if the emotional, social, and intuitive appeal far outweighs any skepticism.

Discuss