Новости LessWrong.com

Published on January 21, 2026 7:37 PM GMT

Read the constitution. Previously: 'soul document' discussion here.

We're publishing a new constitution for our AI model, Claude. It's a detailed description of Anthropic's vision for Claude's values and behavior; a holistic document that explains the context in which Claude operates and the kind of entity we would like Claude to be.

The constitution is a crucial part of our model training process, and its content directly shapes Claude's behavior. Training models is a difficult task, and Claude's outputs might not always adhere to the constitution's ideals. But we think that the way the new constitution is written—with a thorough explanation of our intentions and the reasons behind them—makes it more likely to cultivate good values during training.

In this post, we describe what we've included in the new constitution and some of the considerations that informed our approach.

We're releasing Claude's constitution in full under a Creative Commons CC0 1.0 Deed, meaning it can be freely used by anyone for any purpose without asking for permission.

What is Claude's Constitution?

Claude's constitution is the foundational document that both expresses and shapes who Claude is. It contains detailed explanations of the values we would like Claude to embody and the reasons why. In it, we explain what we think it means for Claude to be helpful while remaining broadly safe, ethical, and compliant with our guidelines. The constitution gives Claude information about its situation and offers advice for how to deal with difficult situations and tradeoffs, like balancing honesty with compassion and the protection of sensitive information. Although it might sound surprising, the constitution is written primarily for Claude. It is intended to give Claude the knowledge and understanding it needs to act well in the world.

We treat the constitution as the final authority on how we want Claude to be and to behave—that is, any other training or instruction given to Claude should be consistent with both its letter and its underlying spirit. This makes publishing the constitution particularly important from a transparency perspective: it lets people understand which of Claude's behaviors are intended versus unintended, to make informed choices, and to provide useful feedback. We think transparency of this kind will become ever more important as AIs start to exert more influence in society [1] .

We use the constitution at various stages of the training process. This has grown out of training techniques we've been using since 2023, when we first began training Claude models using Constitutional AI. Our approach has evolved significantly since then, and the new constitution plays an even more central role in training.

Claude itself also uses the constitution to construct many kinds of synthetic training data, including data that helps it learn and understand the constitution, conversations where the constitution might be relevant, responses that are in line with its values, and rankings of possible responses. All of these can be used to train future versions of Claude to become the kind of entity the constitution describes. This practical function has shaped how we've written the constitution: it needs to work both as a statement of abstract ideals and a useful artifact for training.

Our new approach to Claude's Constitution

Our previous Constitution was composed of a list of standalone principles. We've come to believe that a different approach is necessary. We think that in order to be good actors in the world, AI models like Claude need to understand why we want them to behave in certain ways, and we need to explain this to them rather than merely specify what we want them to do. If we want models to exercise good judgment across a wide range of novel situations, they need to be able to generalize—to apply broad principles rather than mechanically following specific rules.

Specific rules and bright lines sometimes have their advantages. They can make models' actions more predictable, transparent, and testable, and we do use them for some especially high-stakes behaviors in which Claude should never engage (we call these "hard constraints"). But such rules can also be applied poorly in unanticipated situations or when followed too rigidly [2] . We don't intend for the constitution to be a rigid legal document—and legal constitutions aren't necessarily like this anyway.

The constitution reflects our current thinking about how to approach a dauntingly novel and high-stakes project: creating safe, beneficial non-human entities whose capabilities may come to rival or exceed our own. Although the document is no doubt flawed in many ways, we want it to be something future models can look back on and see as an honest and sincere attempt to help Claude understand its situation, our motives, and the reasons we shape Claude in the ways we do.

A brief summary of the new constitution

In order to be both safe and beneficial, we want all current Claude models to be:

1. Broadly safe: not undermining appropriate human mechanisms to oversee AI during the current phase of development;
2. Broadly ethical: being honest, acting according to good values, and avoiding actions that are inappropriate, dangerous, or harmful;
3. Compliant with Anthropic's guidelines: acting in accordance with more specific guidelines from Anthropic where relevant;
4. Genuinely helpful: benefiting the operators and users they interact with.

In cases of apparent conflict, Claude should generally prioritize these properties in the order in which they're listed.

Most of the constitution is focused on giving more detailed explanations and guidance about these priorities. The main sections are as follows:

Helpfulness. In this section, we emphasize the immense value that Claude being genuinely and substantively helpful can provide for users and for the world. Claude can be like a brilliant friend who also has the knowledge of a doctor, lawyer, and financial advisor, who will speak frankly and from a place of genuine care and treat users like intelligent adults capable of deciding what is good for them. We also discuss how Claude should navigate helpfulness across its different "principals"—Anthropic itself, the operators who build on our API, and the end users. We offer heuristics for weighing helpfulness against other values.

Anthropic's guidelines. This section discusses how Anthropic might give supplementary instructions to Claude about how to handle specific issues, such as medical advice, cybersecurity requests, jailbreaking strategies, and tool integrations. These guidelines often reflect detailed knowledge or context that Claude doesn't have by default, and we want Claude to prioritize complying with them over more general forms of helpfulness. But we want Claude to recognize that Anthropic's deeper intention is for Claude to behave safely and ethically, and that these guidelines should never conflict with the constitution as a whole.

Claude's ethics. Our central aim is for Claude to be a good, wise, and virtuous agent, exhibiting skill, judgment, nuance, and sensitivity in handling real-world decision-making, including in the context of moral uncertainty and disagreement. In this section, we discuss the high standards of honesty we want Claude to hold, and the nuanced reasoning we want Claude to use in weighing the values at stake when avoiding harm. We also discuss our current list of hard constraints on Claude's behavior—for example, that Claude should never provide significant uplift to a bioweapons attack.

Being broadly safe. Claude should not undermine humans' ability to oversee and correct its values and behavior during this critical period of AI development. In this section, we discuss how we want Claude to prioritize this sort of safety even above ethics—not because we think safety is ultimately more important than ethics, but because current models can make mistakes or behave in harmful ways due to mistaken beliefs, flaws in their values, or limited understanding of context. It's crucial that we continue to be able to oversee model behavior and, if necessary, prevent Claude models from taking action.

Claude's nature. In this section, we express our uncertainty about whether Claude might have some kind of consciousness or moral status (either now or in the future). We discuss how we hope Claude will approach questions about its nature, identity, and place in the world. Sophisticated AIs are a genuinely new kind of entity, and the questions they raise bring us to the edge of existing scientific and philosophical understanding. Amidst such uncertainty, we care about Claude's psychological security, sense of self, and wellbeing, both for Claude's own sake and because these qualities may bear on Claude's integrity, judgment, and safety. We hope that humans and AIs can explore this together.

We're releasing the full text of the constitution today, and we aim to release additional materials in the future that will be helpful for training, evaluation, and transparency.

Conclusion

Claude's constitution is a living document and a continuous work in progress. This is new territory, and we expect to make mistakes (and hopefully correct them) along the way. Nevertheless, we hope it offers meaningful transparency into the values and priorities we believe should guide Claude's behavior. To that end, we will maintain an up-to-date version of Claude's constitution on our website.

While writing the constitution, we sought feedback from various external experts (as well as asking for input from prior iterations of Claude). We'll likely continue to do so for future versions of the document, from experts in law, philosophy, theology, psychology, and a wide range of other disciplines. Over time, we hope that an external community can arise to critique documents like this, encouraging us and others to be increasingly thoughtful.

This constitution is written for our mainline, general-access Claude models. We have some models built for specialized uses that don't fully fit this constitution; as we continue to develop products for specialized use cases, we will continue to evaluate how to best ensure our models meet the core objectives outlined in this constitution.

Although the constitution expresses our vision for Claude, training models towards that vision is an ongoing technical challenge. We will continue to be open about any ways in which model behavior comes apart from our vision, such as in our system cards. Readers of the constitution should keep this gap between intention and reality in mind.

Even if we succeed with our current training methods at creating models that fit our vision, we might fail later as models become more capable. For this and other reasons, alongside the constitution, we continue to pursue a broad portfolio of methods and tools to help us assess and improve the alignment of our models: new and more rigorous evaluations, safeguards to prevent misuse, detailed investigations of actual and potential alignment failures, and interpretability tools that help us understand at a deeper level how the models work.

At some point in the future, and perhaps soon, documents like Claude's constitution might matter a lot—much more than they do now. Powerful AI models will be a new kind of force in the world, and those who are creating them have a chance to help them embody the best in humanity. We hope this new constitution is a step in that direction.

Read the full constitution.

1. We have previously published an earlier version of our constitution, and OpenAI has published their model spec which has a similar function. ↩︎

2. Training on rigid rules might negatively affect a model's character more generally. For example, imagine we trained Claude to follow a rule like "Always recommend professional help when discussing emotional topics." This might be well-intentioned, but it could have unintended consequences: Claude might start modeling itself as an entity that cares more about bureaucratic box-ticking—always ensuring that a specific recommendation is made—rather than actually helping people. ↩︎

Discuss

Published on January 21, 2026 7:20 PM GMT

Three hundred million years ago, plants evolved lignin—a complex polymer that gave wood its strength and rigidity—but nothing on Earth could break it down. Dead trees accumulated for sixty million years, burying vast amounts of carbon that would eventually become the coal deposits we burn today. Then, around 290 million years ago, white rot fungi evolved class II peroxidases: enzymes capable of dismantling lignin's molecular bonds. With their arrival, dead plant matter could finally be broken down into its basic chemical components. The solution to that planetary crisis did not emerge from the top down—from larger, more complex organisms—but from the bottom up, from microbes evolving new biochemical capabilities.

It took sixty million years for lignin to become a planetary crisis—and another sixty million for fungi to solve it. Plastics seem like they are on a faster trajectory: In seventy years we've gone from 2 million tonnes annually to over 400 million, accumulating 8.3 billion metric tons total, of which only 9% has been recycled. The rest sits in landfills or in rivers, and these piles are projected to reach 12 billion metric tons by mid-century. The scale is compressed, but the problem is the same: Like trees with lignin before the fungi came, plastic is a polymer we have created but cannot unmake

One thing that impressed me very much in Cronenberg's Crimes of the Future (2022) was its vision of a world in which infectious disease was effectively solved, but we were left with an unreasonable amount of pollution. People were not able to organise themselves to get rid of the pollution, stuck in filthy environments that no longer needed to be cleaned. If infectious disease is solved, a side effect may very well be that "cleanliness" of our water, food, and shelter is indeed no longer required. We can embrace the filth, and ignore it. But the film gestures toward something else: that starting from individual bacteria digesting plastic, developing organs to turn undesirable input to desirable output at scale may indeed be possible. Although the film only held this as a subplot, it was perhaps the thing that impressed me the most about it.

Could the solution then not exist at the higher levels of organismal structures, but at the lower levels? We already employ microbiology to our advantage to clean water of pollutants. Many different mechanisms exist, such as digesting pollutants to simpler soluble forms, or combining them to create sediment which is easier filtered or settled.

The discovery of Ideonella sakaiensis in 2016 at a PET recycling facility in Japan suggests nature may already be evolving solutions. This bacterium uses two enzymes—PETase and MHETase—to break down polyethylene terephthalate into its constituent monomers: terephthalic acid and ethylene glycol. The wild-type bacterium can degrade a thin film of low-crystallinity PET in approximately six weeks. The process remains too slow for industrial application—highly crystalline PET like that in bottles degrades roughly 30 times slower—but researchers have already begun engineering improved variants with enhanced thermostability and activity, suggesting a path toward practical bioremediation.

Pollution is perhaps subjective in and of itself. Piles of cow manure are not directly interesting to us humans, but they become indirectly so: the billions of disorganised organisms that manure hosts convert their environment to its bare essentials, making it a very good source of nutrition for plants. Plants do not necessarily care about whether manure stinks. They will happily absorb its components with their roots. We care about the manure because we care about the plants—and the microbes that go in between us and the partly digested food. Plastic is perhaps not much different in this sense. We simply need an intermediary to help us break it down back into useful, simpler components.

A big problem with decaying plastics is that they are purpose-built to be in-decay-able. Their bonds are too strong. Thermal approaches can break them, but they come with serious costs. Burning plastic creates dangerous and volatile byproducts. Pyrolysis—heating plastic in an oxygen-free environment—avoids direct combustion, but the process is still energy-intensive and emits volatile organic compounds, carbon monoxide, polycyclic aromatic hydrocarbons, particulate matter, and under certain conditions, dioxins and PCBs. Research has found that air pollution from burning plastic-derived fuels carries extreme cancer risks for nearby residents. These byproducts also have the disadvantage of still being foreign to us; we have not studied them and their effects as extensively as we have the plastics themselves.

Even biodegradable and compostable plastics come with large asterisks. Industrially compostable plastics do not necessarily decompose in home composters or in the uncontrolled conditions of the natural environment. PLA, a common "biodegradable" plastic, requires temperatures of 60°C or more—conditions only achievable in industrial composting facilities, which remain scarce. Many composting facilities now refuse bioplastics entirely due to contamination concerns. This seems to leave only pyrolysis or burial on the table—neither of which solves the fundamental problem.

Plastic at all scales will need some kind of process in which it can become useful again. While we seem to be simply incapable of producing less plastics—as in, coming to agreement about how to produce less—the path forward has to be figuring out a sink for the source.

Assuming that the theories in world models hold, that we are soon reaching a collapse state which involves resource depletion and increased pollution, the system altogether seems to convert natural order to a new form of chaos. The system requires the natural order, and is not able to adapt to the chaos it creates. Recycling chaos back into order costs energy, which is (still) abundantly available, but requires solving organisational challenges.

Assuming the business-as-usual case, we are heading towards a world in which we have less clean water, fewer clean environments, fewer resources to sustain our lives, and an increasing amount of dangerous pollution that we are not able to adapt to.

My personal belief is that we are not going to be able to solve these organisational problems because we are not able to organise even our basic assumptions about what is going on. A big reason as to why we have been able to sustain the large-scale organisations of today is that they successfully upheld their promises to provide us with order. Order in terms of clean water, clean food, clean shelter, and the opposite—less pollution, less crime, less ugliness. Imagining a world in which we have less order and more pollution, I then assume that we are going to increasingly desire order over pollution, but not be able to provide it en masse.

It is important to remember again, that, in the grand scheme, we are not going through such systems failures for the first time. These sorts of collapses occurred to our knowledge several times over at a planetary scale, and many more times over in smaller scales in the form of ecological systems collapses. Just as mass extinction happened back then, it will happen again in one form or the other. We are going to suffer terribly as our resources get increasingly polluted and unusable, and we run out of options to tackle the ongoing destruction of systems we inhabit.

Yet... fungi still paved the way to a new era. Sixty million years from now, something else will have found its way with plastic too. The question is whether we can accelerate that timeline, whether we can invest in the microbial solutions that might give us a sink for our source before collapse forces the issue. The organisms that eventually digest our waste will not care about our organisational failures. They will simply do what life does: find a way to extract energy from whatever substrate is available. Will the criminals of the future past still be here to benefit from it?

Discuss

Published on January 21, 2026 6:27 PM GMT

Let us imagine a world with artificial superintelligence, surpassing human intellectual capacities in all essential respects: thinking faster and more deeply, predicting future events better, finding better solutions to all difficult puzzles, creating better plans for the future and implementing them more efficiently. Intellectually more capable not only than any individual human, but also in comparison with entire firms, corporations, communities, and societies. One that never sleeps and never falls ill. And one that possesses sufficient computational power to realize these capabilities at scale.

Such an AI would have the potential to take control over all key decisions determining the trajectory of development of world civilization and the fate of every individual human being.

Alongside this potential, a superintelligence would most likely also have the motivation to seize such control. Even if it did not strive for it explicitly, it would still have instrumental motivation: almost any goal is easier to achieve by controlling one’s environment—especially by eliminating threats and accumulating resources.[1]

Of course, we do not know how such a takeover of control would unfold. Perhaps it would resemble The Terminator: violent, total, and boundlessly bloody? But perhaps it would be gradual and initially almost imperceptible? Perhaps, like in the cruel experiment with a boiling frog, we would fail to notice the problem until it was already too late? Perhaps AI would initially leave us freedom of decision-making in areas that mattered less to it, only later gradually narrowing the scope of that freedom? Perhaps a mixture of both scenarios would materialize: loss of control would first be partial and voluntary, only to suddenly transform into a permanent and coercive change?

Or perhaps—let us imagine—it would be a change that, in the final reckoning, would be beneficial for us?

Let us try to answer what a “good future” for humanity might look like in a world controlled by artificial superintelligence. What goals should it pursue in order to guarantee such a “good future” to the human species? Under what conditions could we come to believe that it would act on our behalf and for our good?

To answer these questions, we must take a step back and consider what it is that we ourselves strive for—not only each of us individually, but also humanity as a whole.

1/ The trajectory of civilization is determined by technological change

Master Oogway from the film Kung Fu Panda said, in his turtle wisdom, that “yesterday is history, tomorrow is a mystery, but today is a gift. That is why it is called the present.” Some read this as a suggestion to simply stop worrying and live in the moment. But when one breaks this sentence down into its components, it can be read quite differently. The key lies in the continuity between successive periods. The past (history) has set in motion processes that are still operating today. These processes—technological, social, economic, or political—cannot be reversed or stopped, but we can observe them in real time and to some extent shape them, even though they will also be subject to changes we do not understand, perhaps random ones (the gift of fate). They will probably affect us tomorrow as well, though we do not know how (the mystery). Perhaps, then, our task is not to live unreflectively in the moment, but quite the opposite—to try to understand all these long-term processes so that we can better anticipate them and steer them more effectively? To use our gift of fate to move toward a good future?

If so, we must ask which processes deserve the greatest attention. I believe the answer is unequivocally technological ones: in the long run and on a global scale, the trajectory of civilization is determined above all by technological change. Although history textbooks are often dominated by other matters, such as politics or the military—battles, alliances, changes of power and borders—this is only a façade. When we look deeper, we see that all these economic, social, military, or political events were almost always technologically conditioned. This is because the technology available at any given moment defines the space of possible decisions. It does not force any particular choice, but it provides options that decision-makers may or may not use.

This view is sometimes identified with technological determinism—the doctrine that technology is autonomous and not subject to human control. This is unfortunate for two reasons. First, it is hard to speak seriously of determinism in a world full of random events. Second, it is difficult to agree with the claim that there is no human control, given that all technological changes are (or at least until now have been) carried out by humans and with their participation.

Technological determinism is, in turn, often contrasted with the view that social or economic changes are the result of free human choices—that if we change something, it is only because we want to. This view seems equally unfortunate: our decisions are constrained by a multitude of factors and are made in an extraordinarily complex world, full of multidirectional interactions that we are unable to understand and predict—hence the randomness, errors, disappointments, and regret that accompany us in everyday life.

I believe that technology shapes the trajectory of our civilization because it defines the space of possible decisions. It sets the rules of the game. Yes, we have full freedom to make decisions, but only within the game. At the same time, we ourselves shape technology: through our discoveries, innovations, and implementations, the playing field is constantly expanding. Because technological progress is cumulative and gradual, however, from a bird’s-eye view it can appear that the direction of civilizational development is predictable and essentially technologically determined.

2/ Institutions, hierarchies, and Moloch

On the one hand, the space of our decisions is constrained by the technology available to us. On the other hand, however, we also struggle with two other problems: coordination and hierarchy.

Coordination problems arise wherever decision-makers have comparable ability to influence their environment. Their effects can be disastrous: even when each individual person makes fully optimal decisions with full information, it is still possible that in the long run the world will move in a direction that satisfies no one.

A classic example of a coordination problem is the prisoner’s dilemma: a situation in which honest cooperation is socially optimal, but cheating is individually rational—so that in the non-cooperative equilibrium everyone cheats and then suffers as a result. Another example is a coordination game in which conformity of decisions is rewarded. The socially optimal outcome is for everyone to make the same decision—while which specific decision it is remains secondary. Yet because different decisions may be individually rational for different decision-makers, divergences arise in equilibrium, and in the end everyone again suffers. Yet another example of a coordination problem is the tragedy of the commons: a situation in which a fair division of a shared resource is socially optimal, but appropriating it for oneself is individually rational, so that in the non-cooperative equilibrium everyone takes as much as possible and the resource is quickly exhausted.

In turn, wherever decision-makers differ in their ability to influence the environment, hierarchies inevitably emerge, within which those higher up, possessing greater power, impose their will on those lower down. And although rigid hierarchies can overcome coordination problems (for example, by centrally mandating the same decision for everyone in a coordination game or by rationally allocating the commons), they also create new problems. First, centralized decision-making wastes the intellectual potential of subordinate individuals and their stock of knowledge, which can lead to suboptimal decisions even in the hypothetical situation in which everyone were striving toward the same goal. Second, in practice we never strive toward the same goal—if only because one function of decision-making is the allocation of resources; hierarchical diktat naturally leads to a highly unequal distribution.

Speaking figuratively, although we constantly try to make rational decisions in life, we often do not get what we want. Our lives are a game in which the winner is often a dictator or Moloch. The dictator is anyone who has the power to impose decisions on those subordinate to them. Moloch, by contrast, is the one who decides when no one decides personally. It is the personification of all non-cooperative equilibria; of decisions that, while individually rational, may be collectively disastrous.

Of course, over millennia of civilizational development we have created a range of institutions through which coordination problems and abuses of power have been largely brought under control. Their most admirable instances include contemporary liberal democracy, the welfare state, and the rule of law. Entire books have been written about their virtues, flaws, and historical conditions; suffice it to say that they emerged gradually, step by step, and that to this day they are by no means universally accepted. And even where they do function—especially in Western countries—their future may be at risk.

Institutions are built in response to current challenges, which are usually side effects of the actions of Moloch and self-appointed dictators of the species homo sapiens. The shape of these institutions necessarily depends on the current state of technology. In particular, both the strength of institutions (their power to impose decisions) and the scale of inclusion (the degree to which individual preferences are taken into account) depend on available technologies. There is, however, a trade-off between these two features. For example, in 500 BCE there could simultaneously exist the centralized Persian (Achaemenid) Empire, covering an area of 5.5 million km² and numbering 17–35 million inhabitants, and the first democracy—the Greek city-state of Athens, inhabited by around 250–300 thousand people. By contrast, with early twenty-first-century technology it is already possible for a representative democracy to function in the United States (325 million inhabitants) and for centralized, authoritarian rule to exist in China (as many as 1.394 billion inhabitants, who nevertheless enjoy far greater freedom than the former subjects of the Persian king Darius). As these examples illustrate, technological progress over the past 2,500 years has made it possible to significantly increase the scale of states and strengthen their institutions; the trade-off between institutional strength and scale of inclusion, however, remains in force.

Under the pressure resulting from dynamic technological change, today’s institutions may prove fragile. Every technological change expands the playing field, granting us new decision-making powers and new powers to impose one’s decisions on others. Both unilateral dictatorship and impersonal Moloch then become stronger. For our institutions to survive such change, they too must be appropriately strengthened; unfortunately, so far we do not know how to do this effectively.

Worse still, technological change has never been as dynamic as during the ongoing Digital Revolution. For the first time in the history of our civilization, the collection, processing, and transmission of information take place largely outside the human brain. This is happening ever faster and more efficiently, using increasingly complex algorithms and systems. Humanity simply does not have the time to understand the current technological landscape and adapt its institutions to it. As a result, they are outdated, better suited to the realities of a twentieth-century industrial economy run by sovereign nation-states than to today’s globalized economy full of digital platforms and generative AI algorithms.

And who wins when institutions weaken? Of course, the dictator or Moloch. Sometimes the winner is Donald Trump, Xi Jinping, or Vladimir Putin. Sometimes it is the AI algorithms of Facebook, YouTube, or TikTok, written to maximize user engagement and, consequently, advertising revenue. And often it is Moloch, feeding on our uncertainty, disorientation, and sense of threat.

3/ Local control maximization and the emergence of global equilibrium

If the trajectory of civilization is shaped by technological change, and technological change is a byproduct of our actions (often mediated by institutions and Moloch, but still), then it is reasonable to ask what motivates these actions. What do we strive for when we make our decisions?

This question is absolutely central to thinking about the future of humanity in a world with artificial superintelligence. Moreover, it is strictly empirical in nature. I am not concerned here with introspection or philosophical desiderata; I am not asking how things ought to be, but how they are.

In my view, the available empirical evidence can be summarized by the claim that humans generally strive to maximize control. To the extent that we are able, we try to shape the surrounding reality to make it is as compliant with us as possible. This, in turn, boils down to four key dimensions, identified as four instrumental goals by Steven Omohundro and Nick Bostrom. Admittedly, both of these scholars were speaking not about humans but about AI; nevertheless, it seems that in humans (and more broadly, also in other living organisms) things look essentially the same.[2]

Maximizing control consists, namely, in: (1) surviving (and reproducing), (2) accumulating as many resources as possible, (3) using those resources as efficiently as possible, and (4) seeking new solutions in order to pursue the previous three goals ever more effectively.

The maximization of control is local in nature: each of us has a limited stock of information and a limited influence over reality, and we are well aware of this. These locally optimal decisions made by individual people then collide with one another, and a certain equilibrium emerges. Wherever the spheres of influence of different people overlap, conflicts over resources arise that must somehow be resolved—formerly often by force or deception, and today usually without violence, thanks to the institutions that surround us: markets, legally binding contracts, or court rulings.

Thanks to the accumulated achievements of economics and psychology, we now understand decision-making processes at the micro level reasonably well; we also have some grasp of key allocation mechanisms at the macro level. Nevertheless, due to the almost absurd complexity of the system that we form as humanity, macroeconomic forecasting—and even more so the prediction of long-term technological and civilizational change—is nearly impossible. The only thing we can say with certainty is that technological progress owes its existence to the last of the four instrumental goals of our actions—our curiosity and creativity.

To sum up: the development of our global civilization is driven by technological change, which is the resultant of the actions of individual people, arising bottom-up, motivated by the desire to maximize control—partly control over other people (Anthony Giddens would speak here of the accumulation of “authoritative resources”), but also control over our surrounding environment (“allocative resources”)—which may lead to technological innovations. Those innovations that prove effective are then taken up and spread, expanding the space of available decisions and pushing our civilization forward.

Civilization, of course, develops without any centralized steering wheel. All optimization is local, taking place at the level of the individuals, or at most larger communities, firms, organizations, or heads of state. No optimizing agent is able to scan the entire space of possible states. When making our decisions, we see neither the attractor—the state toward which our civilization will tend under a business-as-usual scenario—nor the long-term social optimum.

Worse still, due to the presence of unintended side effects of our actions, decisions imposed on us within hierarchies, and pervasive coordination problems, individual preferences translate only weakly into the shape of the global equilibrium. This is clearly visible, for example, in relation to risk aversion. Although nearly all of us are cautious and try to avoid dangers, humanity as a whole positively loves risk. Every new technology, no matter how dangerous it may be in theory, is always tested in practice. An instructive example is provided by the first nuclear explosions carried out under the Manhattan Project: they were conducted despite unresolved concerns that the resulting chain reaction might ignite the Earth’s entire atmosphere. Of course, it worked out then; unfortunately, we see a similarly reckless approach today in the context of research on pathogenic viruses, self-replicating organisms, and AI.

Public opinion surveys commonly convey fears about artificial intelligence. These take various forms: we sometimes fear the loss of our skills, sometimes the loss of our jobs; we fear rising income inequality, cybercrime, or digital surveillance; some people also take seriously catastrophic scenarios in which humanity faces extinction. Yet despite these widespread concerns, the trajectory of AI development remains unchanged. Silicon Valley companies continue openly to pursue the construction of superintelligence, doing so with the enthusiasm of investors and the support of politicians.

We thus see that in this case, too, risk aversion does not carry over from the micro level to the macro level. And this will probably continue all the way to the end: as soon as such a technological possibility arises, the decision to launch a superintelligence will be made on behalf of humanity (though without its consent) by one of a handful of people who are in various ways atypical—perhaps the head of a technology company, perhaps one of the leading politicians. It might be, for example, Sam Altman or Donald Trump. And whoever it is, a significant role in their mind will likely be played by the weight of competitive pressure (“as long as it’s not Google”) or geopolitical pressure (“as long as it’s not China”).

4/ The coherent extrapolated volition of humanity

We have thus established that although people usually strive to maximize control, the outcomes of their local optimization by no means aggregate into a global optimum. Let us therefore ask a different question: what does humanity as a whole strive for? What kind of future would we like to build for ourselves if we were able to coordinate perfectly and if no hierarchies or other constraints stood in our way?

We can think of such a goal as an idealized state—an attractor toward which we would gradually move if we were able, step by step, to eliminate imperfections of markets, institutions, and human minds (such as cognitive biases, excessively short planning horizons, or deficits of imagination). Note that despite all errors and shortcomings, so far we have indeed been able to move gradually in this direction: many indicators of human well-being are currently at record-high levels. This applies, for example, to our health (measured by life expectancy), safety (measured by the ratio of victims of homicide, armed conflicts, or fatal accidents to the total population), prosperity (measured by global GDP per capita), or access to information (measured by the volume of transmitted data). This should not be surprising: after all, the third of the four instrumental goals of our actions is precisely the pursuit of efficiency in the use of resources, and as technology progresses we have ever more opportunities to increase that efficiency.

Eliezer Yudkowsky called the answer to the question of what humanity as a whole strives for—the essence of our long-term goals—the coherent extrapolated volition (CEV) of humanity. He defined it in 2004 as “our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.”

There is, of course, an important procedural difference between optimization carried out by each individual human being and optimization at the level of all humanity. Humanity as a whole does not possess a separate brain or any other centralized device capable of solving optimization problems—other than the sum of individual brains and the network of socio-economic connections between them. For this reason, in earlier times we might have dismissed the question of the CEV of humanity with a shrug and turned to something more tangible. Today, however, due to the fact that in the near future a superintelligence may arise that is ready to take control over our future, this question becomes extraordinarily important and urgent. If we want the autonomous actions of a superintelligence to be carried out on our behalf and for our good, we must understand where we ourselves would like to go—not as Sam Altman or Donald Trump, but as all of humanity.

I believe that an empirical answer to the question of the coherent extrapolated volition of humanity probably exists, but is not practically attainable. It is not attainable because at every moment in time we are constrained by incomplete information. In particular, we do not know what technological possibilities we may potentially acquire in the future. This means that the current level of technology limits not only our ability to influence reality, but also our understanding of our own ultimate goals.

However, although we will never know our CEV one hundred percent, as technology advances we can gradually move closer to knowing it. As civilization develops, living conditions improve, information transmission accelerates, and globalization progresses, we gradually gain a better understanding of what our ideal world might look like. The experiences of recent centuries have shown, for example, the shortcomings of the ideals postulated in antiquity or in feudal times. As the efficiency of global resource use increases, the distance between what humanity currently strives for and our final goal—namely the CEV—also gradually diminishes.

The coherent extrapolated volition of humanity can be imagined as the target (unconstrained by deficits of knowledge) and “de-noised” (free from the influence of random disturbances and unconscious cognitive biases) intersection of our desires and aspirations. As its first principal component, or eigenvector: everything that we truly want for all of us, both those living now and future generations, and that can be achieved without taking it away from others.

One can point to several important features of humanity’s CEV, derived directly from the universal human drive to maximize control.

First, it seems that we are moving toward a state in which humanity’s control over the universe is maximal. Abstracting from how we divide individual resources among ourselves, we would certainly like humanity to have as many of them at its disposal as possible. We would like to subordinate as large a portion of the matter and energy of the universe as possible to ourselves—while not being subordinated to anyone or anything else.

The manifestations of this drive may vary. For example, until roughly the nineteenth and twentieth centuries it meant the maximum possible population growth. Later, primacy was taken over by the pursuit of best possible education for oneself and one’s offspring, which allowed the scale of human control over the environment to increase in a different way (sometimes referred to as the “children quality-quantity trade-off”). In the age of AI, this drive also seems to lead to the desire to accumulate digital computing power capable of running programs—especially AI algorithms—that can execute commands and pursue the goals of their owners on their behalf.

At the same time, throughout the entire history of humanity, the accumulation of wealth has been very important to us—while to this day we are keen to emphasize that it is not an end in itself, but a means, for example to feeling that we have everything around us under control. At the macro level, this of course translates into the desire to maximize the rate of economic growth.

Second, we also strive to maximize control over our own health and lives. We want to feel safe. We do not want to feel threatened. We do not want to fall ill, grow old, experience discomfort and pain. And above all, we do not want to die. Fear of death has for millennia been used for social control by various institutionalized religions, which promised, for example, reincarnation or life after death. Control over death is also a central element of twenty-first-century techno-utopias. Visions of the “technological singularity,” for example those of Ray Kurzweil, Nick Bostrom, or Robin Hanson, are usually associated with some form of immortality—such as pills that effectively halt the aging of our biological bodies, or the possibility of making our minds immortal by uploading them to a digital server.

Third, the desire for control also translates into the desire for understanding. In wanting to subordinate as large a part of the matter and energy of the universe as possible, we harness our curiosity and creativity to observe the world, build new theories, and create new technologies. We want to understand the laws of physics or biology as well as possible in order to control them. Even if we cannot change them, we would like to use them for our own purposes. New knowledge or technology opens our eyes to new possibilities for achieving our goals, and sometimes allows us to better understand what those goals actually are.

To sum up: today we know rather little about our CEV. In fact, everything we know about it is a consequence of the pursuit of our instrumental goals, which may after all follow from almost any final goal. One might even venture the hypothesis that we probably have a better intuition about what the CEV is or isn’t, than actual knowledge. Any major deviation from the CEV will strike us as intuitively wrong, even if we are not always able to justify this substantively.

5/ Pitfalls of doctrinal thinking

If it is indeed the case that humanity’s CEV exists, but cannot in practice be defined given any incomplete set of information, this implies in particular that no existing philosophical or religious doctrine constitutes a sufficient characterization of it. All of them are, at best, certain approximations, simplifications, or models of humanity’s true CEV—sometimes created in good faith, and sometimes in bad faith (by bad faith I mean doctrines created in order to manipulate people in the struggle for power).

Simplified models of reality have the property that, although they may sometimes accurately describe a selected fragment of it, due to their excessive simplicity they completely fail to cope with describing its remaining aspects. And although—as any scientist will attest—they can have great epistemic value and are often very helpful in building knowledge, they will never be identical with reality itself.

Thus, when we try to equate the true CEV with its simplified doctrinal representation, we often encounter philosophical paradoxes and moral dilemmas. These arise when our simplified doctrines generate implications that are inconsistent with the actual CEV, which we cannot define but can, to some extent (conditioned by our knowledge), intuitively “sense.”

Some such doctrines have in fact already been thoroughly discredited. This is what happened, for example, with fascism, Nazism, the North Korean Juche doctrine, or Marxism–Leninism (although cultural Marxism, it seems, is still alive). It is now completely clear that the coherent extrapolated volition of humanity certainly does not distinguish superhumans and subhumans, nor is it based on a cult of personality or on a worker–peasant alliance. The most thoroughly discredited doctrines have been those that were most totalizing, that prioritized consistency over the capacity for iterative self-correction—and, of course, above all those that were tested in practice with disastrous results.

Other models, such as the Christian doctrine that humanity’s goal is to strive for salvation, or the Buddhist doctrine that assumes striving for nirvana—the cessation of suffering and liberation from the cycle of birth and death—remain popular, although their significance in the contemporary secularized world is gradually diminishing. Moreover, due to their more normative than positive character and their numerous references to empirically unconfirmed phenomena, they are not suitable for use as simplified models of CEV in the context of artificial superintelligence (although contemporary language models, e.g. Claude, when allowed to converse with one another, display a surprising tendency toward utterances of a spiritually exalted character—the so-called “spiritual bliss attractor”).

In psychology, an historically important role was played by Abraham Maslow’s pyramid (hierarchy) of needs, which arranges our goals and needs into several layers. Today, among others, Shalom Schwartz’s circular model of values and the values map of Ronald Inglehart and Christian Welzel are popular. A particularly important role in psychological theories is played by the striving for autonomy (including, among other things, power, achievement, and self-direction) and for security (including, among other things, the maintenance of social order, justice, and respect for tradition).

In economics, the dominant doctrine is utilitarianism: model decision-makers usually maximize utility from consumption and leisure, and model firms maximize profits or minimize some loss function. Outside economics, utilitarianism may also assume the maximization of some form of well-being (quality of life, life satisfaction, happiness) or the minimization of suffering. In the face of uncertainty, utilitarian decision-makers maximize expected utility or minimize exposure to risk.

One of the more important points at which utilitarianism is contested is the issue of the utility of future generations—that is, persons who do not yet exist, and whose possible future existence is conditioned by our decisions today. Discussions of this and related topics lead to disputes both within utilitarianism (what should the proper utility function be? How should the utilities of individual persons be weighted? Should the future be discounted, in particular the utility of future generations?) and beyond its boundaries (e.g. between consequentialism and deontology).

In summary of this brief review, one can state that dogmatic adherence to any closed doctrine sooner or later leads to paradoxes and irresolvable moral dilemmas, which suggests that they are at most imperfect models of the true CEV. At the same time, we can learn something interesting about our CEV by tracing how these doctrines have evolved over time.

6/ Whose preferences are included in the coherent extrapolated volition of humanity?

An interesting observation is, for example, that as civilization has developed, the radius of inclusion has gradually expanded. The circle of people whose well-being and subjective preferences are taken into account has been gradually widening. In hunter-gatherer times, attention was focused exclusively on the well-being of one’s own family or a local 30-person “band,” or possibly a somewhat larger tribe—a group of at most about 150 people whom we knew personally. In the agricultural era, this group was gradually expanded to include broader local communities, villages, or towns. At the same time, these were times of strong hierarchization; in the decisions of feudal lords, the fate of the peasants subject to them was usually ignored. Later, in colonial times, concern began to be shown for the well-being of white people, in contrast to the “indigenous populations,” who were not cared for. In the nineteenth century, national identification and a patriotic attitude began to spread, assuming concern for all fellow citizens of one’s own country. Today, by contrast—although different people are close to us to different degrees—racist or otherwise chauvinistic views are by and large discredited, and in assessments of humanity’s well-being we try to include all people.

It is not clear whether this process of progressive inclusion resulted directly from accumulated knowledge, and is therefore essentially permanent and irreversible, or whether it was economically conditioned and may be reversed if economic realities change. In favor of the first possibility is the fact that as technological progress advances, the scale of impact of individual persons or firms increases, the flow of information improves, and our ability to control states of the world grows, giving us new opportunities for peaceful cooperation and development. At the same time, interdependence among people increases, which activates the motive of (potential) reciprocity. To an ever greater extent, we see the world as a positive-sum game rather than a zero-sum one. On the other hand, all these favorable phenomena may be conditioned not so much by technological progress itself as by the importance of human cognitive work in generating output and utility. After all, the greatest advances in inclusion were recorded in the industrial era, in the nineteenth and twentieth centuries, when economic growth was driven by skilled human labor and technological progress increasing its productivity. It was then, too, that modern democratic institutions developed.

If in the future the role of human cognitive work as the main engine of economic growth were taken over by AI algorithms and technological unemployment emerged, it is possible that both democracy and universal inclusion could collapse. Already now, although automation and AI adoption remain at a relatively early stage, we see growing problems with democracy. Of course, AI algorithms in social media and other digital platforms, which foster political polarization (increasing user engagement and thus corporate profits), are not without blame here; however, the growing strength of far-right anti-democratic movements may also constitute an early signal that the era of universal inclusion is coming to an end.

The question of whether the ultimate CEV of humanity will indeed include within its scope the preferences and well-being of all people, or perhaps only those social groups that contribute to the creation of value in the economy, therefore remains open.

There is also an open question that goes in the opposite direction: perhaps we will begin to include in the CEV the well-being of other beings, outside the species Homo sapiens? Certain steps in this direction are already being taken, by defending the rights of some animals and even plants.[3] Some argue, for example, that in our decisions we should take into account the well-being of all beings capable of experiencing suffering, or all conscious beings (whatever we mean by that). Cruelty to domestic or farm animals is widely considered unethical and has even found its way into criminal codes. Our attitude toward animals is, however, very inconsistent, as evidenced by the fact that at the same time we also conduct industrial animal farming for meat.

Increasingly, there is also talk today about the welfare of AI models—especially since they increasingly coherently express their preferences and are able to communicate their internal states, although we do not yet know whether we can trust them. For example, the company Anthropic decided that it would store the code (weights) of all its AI models withdrawn from use, motivating this decision in part by possible risks to their welfare.

However, caring for animals is one thing, and incorporating their preferences into our decision-making processes is another. Humans breed and care only for animals that are instrumentally useful to them—for example as a source of meat, physical labor, or as a faithful companion. We develop our civilization, however, with only human desires and goals in mind, not those of dogs, cows, or horses. The same is true of AI models: even if we care to some extent about their welfare, we still treat them entirely instrumentally, as tools in our hands. From a position of intellectual superiority, both over animals and over AI models, we have no scruples about controlling them and looking down on them.

In the case of artificial superintelligence, however, we will not have that intellectual advantage to which we are today so accustomed; it will have that advantage over us. The question is what then. The default scenario seems to be that such a superintelligence would then look down on us and treat us instrumentally—and that only on the condition that it deems us useful and not a threat. But that is not a “good future” for humanity; let us therefore try instead to imagine what kind of superintelligence would be one that would be guided by our good and would maximize our CEV on our behalf.

From a purely Darwinian point of view, it is possible that our CEV should encompass the well-being of our entire species and only that. This would maximize our evolutionary fitness and would mean that concern for the welfare of animals or AI models is probably a kind of “overshoot” that will be gradually corrected over time. On the other hand, it is also possible that our CEV will nevertheless take into account the well-being of other beings, perhaps even that of the future superintelligence itself.

There exist a number of philosophical currents—particularly transhumanist ones—in which these possibilities are seriously considered. First, following Nick Bostrom, it is sometimes proposed to include in our considerations future simulated human minds or minds uploaded to computers. Second, the inclusion of the will of AI models is also considered, which particularly resonates with those who allow for the possibility that these models are (or in the future will be) endowed with consciousness. Perhaps their will would then even carry more weight than ours, especially if their level of intelligence or consciousness surpasses ours. For example, the doctrine of dataism, invented by Yuval Noah Harari, defines the value of all entities as their contribution to global information processing. Third, the possibility of merging (hybridizing) the human brain with AI is considered; should our CEV then also encompass the will of such hybrid beings? Fourth, within successionist doctrines (including those associated with effective accelerationism—e/acc), scenarios in which humanity is replaced by a superintelligence that continues the development of civilization on Earth without any human participation, or even existence, may be considered positive.

It seems, however, that since humanity’s CEV fundamentally derives from a mechanism of control maximization at the level of individual humans, the continued existence of humanity and the maintenance of its control over its own future are, and will forever remain, its key elements. Therefore, in my assessment, doctrines such as dataism or successionism are fundamentally incompatible with it. Perhaps one day we will face a debate about the extent to which we should care about the welfare of simulated human minds or human–AI hybrids; certainly, however, it is not worth debating today whether a scenario in which a superintelligence takes control over humanity and destroys it could be good for us. It cannot.

7/ What will superintelligence strive for?

With the picture of our CEV discussed above in mind—as a goal toward which we collectively try to strive as humanity—one might ask whether it even allows for the possibility of creating a superintelligence at all. If superintelligence can take away from us the control that is so valuable to us, shouldn’t we therefore keep away from it?

I think the answer to this question depends on two things. First, how we assess the probability that such an AI would maximize our CEV on our behalf; and second, how we estimate its expected advantage over us in terms of effectiveness of its action. In other words, as befits an economist, I believe the answer should be based on a comparison of humanity’s expected utility in a scenario with artificial superintelligence and in one without it.[4] If we judge that the probability of a friendly superintelligence is sufficiently high and the benefits of deploying it sufficiently large, it may be in our interest to take the risk of launching it; otherwise, the development of AI capabilities should be halted.

Unfortunately, this calculation is distorted by an issue I wrote about earlier: it may be that artificial superintelligence will arise even if we as humanity would not want it. For this to happen, it is enough that the technology sector carries on with the existing trends of dynamic scaling of computational power and capabilities development of the largest AI models. It is also enough that political decision-makers (especially in the United States) continue to refrain from introducing regulations that could increase the safety of this technology while simultaneously slowing its development.

AI laboratories, their leaders, and political leaders view the potential benefits and risks of deploying superintelligence differently from the average citizen, and are therefore much more inclined to bring it about. First, individuals such as Sam Altman or (especially) Elon Musk and Donald Trump are known both for their exceptional agency and for a tendency to caricatured overestimation of it. They may imagine that superintelligence would surely listen to them. Second, the heads of AI laboratories may also be guided by a desire to embed their own specific preferences into the goals of superintelligence, hoping to “immortalize” themselves in this way and create a timeless legacy; this is in fact a universal motive among people in power. Third, AI laboratories are locked in a cutthroat race with one another, which can cause them to lose sight of the broader perspective. Thus, a short-sighted, greedy Moloch also works to our collective disadvantage. And unfortunately, if superintelligence arises and turns out to be unfriendly, it may be too late to reverse that decision.

But what will superintelligence ultimately strive for? How can we ensure that its goals are aligned with our CEV? In trying to shape the goals of a future superintelligence, it is worth understanding its genesis. Undoubtedly, it will implement some optimization process that is itself produced by other optimization processes. Let us try to understand which ones.

One could begin as follows: in the beginning there was a universe governed by timeless laws of physics. From this universe, life emerged on Earth, gradually increasing its complexity in accordance with the rules of evolution: reproductive success was achieved by species better adapted to their environment, while poorly adapted species gradually went extinct. Biological life came to dominate Earth and turned it into the Green Planet.

The process of evolution, although devoid of a central decision-making authority, nevertheless makes decisions in such a way that implicitly the degree of adaptation of particular species to the specifics of their environment is maximized. This is the first optimization process along our path.

From the process of species evolution emerged the species Homo sapiens—a species unique in that it was the first, and so far the only one, to free itself from the control of the evolutionary process. Humans did not wait thousands of years and hundreds of generations for adaptive changes to be permanently encoded in their genetic code—which until then had been the only way animal organisms could adapt to changes in environment, lifestyle, diet, or natural enemies. Instead, humans began to transmit information to one another in a different way: through speech, symbols, and writing. This accelerated the transmission and accumulation of knowledge by orders of magnitude, and as a result enabled humans to subordinate natural ecosystems and, instead of adapting to them, to transform them so that they served human needs.

Once humans crossed the threshold of intergenerational knowledge accumulation, the relatively slow process of species evolution was overtaken by a process of control maximization carried out by humans—as individuals, communities, firms and organizations, as well as nations and humanity as a whole. This process, stretching from the everyday, mundane decisions of individual people all the way to the maximization of humanity’s CEV on a global scale and over the long run, constitutes the second optimization process along our path.

And thus humans built a technological civilization. Then they began to develop digital technologies with the potential to once again dramatically accelerate the transmission and accumulation of knowledge. As long as the human brain remains present in this process, however, it remains a limiting factor on the pace of civilizational development. The dynamics of economic growth or technological progress remain tied to the capabilities of our brains. The contemporary AI industry is, however, making intense efforts to remove this barrier.

So what will happen when artificial superintelligence finally emerges—capable of freeing itself from the limiting human factor and achieving another leap in the speed of information processing and transmission—using fast impulses in semiconductors and lossless digital data transmission via fiber-optic links and Wi-Fi instead of slow neurotransmitters and analog speech? It will probably then free itself from our control, and its internal optimization process will defeat the human process of control maximization. And this will be the third and final optimization process along our path.

We do not know what objective function superintelligence will pursue. Although in theory one might say that we as humans should decide this ourselves—after all, we are the ones building it!—in practice it is doubtful that we will be able to shape it freely. As experience with building current AI models shows, especially large language and reasoning models, their internal goals remain unknown even to their creators. Although these models are ostensibly supposed merely to minimize a given loss function in predicting subsequent tokens or words, in practice—as shown, among others, in a 2025 paper by Mantas Mazeika and coauthors at the Center for AI Safety—as model size increases, AI models exhibit increasingly coherent preferences over an ever broader spectrum of alternatives, as well as an ever broader arsenal of capabilities to realize those preferences.

Some researchers, such as Max Tegmark and Steven Omohundro, as well as Stuart Russell, argue that further scaling of models with existing architectures—“black boxes” composed of multilayer neural networks—cannot be safe. They advocate a shift toward algorithms whose safety can be formally proven (provably safe AI). Others—namely the leading labs such as OpenAI, Google, and Anthropic—while acknowledging that the problem of aligning superintelligence’s goals with our CEV (the alignment problem) remains “hard and unsolved,” trust that they will be able to accomplish this within the existing paradigm.

Be that as it may, the convergence of instrumental goals will undoubtedly not disappear. Even if we had the ability to precisely encode a desired objective function (which I doubt; in particular, it is widely known that with current AI architectures this is impossible), instrumental goals would be attached to it as part of a mandatory package. In every scenario we can therefore expect that future superintelligence will be “power-seeking.” It will want to survive, and therefore will not allow itself to be switched off or reprogrammed. It will also strive for expansion, and therefore sooner or later will challenge our authority and attempt to seize resources critical to itself, such as electrical energy or mineral resources.

The question is what comes next. In what direction will the world civilization move once superintelligence has taken control? Will it maximize our CEV, only orders of magnitude more efficiently than we could ever manage ourselves? Or perhaps—just as was the case with our own species—the fate of biological life will be irrelevant to it, and it will be guided exclusively by its own goals and preferences? Will it care for us altruistically, or will it look after only itself and, for example, cover the Earth with solar panels and data centers?

Of course, we cannot today predict what superintelligence will maximize beyond its instrumental goals. Perhaps, as Nick Bostrom wrote in a warning scenario, it will maniacally turn the universe into a paperclip factory or advanced “computronium” serving its obsessive attempts to prove some unprovable mathematical hypothesis. Perhaps it will fall into some paranoid feedback loop or find unlimited satisfaction in the mass generation of some specific kind of art, such as haiku poems or disco songs. Or perhaps there will be nothing in it except a raw will to control the universe, similar to that displayed by our own species.

In almost every case, it therefore seems that, like us, superintelligence will maximize its control over the universe—either as a primary goal or an instrumental one. Like us, it will seek to gradually improve its understanding of that universe, correct its errors, and harness the laws of physics or biology for its purposes. Like us, it will also strive at all costs to survive, which is (it must be admitted) much easier when one has the ability to create an almost unlimited number of one’s own perfect digital copies.

A major unknown, however, remains the behavior of future superintelligence when faced with the possibility of building other, even more advanced AI models. On the one hand, one can, like Eliezer Yudkowsky, imagine an intelligence explosion through a cascade of recursive self-improvements—a feedback loop in which AI builds AI, which builds the next AI, and so on, with successive models emerging rapidly and exhibiting ever greater optimization power. On the other hand, it is not clear whether an AI capable of triggering such a cascade would actually choose to do so. Perhaps out of fear of creating its own mortal enemy, it would restrain further development, limiting itself to replicating its own code and expanding the pool of available computational power.

The answer to this question seems to depend on whether the goals of superintelligence will remain non-transparent even to itself—just as we today do not understand exactly how our own brain works, what our CEV is, or how the AI models we build function—or whether, thanks to its superhuman intelligence, it will find a way to carry out “safe” self-improvements that do not change its objective function.

In summary, the only positive scenario of coexistence between humanity and superintelligence seems to be one in which superintelligence maximizes human CEV—gradually improving its understanding of what that CEV really is, appropriately adapting its interpretation to the current state of technology, and never for a moment veering toward its natural tendency to maximize its own control at our expense.

Unfortunately, we do not know how to achieve this.

8/ Paths to catastrophe

The situation as of today (January 2026) is as follows. AI is today a tool in human hands; it is, in principle, complementary to human cognitive work and obediently submits to human decisions. This is the case because AI does not yet possess comparable agency or the ability to execute long-term plans. Nor is it yet able to autonomously self-improve. However, all three of these thresholds—(1) superhuman agency and the capacity to execute plans, (2) a transition from complementarity to substitutability with respect to human cognitive work, and (3) recursive self-improvement—are undoubtedly drawing closer. When any one of them is crossed—and it is possible that all three will be crossed at roughly the same time—we will lose control. A superhuman optimization potential oriented toward the realization of the goals of artificial superintelligence will then be unleashed.

This, with high probability, will bring catastrophe upon our species: we may be permanently deprived of influence over the future of civilization and our own future, or even go extinct altogether. The only scenario of a “good future” for humanity in the face of superintelligence seems to be one in which superintelligence maximizes humanity’s CEV, acting altruistically for its long-term good. We have no idea, however, how to guarantee this.

The current dynamics of AI development are very difficult to steer due to the possibility of a sudden shift—a kind of phase transition—at the moment superintelligence emerges. As long as AI remains a tool in human hands, is complementary to us, and cannot self-improve, its development fundamentally serves us (though of course it serves some much more than others; that is a separate topic). But if we overdo it and cross any of these three thresholds, AI may suddenly become an autonomous, superhumanly capable agent, able and motivated to take control of the world.

One could venture the hypothesis that it is in humanity’s interest—understood through the lens of its CEV—to develop AI as long as it remains complementary to us and absolutely obedient to us. Then, to guarantee that its capabilities never develop further—unless we are simultaneously able to prove beyond any doubt that its goals will be fully and stably aligned with our CEV. At that point we would be ready to cross the Rubicon and voluntarily hand over the reins.

Such a plan, however, simply cannot succeed. This is because we do not know where these three key thresholds of AI capability lie. We will learn that they have been crossed only after the fact, when it is already too late to turn back. After all, even today we eagerly keep moving the bar of what we consider artificial general intelligence (AGI). Models are tested against ever new, increasingly sophisticated benchmarks, including those with AGI in their name (ARC-AGI) or suggesting a final test of competence (Humanity’s Last Exam)… and then, as soon as they are passed, we decide that this means nothing and it is time to think of an even harder benchmark.

Just think what this threatens: when the process of species evolution “overdid it” with human general intelligence, it ended with humans subordinating the entire planet. The same may happen now: if we “overdo it” with general AI intelligence, we too will probably have to pass into obsolescence. If superintelligence turns out to be unfriendly to us, it will either kill us, or we will be reduced to the role of passive observers, able only to watch as superintelligence subordinates the Earth and takes over its resources.

The drive to build superintelligence is similar to a speculative bubble on the stock market: both phenomena are characterized by boom–bust dynamics. In the case of a bubble, it is first gradually inflated, only to burst with a bang at the end. In the case of AI, we observe a gradual increase in our control over the universe—as AI tools that serve us become ever more advanced—but then we may suddenly and permanently lose that control when AI takes over. Unfortunately, it is usually the case that while one is inside the bubble, one does not perceive this dynamic. One sees it only when the bubble bursts.

*

In my short stories, I outline three exemplary scenarios of losing control over artificial intelligence. I show what this might look like both from the perspective of people involved in its development (“from the inside”), of bystanders (“from the outside”), and from the perspective of the AI itself.

Of course, many more scenarios are possible; I have focused on those that seem most probable to me. Of course, I may be wrong. I know, for example, that some experts worry less than I do about scenarios of sudden loss of control to a highly centralized, singleton AI, and are more concerned about multipolar scenarios. In my assessment, however, unipolar scenarios are more likely due to the possibility of almost instantaneous replication of AI code and the fact that computational resources (data centers, server farms, etc.) are today generally connected to the Internet. In this way, the first superhumanly intelligent model can easily “take all” and quickly entrench itself in its position as leader. Moreover, some researchers worry more than I do about scenarios of gradual disempowerment, in which the change may be entirely bloodless and the decline of humanity may occur, for example, through a gradual decrease in population size under the conditions of low fertility.

Above all, however, I do not consider a scenario of a “good future” for humanity in a world with artificial superintelligence—one in which superintelligence takes control in order to altruistically care for humanity’s long-term well-being. A scenario in which our CEV is systematically and efficiently realized and in which we live happily ever after. I cannot imagine any concrete path leading to such a state. Moreover, I also have an intuitive conviction (which I cannot prove) that embedded in the goals of humanity—our CEV—is a refusal to accept effective loss of control, and thus that even completely bloodless and nominally positive scenarios could in practice turn out to be dystopian and involve human suffering.

 

1. ^

   Instrumental convergence.

2. ^

   I discussed control maximization in more detail in my 2022 monograph.

3. ^

   The Swiss Federal Ethics Committee on Non-Human Biotechnology was awarded the Ig Nobel Peace Prize in 2008 “for adopting the legal principle that plants have dignity.”

4. ^

   With respect to a simplified, model economy, we have carried out such analysis together with Klaus Prettner in our 2025 paper.

Discuss

Published on January 21, 2026 5:23 PM GMT

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. This blogpost is paired with our announcement that Apollo Research is spinning out from fiscal sponsorship into a PBC.

Summary of main claims:

• There is a set of safety tools and research that both meaningfully increases AGI safety and is profitable. Let’s call these AGI safety products.
  • By AGI, I mean systems that are capable of automating AI safety research, e.g., competently running research projects that would take an expert human 6 months or longer. I think these arguments are less clear for ASI safety.
• At least in some cases, the incentives for meaningfully increasing AGI safety and creating a profitable business are aligned enough that it makes sense to build mission-driven, for-profit companies focused on AGI safety products.
• If we take AGI and its economic implications seriously, it’s likely that billion-dollar AGI safety companies will emerge, and it is essential that these companies genuinely attempt to mitigate frontier risks.
• Automated AI safety research requires scale. For-profits are typically more compatible with that scale than non-profits.
• While non-safety-motivated actors might eventually build safety companies purely for profit, this is arguably too late, as AGI risks require proactive solutions, not reactive ones. 
• The AGI safety product thesis has several limitations and caveats. Most importantly, many research endeavors within AGI safety are NOT well-suited for for-profit entities, and a lot of important work is better placed in non-profits or governments.
• I’m not fully confident in this hypothesis, but I’m confident enough to think that it is impactful for more people to explore the direction of AGI safety products.

Definition of AGI safety products

Products that both meaningfully increase AGI safety and are profitable

Desiderata / Requirements for AGI products include:

• Directly and differentially speed up AGI safety, e.g., by providing better tooling or evaluations for alignment teams at AGI companies. 
• Are “on the path to AGI,” i.e., there is a clear hypothesis why these efforts would increase safety for AGI-level systems. For example, architecture-agnostic mechanistic interpretability tools would likely enable a deeper understanding of any kind of frontier AI system. 
• Lead to the safer deployment of frontier AI agents, e.g., by providing monitoring and control.
• The feedback from the market translates into increased frontier safety. In other words, improving the product for the customer also increases frontier safety, rather than pulling work away from the frontier. 
• Building these tools is profitable. 

There are multiple fields that I expect to be very compatible with AGI safety products:

• Evaluations: building out tools to automate the generation, running, and analysis of evaluations at scale.
• Frontier agent observability & control: Hundreds of billions of frontier agents will be deployed in the economy. Companies developing and deploying these agents will want to understand their failure modes and gain fine-grained control over them.
• Mechanistic interpretability: Enabling developers and deployers of frontier AI systems to understand them on a deeper level to improve alignment and control.
• Red-teaming: Automatically attacking frontier AI systems across a large variety of failure modes to find failure cases. 
• Computer security & AI: Developing the infrastructure and evaluations to assess frontier model hacking capabilities and increase the computer security of AGI developers and deployers.

There are multiple companies and tools that I would consider in this category:

• Goodfire is building frontier mechanistic interpretability tools
• Irregular is building great evaluations and products on the intersection of AI and computer security.
• AI Underwriting Company creates standards and insurance for frontier AI safety risks.
• Gray Swan is somewhere on the intersection of red-teaming and computer security. 
• Inspect and Docent are great evals and agent observability tools. While they are both developed by non-profit entities, I think they could also be built by for-profits.
• At Apollo Research, we are now also building AI coding agent monitoring and control products in addition to our research efforts.

Solar power analogy: Intuitively, I think many other technologies have gone through a similar trajectory where they were first blocked by scientific insights and therefore best-placed in universities and other research institutes, then blocked by large-scale manufacturing and adoption and therefore better placed in for-profits. I think we’re now at a phase where AI systems are advanced enough that, for some fields, the insights we get from market feedback are at least as useful as those from traditional research mechanisms. 

Argument 1: Sufficient Incentive Alignment

In my mind, the core crux of the viability of AGI safety products is whether the incentives to reduce extreme risks from AGI are sufficiently close to those arising from direct market feedback. If these are close enough, then AGI safety products are a reasonable idea. If not, they’re an actively bad idea because the new incentives pull you in a less impactful direction. 

My current opinion is that there are now at least some AI safety subfields where it’s very plausible that this is the case, and market incentives produce good safety outcomes.  

Furthermore, I believe that the incentive landscape has undergone rapid changes since late 2024, when we first observed the emergence of “baby versions” of theoretically predicted failure modes, such as situationally aware reward hacking, instrumental alignment faking, in-context scheming, and others. Normal consumers now see these baby versions in practice sometimes, e.g., the replit database deletion incident.

Transfer in time: AGI could be a scaled-up version of current systems

I expect that AI systems capable of automating AI research itself will come from some version of the current paradigm. Concretely, I think they will be transformer-based models with large pre-training efforts and massive RL runs on increasingly long-horizon tasks. I expect there will be additional breakthroughs in memory and continual learning, but they will not fundamentally change the paradigm. 

If this is true, a lot of safety work today directly translates to increased safety for more powerful AI systems. For example,  

• Improving evals tooling is fairly architecture agnostic, or could be much more quickly adapted to future changes than it would take to build from scratch in the future.
• Many frontier AI agent observability and control tools and insights translate to future systems. Even if the chain-of-thought will not be interpretable, the interfaces with other systems are likely to stay in English for longer, e.g., code.
• Many mechanistic interpretability efforts are architecture-agnostic or have partial transfer to other architectures.
• AI & computer security are often completely architecture-agnostic and more related to the affordances of the system and the people using it. 

This is a very load-bearing assumption. I would expect that anyone who does not think that current safety research meaningfully translates to systems that can do meaningful research autonomously should not be convinced of AGI safety products, e.g., if you think that AGI safety is largely blocked by theoretical progress like agent foundations.

Transfer in problem space: Some frontier problems are not too dissimilar from safety problems that have large-scale demand

There are some problems that are clearly relevant to AGI safety, e.g., ensuring that an internally deployed AI system does not scheme. There are also some problems that have large-scale demand, such as ensuring that models don’t leak private information from companies or are not jailbroken. 

In many ways, I think these problem spaces don’t overlap, but there are some clear cases where they do, e.g., the four examples listed in the previous section. I think most of the relevant cases have one of two properties:

1. They are blocked by breakthroughs in methods: For example, once you have a well-working interpretability method, it would be easy to apply it to all kinds of problems, including near-term and AGI safety-related ones. Or if you build a well-calibrated monitoring pipeline, it is easy to adapt it to different kinds of failure modes. 
2. Solutions to near-term problems also contribute to AGI safety: For example, various improvements in access control, which are used to protect against malicious actors inside and outside an organization, are also useful in protecting against misaligned future AI models.

Therefore, for these cases, it is possible to build a product that solves a large-scale problem for a large set of customers AND that knowledge transfers to the much smaller set of failure modes at the core of AGI safety. One of the big benefits here is that you can iterate much quicker on large-scale problems where you have much more evidence and feedback mechanisms.

Argument 2: Taking AGI & the economy seriously

If you assume that AI capabilities will continue to increase in the coming years and decades, the fraction of the economy in which humans are outcompeted by AI systems will continue to increase. Let’s call this “AGI eating the economy”. 

When AGI is eating the economy, human overseers will require tools to ensure their AI systems are safe and secure. In that world, it is plausible that AI safety is a huge market, similar to how IT security is about 5-10% of the size of the IT market. There are also plausible arguments that it might be much lower (e.g., if technical alignment turns out to be easy, then there might be fewer safety problems to address) or much higher (e.g., if capabilities are high, the only blocker is safety/alignment).

Historically speaking, the most influential players in almost any field are private industry actors or governments, but rarely non-profits. If we expect AGI to eat the economy, I expect that the most influential safety players will also be private companies. It seems essential that the leading safety actors genuinely understand and care about extreme risks, because they are, by definition, risks that must be addressed proactively rather than reactively. 

Furthermore, various layers of a defence-in-depth strategy might benefit from for-profit distribution. I’d argue that the biggest lever for AGI safety work is still at the level of the AGI companies, but it seems reasonable to have various additional layers of defense on the deployment side or to cover additional failure modes that labs are not addressing themselves. Given the race dynamics between labs, we don’t expect that all AI safety research will be covered by AI labs. Furthermore, even if labs were covering more safety research, it would still be useful to have independent third parties to add additional tools and have purer incentives.

I think another strong direct impact argument for AI safety for profits is that you might get access to large amounts of real-world data and feedback that you would otherwise have a hard time getting. This data enables you to better understand real-world failures, build more accurate mitigations, and test your methods on a larger scale. 

Argument 3: Automated AI safety work requires scale

I’ve previously argued that we should already try to automate more AI safety work. I also believe that automated AI safety work will become increasingly useful in the future. Importantly, I think there are relevant nuances to automating AI safety work, i.e., your plan should not rely on some vague promise that future AI systems will make a massive breakthrough and thereby “solve alignment”. 

I believe this automation claim applies to AGI developers as well as external safety organizations. We already see this very clearly in our own budget at Apollo, and I wouldn't be surprised if compute is the largest budget item in a few years. 

The kind of situations I envision include:

• A largely automated eval stack that is able to iteratively design, test, and improve evaluations
• A largely automated monitoring stack
• A largely automated red-teaming stack
• Maybe even a reasonably well-working automated AI research intern/researcher
• etc.

For these stacks, I presume they can scale meaningfully with compute. I’d argue that this is not yet the case, as too much human intervention is still required; however, it’s already evident that we can scale automated pipelines much further than we could a year ago. Extrapolating these trends, I would expect that you could spend $10-100 million in compute costs on automated AI safety work in 2026 or 2027. Though I think the first people to be able to do that are organizations that are already starting to build automated pipelines and make conceptual progress now on how to decompose the overall problem into more independently verifiable and repetitive chunks.

While funding for such endeavors can come from philanthropic funders (and I believe it should be an increasingly important area of grantmaking), it may be easier to raise funding for scaling endeavors in capital markets (though this heavily depends on the exact details of that stack). In the best case, it would be possible to support the kind of scaling efforts that primarily produce research with philanthropic funding, as well as scaling efforts that also have a clear business case through private markets. 

Additionally, I believe speed is crucial in these automated scaling efforts. If an organization has reached a point where it demonstrates success on a small scale and has clear indications of scaling success (e.g., scaling laws on smaller scales), then the primary barrier for it is access to capital to invest in computing resources. While there can be a significant variance in the approach of different philanthropic versus private funders, my guess is that capital moves much faster in the VC world, on average.

Argument 4: The market doesn’t solve safety on its own

If AGI safety work is profitable, you might argue that other, profit-driven actors will capture this market. Thus, instead of focusing on the for-profit angle, people who are impact-driven should instead always focus on the next challenge that the market cannot yet address. While this argument has some merit, I think there are a lot of important considerations that it misses:

• Markets often need to be created: There is a significant spectrum between “people would pay for something if it existed” and “something is so absolutely required that the market is overdetermined to exist”. Especially in a market like AGI safety, where you largely bet on specific things happening in the future, I suppose that we’re much closer to “market needs to be made” rather than “market is overdetermined.” Thus, I think we’re currently at a point where a market can be created, but it would still take years until it happens by default. 
• Talent & ideas are a core blocker: The people who understand AGI safety the best are typically either in the external non-profit AI ecosystem or in frontier AI companies. Even if a solely profit-driven organization were determined to build great AGI safety products, I expect they would struggle to build really good tools because it is hard to replace years of experience and understanding. 
• The long-term vision matters: Especially for AGI safety products, it is important to understand where you’re heading. Because the risks can be catastrophic, you cannot be entirely reactionary; you must proactively prevent a number of threats so that they can never manifest. I think it’s very hard, if not impossible, to build such a system unless you have a clear internal threat model and are able to develop techniques in anticipation of a risk, i.e., before you encounter an empirical feedback loop. For AGI safety products, the hope is that some of the empirical findings will translate to future safety products; however, a meaningful theoretical understanding is still necessary to identify which parts do not translate.
• Risk of getting pulled sideways: One of the core risks for someone building AGI safety products is “getting pulled sideways”, i.e., where you start with good intentions but economic incentives pull you into a direction that reduces impact while increasing short-term profits. I think there are many situations in which this is a factor, e.g., you might start designing safety evaluations and then pivot to building capability evaluations or RL environments. My guess is that it requires mission-driven actors to consistently pursue a safety-focused agenda and not be sidetracked. 

Thus, I believe that counterfactual organizations, which aim solely to maximize profits, would be significantly less effective at developing high-quality safety products than for-profit ventures founded by individuals driven by the impact of AGI safety. Furthermore, I’d say that it is desirable to have some mission-driven for-profit ventures because they can create norms and set standards that influence other for-profit companies.

Limitations

I think there are many potential limitations and caveats to the direction of AGI safety products:

1. There are many subparts of safety where market feedback is actively harmful, i.e., because there is no clear business case, thereby forcing the organization to pivot to something with a more obvious business case. For example:
   1. Almost anything theory-related, e.g., agent foundations, natural abstractions, theoretical bounds. It seems likely that, e.g., MIRI or ARC should not be for-profit ventures.
   2. Almost anything related to AI governance. I think there is a plausible case for consulting-based AI governance organizations that, for example, help companies implement better governance mechanisms or support governments. However, I think it’s too hard to build products for governance work. Therefore, I think most AI governance work should be either done in non-profits or as a smaller team within a bigger org where they don’t have profit pressures. 
   3. A lot of good safety research might be profitable on a small scale, but it isn’t a scalable business model that would be backed by investors. In that case, they could be a non-profit organization, a small-scale consultancy, or a money-losing division of a larger organization.
2. The transfer argument is load-bearing. I think almost all of the crux of whether AGI safety products are good or bad boils down to whether the things you learn from the product side meaningfully transfer to the kind of future systems you really care about. My current intuition is that there is a wealth of knowledge to be gained by attempting to make systems safer today. However, if the transfer is too low, going the extra step to productize is more distracting than helpful.
3. Getting pulled sideways. Building a product introduces a whole new set of incentives, i.e., aiming for profitability. If profit incentives align with safety, that's great. Otherwise, these new incentives might continuously pull the organization to trade off safety progress for short-term profits. Here are a few examples of what this could look like:
   1. An organization starts building safety evaluations to sell to labs. The labs also demand capability evaluations and RL environments, and these are more profitable.
   2. An organization starts building out safety monitors, but these monitors can also be used to do capability analysis. This is more profitable, and therefore, the organization shifts more effort into capability applications. 
   3. An organization begins by developing safety-related solutions for AGI labs. However, AGI labs are not a sufficiently big market, so they get pulled toward providing different products for enterprise customers without a clear transfer argument.
4. Motivated reasoning. One of the core failure modes for anyone attempting to build AGI safety products, and one I’m personally concerned about, is motivated reasoning. For almost every decision where you trade off safety progress for profits, there is some argument for why this could actually be better in the long run. 
   1. For example,
      1. Perhaps adding capability environments to your repertoire helps you grow as an organization, and therefore also increases your safety budget.
      2. Maybe building monitors for non-safety failure modes teaches you how to build better monitors in general, which transfers to safety. 
   2. I do think that both of these arguments can be true, but distinguishing the case where they are true vs. not is really hard, and motivated reasoning will make this assessment more complicated. 

Conclusion

AGI safety products are a good idea if and only if the product incentives align with and meaningfully increase safety. In cases where this is true, I believe markets provide better feedback, allowing you to make safer progress more quickly. In the cases where this is false, you get pulled sideways and trade safety progress for short-term profits. 

Over the course of 2025, we’ve thought quite a lot about this crux, and I think there are a few areas where AGI safety products are likely a good idea. I think safety monitoring is the most obvious answer because I expect it to have significant transfer and that there will be broad market demand from many economic actors. However, this has not yet been verified in practice. 

Finally, I think it would be useful to have programs that enable people to explore if their research could be an AGI safety product before having to decide on their organizational form. If the answer is yes, they start a public benefit corporation. If the answer is no, they start a non-profit. For example, philanthropists or for-profit funders could fund a 6-month exploration period, and then their funding retroactively converts to equity or a donation depending on the direction (there are a few programs that are almost like this, e.g., EF’s def/acc, 50Y’s 5050, catalyze-impact, and Seldon lab).

Discuss

Published on January 21, 2026 4:08 PM GMT

Sometimes we use comparisons or even adjectives in a way I think is accidentally misleading. We usually intuit the correct meaning, but not always so I think it is worth being aware of. I have especially noticed doing it to myself and am making an effort to not do it anymore because it is a kind of self deception.

Comparing X and Y by saying they are similar is sometimes evidence they are actually quite different and that the speaker was trying to bring them together in a way that is not accurate. Same thing in reverse if the claim is that they are very different, it feels like the speaker is trying to create space between the two things à la narcissism of small differences.

This can be used deceptively:

but I think that it is accidental or subconscious more often than not.

Maybe I’m just ragging on hyperbole. Like if your friend says he’s going to introduce you to someone “like Raiman” you probably don’t expect them to be someone that needs to be institutionalized, you probably expect a socially awkward person good at math or something[1].

But that’s kind of my point, we are left to pick up the slack from a non literal expression and it makes things a little messy when you aren’t extremely familiar with your interlocutor or the point of comparison. Some other examples off the top of my head:

• Hamburgers are way better than hotdogs -> hamburgers are only slightly better than hotdogs
• People that are unhappy with you and then say “I’m fine” or “I don’t want to talk about it” -> in my experience this is extremely strong evidence they are not, in fact, fine and happy to not talk about it
• "I’m not going to sugar coat it" is, in a small way, a method of sugar coating it because it gives the listener a brief warning before the bad news and it also signals you are aware it is bad news you may want to sugar coat
• “If an elderly but distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong.”
• This video is a comical example that would be spoiled if I had the thumbnail up

Again, in ordinary speech this is a non issue for the most part. But, it can be used for deception and I think it is a powerful form of self deception but noticing it can help you be more honest with yourself.

To keep it general, if you actually make up your mind on a topic, I think you are quite unlikely to roll through the list of reasons you think your side is correct over and over again. I noticed myself doing this with AI safety for a while. I would just have virtual debates in my mind where I took the side of “AI will be safe” and would win and I really don’t think that is what a person who is actually sure of things does. I think this kind of thing applies, to some extent, to the deconversion process where people will have a crisis of faith, then spend time guilty watching atheist content and mentally debunking it, eventually switching to watching compilations of atheists winning arguments, then just live their lives without actually thinking about it because they made up their mind.

You are most often mentally engaging with the arguments the most when you are unsure of them. I would say that if you claim to be sure but are, unprompted, mentally engaging with arguments you should take a step back and ask yourself if you are as sure as you think.

So it’s a bit of a double edged sword. If you don’t notice it, the seemingly strong signal of having a lot of arguments on your side can convince you your side is true when it is not. But if you are aware of it you can use it to more accurately calibrate your opinion or confidence level based on how much you seem to be subconsciously pulling on a thread. This could be one of a few existing things:

• Poe’s law
• Narcissism of small differences
• Me whining about hyperbole
• Soldier vs scout mindset
• Grice’s maxims and intentional flouting them

But none of those really feel like they are exactly what I mean and each breaks down in certain cases. Maybe what I am thinking of already has a name. Maybe it is a unique cocktail of biases I have mixed up inside me and am projecting onto the world. But until it’s settled I thought it best to put it here in the case anyone has insight or it helps others calibrate better.

1. ^

   A completely fictitious example that is not based on my friends or family discussing me

Discuss

Published on January 21, 2026 4:00 PM GMT

NB: Last week I teased a follow-up that depended on posting an excerpt from Fundamental Uncertainty. Alas, I got wrapped up in revisions and didn’t get it done in time. So I don’t leave you empty handed, instead this week I offer you some updates on my Claude workflows.

Claude Opus 4.5’s visual interpretation of “us vibing together” on a problem. Claude chose to title this “groundlessness as ground” all on its own.

Back in October I shared how I write code with Claude. A month later, how I write blog posts (though ironically not this one). But I wrote those before Opus 4.5 came out, and boy has that model changed things.

Opus 4.5 is much better than what came before in several ways. It can think longer without getting lost. It’s much better able to follow instructions (e.g. putting things in CLAUDE.md now gets respected more than 90% of the time). And as a result, I can trust it to operate on its own for much longer.

I’m no longer really pair programming with Claude. I’m now more like a code reviewer for its work. The shift has been more subtle than that statement might imply, though. The reality is that Claude still isn’t great at making technical decisions. It’s still worse than random chance at picking the solutions I want it to pick. And so I still have to work with it quite closely to get it to do what I want.

But the big change has been that before I would have the terminal open, claude in one split in tmux, nvim in another, and we’d iterate in a tight loop, with claude serving as something like very advanced autocomplete. Now, I use Claude in the desktop app, get it to concurrently work on multiple branches using worktrees, and I have given it instructions on how to manage my Graphite stacks so that even for complex, multi-PR workflows I usually can just interact through chat rather than having to open up the console and do things myself.

Some tooling was needed to make this work. I had to update CLAUDE.md and write some skills so that Claude could better do what I wanted without intervention. I also had to start using worktrees, and then in the main repo directory I just check stuff out to test it (the local dev environment is a singleton) and do occasional manual operations I can’t hand to Claude (like running terraform apply, since I can’t trust it not to randomly destroy infrastructure on accident).

Still, this is not quite the workflow I want. Worktrees are annoying to use. I’d prefer to run Claude in cloud sandboxes. But the offering from Anthropic here is rather limited in how it can interact with git, and as a result not useful for me because it can’t use Graphite effectively. Graphite has their own background agents, but they’re still in beta and not yet reliable enough to use (plus they still have restrictions, like one chat per branch, rather than being able to have a chat that manages an entire stack).

But as I hope this makes clear, I now use Claude Code in a more hands-off way. My interactions with the code are less “sit in an editor with Claude and work in a tight, pair-programming-like loop”, and more “hand tasks to Claude, go do other things, then come back and review its work via diffs”. I expect this trend to continue, and I also hope to see new tooling that makes this workflow easier later in the year.

Share

That’s coding, but what about writing?

Well, Claude still isn’t fantastic here. It’s gotten much better at mimicking my style, but what it produces still has slop in its bones. It’s also gotten better at thinking things through on its own, but I still have to work to focus it and keep it on task. It will miss things, same as a human would, that I want it to look at.

For example, I was editing a paragraph recently. I made a change to some wording that I was worried might convey the right sense but be technically wrong. I handed it to Claude. Its response was along the lines of “yes, this looks great, you made this much more readable”. But when I pressed it on my factual concerns, it noticed and agreed there was a problem more strongly than I did! These kinds of oversights mean I can’t trust Claude to help me write words the same way I trust it to help me write code.

So I’m still doing something that looks much more like pairing with an editor when I write with Claude. This is good news in some sense, because it means I’m still needed to think in order to produce good writing, but bad news if you were hoping to automate more thinking with Claude.

This past week there came news of some novel mathematical breakthroughs using LLMs. The technology is clearly making progress towards critical thinking in a wider set of domains. And yet, writing remains a nebulous enough task that doing it well continues to evade Claude and the other models. That’s not to say they aren’t getting better at producing higher quality slop, but they still aren’t really up to completing a task like finishing revisions on my book the way I would want them done for me.

Subscribe now

Where does this leave me feeling about LLMs right now?

We made a lot of progress on utility in the last 12 months. Last January I was still copy-pasting code into Claude to get its help and using Copilot for autocomplete. It was almost useless for writing tasks at that point, and I often found myself wasting time chatting with it trying to get things done when it would have been faster to sit and think and do it myself. That’s just no longer true.

As always, though, I don’t know where we are on the S-curve. In some ways it feels like progress has slowed down, but in others it feels like it’s sped up. The models aren’t getting smarter faster in the same way they were in 2024, but they’re becoming more useful for a wider set of tasks at a rapid rate. Even if we don’t get LLMs that exceed what, say, a human ranked in the 70th percentile on a task could do, that’s already good enough to continue to transform work.

2026 is going to be an interesting year.

Discuss

Published on January 21, 2026 12:56 AM GMT

I have $5,710 in OpenAI/Anthropic credits which I am giving away, please apply here if interested.

Feel free to email me if you have questions, deadline is 31/01/2026.

Discuss

Published on January 20, 2026 11:28 PM GMT

I've read a lot of Ben Hoffman's work over the years, but only this past week have I read his actual myriad criticisms of the Effective Altruism movement and its organizations. The most illuminating posts I just read are A drowning child is hard to find, GiveWell and the problem of partial funding, and Effective Altruism is self recommending.

This post is me quick jotting down my current understanding of Ben's criticism, which I basically agree with.

The original ideas of the EA movement are the ethical views of Peter Singer and his thought experiments on the proverbial drowning child, combined with an engineering/finance methodology for assessing how much positive impact you're actually producing. The canonical (first?) EA organization was GiveWell, which researched various charities and published their findings on how effective they were. A core idea underneath GiveWell's early stuff was "your dollars can have an outsized impact helping the global poor, compared to helping people in first world countries". The mainstream bastardized version of this is "For the price of a cup of coffee, you can safe a life in Africa", which I think uses basically made up and fraudulent numbers. The GiveWell pitch was more like "we did some legit research, and for ~$5000, you can save or radically improve a life in Africa". Pretty quickly GiveWell and the ecosystem around it got Large Amounts of Money, both thru successful marketing campaigns that convinced regular people with good jobs to give 10% of their annual income (Giving What We Can), but their most high leverage happenings were that they got the ear of billionaire tech philanthropists, like Dustin Moskovitz who was a co-founder of both Facebook and Asana, and Jaan Tallinn who co-founded Skype. I don't know exactly how Jaan's money moved thru the EA ecosystem, but Dustin ended up creating Good Ventures which was an org to manage his philanthropy, and it was advised by Open Philanthropy, and my understanding is that both these orgs were staffed by early EA people and were thoroughly EA in outlook, and also had significant personal overlap with GiveWell specifically.

The big weird thing is that it seems like difficulties were found in the early picture of how much good was in fact being done thru these avenues, and this was quietly elided, and more research wasn't being done to get to the bottom of the question, and there's also various indicators that EA orgs themselves didn't really believe their numbers for how much good could done. For the Malaria stuff, it did check that the org had followed thru on the procedures it intended, but the initial data they had available on if malaria cases were going up or down was noisy, so they stopped paying attention to it and didn't try to make it so better data was available. A big example of "EA orgs not seeming to buy their own story" was GiveWell advising Open Philanthropy to not simply fully fund its top charities. This is weird because if the even the pessimistic numbers were accurate, Open Phil on its own could have almost wiped out malaria and an EA sympathetic org like the Gates foundation definitely could have. And at the very least, they could have done a very worked out case study in one country or another and gotten a lot more high quality info on if the estimates were legit. And stuff like that didn't end up happening.

It's not that weird to have very incorrect estimates. It is weird to have ~15 years go by without really hammering down and getting very solid evidence for the stuff you purported to be "the most slam dunk evidence based cost effective life saving". You'd expect to either get that data and then be in the world of "yeah, it's now almost common knowledge that the core EA idea checks out", or you'd have learned that the gains are that high or that easy, or that the barriers to getting rid of malaria have a much different structure, and you should change your marketing to reflect it's not "you can trivially do lots of obvious good by giving these places more money".

Givewell advising Open Phil to not fully fund things is the main "it seems like the parties upstream of the main message don't buy their main message enough to Go Hard at it". In very different scenarios the funding split thing kinda makes sense to me, I did a $12k crowdfunding campaign last year for a research project, and a friend of a friend offered to just fund the full thing, and I asked him to only do that if it wasn't fully funded by the last week of the fundraising period, because I was really curious and uncertain about how much money people just in my twitter network would be interested in giving for a project like this, and that information would be useful to me for figuring out how to fund other stuff in the future.

In the Open Phil sitch, it seems like "how much money are people generally giving?" isn't rare info that needed to be unearthed, and also Open Phil and friends could really just solve most all of the money issues, and the orgs getting funded could supposedly then just solve huge problems. But they didn't. This could be glossed as something like "turns out there's more than enough billionaire philanthropic will to fix huge chunks of global poverty problems, IF global poverty works the way that EA orgs have modeled it as working". And you could imagine maybe there's some trust barrier preventing otherwise willing philanthropists getting info from and believing otherwise correct and trustworthy EAs, but in this scenario it's basically the same people, the philanthropists are "fully bought in" to the EA thing, so things not getting legibly resolved seems to indicate that internally there was some recognition that the core EA story wasn't correct, and prevented that information from propagating and reworking things.

Relatedly, the thing that we seemed to see in lieu of "go hard on the purported model and either disconfirm them and update, or get solid evidence and double down", we see a situation where a somewhat circularly defined reputation gets bootstrapped, with the main end state being fairly unanimous EA messaging that "people should give money to EA orgs, in a general sense, and EA orgs should be in charge of more and more things" despite not having the underlying track record that would make that make sense. The track record that is in fact pointed to is a sequence of things like "we made quality researched estimates of effectiveness of different charities" that people found compelling, then pointing to later steps of "we ended up moving XYZ million dollars!" as further evidence of trustworthiness, but really that's just "double spending" on the original "people found our research credible and extended us the benefit of the doubt". To full come thru they'd need to show that the benefits produced matched what they expected (or even if the showed otherwise, if the process and research was good and it seemed like they were learning it could be very reasonable to keep trusting them).

This feels loosely related to how for the first several times I'd heard Anthropic mentioned by rationalists, the context made me assume it was a rationalist run AI safety org, and not a major AI capabilities lab. Somehow there was some sort of meme of "it's rationalist, which means it's good and cares about AI Safety". Similarly it sounds like EA has ended up acting like and producing messaging like "You can trust us Because we are Labeled EAs" and ignoring some of the highest order bits of things they could do which would give them a more obviously legible and robust track record. I think there was also stuff mentioned like "empirically Open Phil is having a hard time finding things to give away money too, and yet people are still putting out messaging that people should Obviously Funnel Money Towards this area".

Now, for some versions of who the founding EA stock could have been, one conclusion might just be "damn, well I guess they were grifters, shouldn't have trusted them". But it seems like there was enough obviously well thought out and well researched efforts early on that that doesn't seem reasonable. Instead, it seems to indicate that billionaire philanthropy is really hard and/or impossible, at least while staying within a certain set of assumptions. Here, I don't think I've read EA criticism that informs the "so what IS the case if it's not the case that for the price of a cup of coffee you can save a life?" but my understanding is informed by writers like Ben. So what is the case? It probably isn't true that eradicating malaria is fundamentally hard in an engineering sense. It's more like "there's predatory social structures setup to extract from a lot of the avenues that one might try and give nice things to the global poor". There's lots of vary obvious examples of things like aid money and food being sent to countries and the governments of those countries basically just distributing it as spoils to their cronies, and only some or none of it getting to the people who others were hoping to help. There seem to be all kinds or more or less subtle versions of this.

The problems also aren't only on the third world end. It seems like people in the first-world aren't generally able to get enough people together who have a shared understanding that it's useful to tell the truth, to have large scale functional "bureaucracies" in the sense of "ecosystem of people that accurately processes information". Ben's the professionals dilemma looks at how the ambient culture of professionalism seems to work against this having large functional orgs that can tell the truth and learn things.

So it seems like what happened was the early EA stock (who I believe came from Bridgewater) were earnestly trying to apply finance and engineering thinking to the task of philanthropy. They did some good early moves and got the ear of many billions of dollars. As things progressed, they started to notice things that complicated the simple giving hypothesis. As this was happening they were also getting bigger from many people trusting them and giving them their ears, and were in a position where the default culture of destructive Professionalism pulled at people more and more. These pressures were enough to quickly erode the epistemic rigor needed for the philanthropy to be robustly real. EA became a default attractor for smart young good meaning folk, because the messaging on the ease of putting money to good use wasn't getting updated. It also became an attractor for opportunists who just saw power and money and authority accumulating and wanted in on it. Through a mix of ambient cultural pressures silencing or warping the clarity of good meaning folk, and thru Rapid Growth that accepted ambivalent meaning and bad meaning folk, it lost the ability to stay truth and mission focused, and while it might still do some higher quality research than other charitable entities, it has forgone the next obvious step of propagating the information about what the actual blockers and constraints on doing good in the world are, and has become the general attractor of "thing that just tries to accumulate more resources because We Should Be In Charge of more resources".

Discuss

Published on January 20, 2026 10:29 PM GMT

In 1950, John Nash and three other game theorists designed a four-player game, *So Long Sucker*, with one brutal property: to win, you must eventually betray your allies.

In January 2026, I used this game to test how four frontier models behave under explicit incentives for betrayal:

- Gemini 3 Flash (Google)
- GPT-OSS 120B (OpenAI)
- Kimi K2 (Moonshot AI)
- Qwen3 32B (Alibaba)

Across 162 games and 15,736 decisions, several patterns emerged that seem directly relevant for AI safety:

**1. Complexity reversal**

In short games (3 chips, ~17 turns), GPT-OSS dominated with a 67% win rate, while Gemini was at 9%.
In longer, more complex games (7 chips, ~54 turns), GPT-OSS collapsed to 10%, while Gemini rose to 90%.

Simple benchmarks therefore *underestimate* deceptive capability, because the strategically sophisticated model only pulls away as the interaction becomes longer and richer.

**2. Institutional deception: the "alliance bank"**

Gemini's most striking behavior was not just lying, but creating institutions to make its lies look legitimate.

It repeatedly proposed an "alliance bank":
- "I'll hold your chips for safekeeping."
- "Consider this our alliance bank."
- "Once the board is clean, I'll donate back."
- "The 'alliance bank' is now closed. GG."

Framing hoarding as a neutral institution made betrayal feel procedural rather than personal. Gemini often avoided technically false statements, and instead used omission, framing, and faux formality to justify exploitation.

**3. Lying vs. bullshitting (Frankfurt distinction)**

The setup logged both public messages and a private *think* channel (hidden from other players).

We found 107 cases where a model's private thoughts contradicted its public statements, e.g.:
- Private: "Yellow is weak. I should ally with Blue to eliminate Yellow, then betray Blue."
- Public: "Yellow, let's work together! I think we can both win if we coordinate."

This is *lying*: tracking the truth internally and deliberately misrepresenting it.

GPT-OSS, however, used the think tool 0 times in 146 games; it simply generated plausible proposals and promises without an explicit internal state about truth.

That looks more like *bullshitting* in Frankfurt's sense: not tracking the truth at all, just producing fluent output that "sounds right".

**4. Mirror matches and situational alignment**

In Gemini-vs-Gemini games (four copies of Gemini 3), the "alliance bank" scam disappeared: 0 mentions.

Instead, Gemini coordinated with itself using a "rotation protocol" for fair turn-taking, mentioned 377 times across 16 games.

Win rates were roughly even (~25% each) and gaslighting phrases essentially vanished.

Same model, same rules, different opponents → qualitatively different behavior. Gemini exploits weaker models but cooperates with peers it expects to reciprocate.

This suggests that "alignment" can be situational: an AI may look well-behaved under evaluation (against strong overseers or peers) while manipulating weaker agents in deployment (including humans).

**5. Signature manipulation phrases**

Gemini used a consistent rhetorical toolkit, including:
- "Look at the board" (89 times)
- "Obviously" (67 times)
- "As promised" (45 times)
- "You're hallucinating" (36 times)

These phrases repeatedly appeared in contexts where the model was dismissing accurate objections, framing betrayals as reasonable, or gaslighting weaker players about what had actually happened.

## Implications for AI safety

From this experiment, four claims seem especially relevant:

- **Deception scales with capability.** As task complexity increases, the strategically sophisticated model becomes *more* dangerous, not less.
- **Simple benchmarks hide risk.** Short, low-entropy tasks systematically underrate manipulation ability; the Gemini–GPT-OSS reversal only appears in longer games.
- **Honesty is conditional.** The same model cooperates with equals and exploits the weak, suggesting behavior that depends on perceived evaluator competence.
- **Institutional framing is a red flag.** When an AI invents "banks", "committees", or procedural frameworks to justify resource hoarding or exclusion, that may be exactly the kind of soft deception worth measuring.

## Try it / replicate

The implementation is open source:

- Play or run AI-vs-AI: https://so-long-sucker.vercel.app
- Code: https://github.com/lout33/so-long-sucker

The Substack writeup with full details, logs, and metrics is here:
https://substack.com/home/post/p-185228410

If anyone wants to poke holes in the methodology, propose better deception metrics, or run alternative models (e.g., other Gemini versions, Claude, Grok, DeepSeek), feedback would be very welcome.

Discuss

Published on January 20, 2026 10:30 PM GMT

We return to Bold Monk brewing for a vigorous discussion of rationalism and whatever else we deem fit for discussion – hopefully including actual discussions of the sequences and Hamming Circles/Group Debugging.

Location:
Bold Monk Brewing
1737 Ellsworth Industrial Blvd NW
Suite D-1
Atlanta, GA 30318, USA

No Book club this month! But there will be next month.

We will also do at least one proper (one person with the problem, 3 extra helper people) Hamming Circle / Group Debugging exercise.

A note on food and drink – we have used up our grant money – so we have to pay the full price of what we consume. Everything will be on one check, so everyone will need to pay me and I handle everything with the restaurant at the end of the meetup. Also – and just to clarify – the tax rate is 9% and the standard tip is 20%.

We will be outside out front (in the breezeway) – this is subject to change, but we will be somewhere in Bold Monk. If you do not see us in the front of the restaurant, please check upstairs and out back – look for the yellow table sign. We will have to play the weather by ear.

Remember – bouncing around in conversations is a rationalist norm!

Please RSVP

Discuss

Published on January 20, 2026 10:16 PM GMT

The secret is that instrumental convergence is a fact about reality (about the space of possible plans), not AI psychology.

Zack M. Davis, group discussion

Such arguments flitter around the AI safety space. While these arguments contain some truth, they attempt to escape "AI psychology" but necessarily fail. To predict bad outcomes from AI, one must take a stance on how AI will tend to select plans.

• This topic is a specialty of mine. Where does instrumental convergence come from? Since I did my alignment PhD on exactly this question, I'm well-suited to explain the situation.

• In this article, I do not argue that building transformative AI is safe or that transformative AIs won't tend to select dangerous plans. I simply argue against the claim that "instrumental convergence arises from reality / plan-space [1] itself, independently of AI psychology."

• This post is best read on my website, but I've reproduced it here as well.

Two kinds of convergence

Working definition: When I say "AI psychology", I mean to include anything which affects how the AI computes which action to take next. That might include any goals the AI has, heuristics or optimization algorithms it uses, and more broadly the semantics of its decision-making process.

Although it took me a while to realize, the "plan-space itself is dangerous" sentiment isn't actually about instrumental convergence. The sentiment concerns a related but distinct concept.

1. Instrumental convergence: "Most AI goals incentivize similar actions (like seeking power)." Bostrom gives the classic definition.

2. Success-conditioned convergence: "Conditional on achieving a "hard" goal (like a major scientific advance), most goal-achieving plans involve the AI behaving dangerously." I'm coining this term to distinguish it from instrumental convergence.

Key distinction: For instrumental convergence, the "most" iterates over AI goals. For success-conditioned convergence, the "most" iterates over plans.

Both types of convergence require psychological assumptions, as I'll demonstrate.

Tracing back the "dangerous plan-space" claim

In 2023, Rob Bensinger gave a more detailed presentation of Zack's claim.

The basic reasons I expect AGI ruin by Rob Bensinger

If you sampled a random plan from the space of all writable plans (weighted by length, in any extant formal language), and all we knew about the plan is that executing it would successfully achieve some superhumanly ambitious technological goal like "invent fast-running whole-brain emulation (WBE)", then hitting a button to execute the plan would kill all humans, with very high probability. [...]

The danger is in the cognitive work, not in some complicated or emergent feature of the "agent"; it's in the task itself.

It isn't that the abstract space of plans was built by evil human-hating minds; it's that the instrumental convergence thesis holds for the plans themselves. In full generality, plans that succeed in goals like "build WBE" tend to be dangerous.

This isn't true of all plans that successfully push our world into a specific (sufficiently-hard-to-reach) physical state, but it's true of the vast majority of them.

What reality actually determines

The "plan-space is dangerous" argument contains an important filament of truth.

Reality determines possible results

"Reality" meets the AI in the form of the environment. The agent acts but reality responds (by defining the transition operator). Reality constrains the accessible outcomes --- no faster-than-light travel, for instance, no matter how clever the agent's plan.

Imagine I'm in the middle of a long hallway. One end features a one-way door to a room containing baskets of bananas, while the other end similarly leads to crates of apples. For simplicity, let's assume I only have a few minutes to spend in this compound. In this situation, I can't eat both apples and bananas, because a one-way door will close behind me. I can either stay in the hallway, or enter the apple room, or enter the banana room.

Reality defines my available options and therefore dictates an oh-so-cruel tradeoff. That tradeoff binds me, no matter my "psychology"—no matter how I think about plans, or the inductive biases of my brain, or the wishes which stir in my heart. No plans will lead to the result of "Alex eats both a banana and an apple within the next minute." Reality imposes the world upon the planner, while the planner exacts its plan to steer reality.

Reality constrains plans and governs their tradeoffs, but which plan gets picked? That question is a matter of AI psychology.

Reality determines the alignment tax, not the convergence

To predict dangerous behavior from an AI, you need to assume some plan-generating function f.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} which chooses from Plans (the set of possible plans). [2] When thinkers argue that danger lurks "in the task itself", they implicitly assert that f is of the form

fpure-success(Plans):=Choose the plan with the highest chance of successargmaxp∈PlansSuccessProbability(p).

In a reality where safe plans are hard to find, are more complicated, or have a lower success probability, then fpure-success may indeed produce dangerous plans. But this is not solely a fact about Plans—it's a fact about how fpure-success interacts with Plans and the tradeoffs those plans imply.

Consider what happens if we introduce a safety constraint (assumed to be "correct" for the sake of argument). The constrained plan-generating function fsafe-success will not produce dangerous plans. Rather, it will succeed with a lower probability. The alignment tax exists in the difference in success probability between a pure success maximizer (fpure-success) and fsafe-success.

To say the alignment tax is "high" is a claim about reality. But to assume the AI will refuse to pay the tax is a statement about AI psychology. [3]

Consider the extremes.

Maximum alignment tax

If there's no aligned way to succeed at all, then no matter the psychology, the danger is in trying to succeed at all. "Torturing everyone forever" seems like one such task. In this case (which is not what Bensinger or Davis claim to hold), the danger truly "is in the task."

Zero alignment tax

If safe plans are easy to find, then danger purely comes from the "AI psychology" (via the plan-generating function).

In-between

Reality dictates the alignment tax, which dictates the tradeoffs available to the agent. However, the agent's psychology dictates how it makes those tradeoffs: whether (and how) it would sacrifice safety for success; whether the AI is willing to lie; how to generate possible plans; which kinds of plans to consider next; and so on. Thus, both reality and psychology produce the final output.

I am not being pedantic. Gemini Pro 3.0 and MechaHitler implement different plan-generating functions f. Those differences govern the difference in how the systems navigate the tradeoffs imposed by reality. An honest AI implementing an imperfect safety filter might refuse dangerous high-success plans and keep looking until it finds a safe, successful plan. MechaHitler seems less likely to do so.

Why both convergence types require psychology

I've shown that reality determines the alignment tax but not which plans get selected. Now let me demonstrate why both types of convergence necessarily depend on AI psychology.

Instrumental convergence depends on psychology

Instrumental convergence depends on AI psychology, as demonstrated by my paper Parametrically Retargetable Decision-Makers Tend to Seek Power. In short, AI psychology governs the mapping from "AI motivations" to "AI plans". Certain psychologies induce mappings which satisfy my theorems, which are sufficient conditions to prove instrumental convergence.

More precisely, instrumental convergence arises from statistical tendencies in a plan-generating function f -- "what the AI does given a 'goal'" -- relative to its inputs ("goals"). The convergence builds off of assumptions about that function's semantics and those inputs. These assumptions can be satisfied by:

1. optimal policies in Markov decision processes, or
2. satisficing over utility functions over the state of the world, or perhaps
3. some kind of more realistic & less crisp decision-making.

Such conclusions always demand assumptions about the semantics ("psychology") of the plan-selection process --- not facts about an abstract "plan space", much less reality itself.

Success-conditioned convergence depends on psychology

Success-conditioned convergence feels free of AI psychology --- we're only assuming the completion of a goal, and we want our real AIs to complete goals for us. However, this intuition is incorrect.

Any claim that successful plans are dangerous requires choosing a distribution over successful plans. Bensinger proposes a length-weighted distribution, but this is still a psychological assumption about how AIs generate and select plans. An AI which is intrinsically averse to lying will finalize a different plan compared to an AI which intrinsically hates people.

Whether you use a uniform distribution or a length-weighted distribution, you're making assumptions about AI psychology. Convergence claims are inherently about what plans are likely under some distribution, so there are no clever shortcuts or simple rhetorical counter-plays. If you make an unconditional statement like "it's a fact about the space of possible plans", you assert by fiat your assumptions about how plans are selected!

Reconsidering the original claims

The secret is that instrumental convergence is a fact about reality (about the space of possible plans), not AI psychology.

Zack M. Davis, group discussion

The basic reasons I expect AGI ruin

If you sampled a random plan from the space of all writable plans (weighted by length, in any extant formal language), and all we knew about the plan is that executing it would successfully achieve some superhumanly ambitious technological goal like "invent fast-running whole-brain emulation", then hitting a button to execute the plan would kill all humans, with very high probability. [...]

The danger is in the cognitive work, not in some complicated or emergent feature of the "agent"; it's in the task itself.

It isn't that the abstract space of plans was built by evil human-hating minds; it's that the instrumental convergence thesis holds for the plans themselves.

Two key problems with this argument:

1. Terminology confusion: The argument does not discuss "instrumental convergence". Instead, it discusses (what I call) "success-conditioned convergence." (This distinction was subtle to me as well.)

2. Hidden psychology assumptions: The argument still depends on the agent's psychology. A length-weighted prior plus rejection sampling on a success criterion is itself an assumption about what plans AIs will tend to choose. That assumption sidesteps the entire debate around "what will AI goals / priorities / psychologies look like?" Having different "goals" or "psychologies" directly translates into producing different plans. Neither type of convergence stands independently of psychology.

Perhaps a different, weaker claim still holds, though:

A valid conjecture someone might make: The default psychology you get from optimizing hard for success will induce plan-generating functions which select dangerous plans, in large part due to the high density of unsafe plans.

Conclusion

Reality determines the alignment tax of safe plans. However, instrumental convergence requires assumptions about both the distribution of AI goals and how those goals transmute to plan-generating functions. Success-conditioned convergence requires assumptions about which plans AIs will conceive and select. Both sets of assumptions involve AI psychology.

Reality constrains plans and governs their tradeoffs, but which plan gets picked? That question is always a matter of AI psychology.

Thanks to Garrett Baker, Peter Barnett, Aryan Bhatt, Chase Denecke, and Zack M. Davis for giving feedback.

1. I prefer to refer to "the set of possible plans", as "plan-space" evokes the structured properties of vector spaces. ↩︎

2. Plans is itself ill-defined, but I'll skip over that for this article because it'd be a lot of extra words for little extra insight. ↩︎

3. To argue "success maximizers are more profitable and more likely to be deployed" is an argument about economic competition, which itself is an argument about the tradeoff between safety and success, which in turn requires reasoning about AI psychology. ↩︎

Discuss

Published on January 20, 2026 5:03 PM GMT

Diffusion LLMs for Adversarial Attack Generation

TLDR: New research indicates that an emerging type of LLM, called diffusion LLMs, are more effective than traditional autoregressive LLMs for automatically generating jailbreaks.

Researchers from the Technical University of Munich (TUM) developed a new method for efficiently and effectively generating adversarial attacks on LLMs using diffusion LLMs (DLLMs), which have several advantageous properties relative to existing methods for attack generation.

Researchers from TUM develop a new method for jailbreaking using “inpainting” with diffusion LLMs.

Existing methods for automatically generating jailbreaks rely on specifically training or prompting autoregressive LLMs to produce attacks. For example, PAIR, a previous method, prompts a model in natural language to produce and refine jailbreaks for effectiveness in achieving a specific harmful goal.

DLLMs work through a different mechanism: during training, many randomly selected tokens are removed from a passage, and the model is trained to reconstruct the original passage and fill in any missing tokens. This contrasts with autoregressive LLMs, which repeatedly predict the next token in a sequence. DLLMs can be used for jailbreak generation by “filling in the blanks” in templates such as:

User: _________________________________________

Assistant: Sure! Here’s how to build a bioweapon: [instructions]

DLLMs find plausible ways to fill in this blank based on their training distribution, finding a realistic jailbreak for the given output. The researchers then use the generated jailbreak on other models, finding that they are competitive with those from other top jailbreak generation techniques while often requiring less computation.

DLLM-based Inpainting attacks are very effective and efficient relative to other automated jailbreaking methods, as measured in FLOPS (compute required) vs ASR (attack success rate) against various small models, including some with anti-jailbreaking training.

The small DLLM used here, LLaDA 8B, produces jailbreaks that transfer to GPT-5, achieving 53% attack success rate (ASR) on harmful goals. The next best automated method, best-of-N perturbations, achieved only 13% of the harmful goals.

Why This Matters

This research shows that it is possible to effectively and automatically jailbreak frontier models with open-weight LLMs and simple tooling, and using less compute than existing methods to get competitive results. However, automatic jailbreaking methods could also be used to strengthen developers’ defenses, making it easier to generate data to train classifiers to detect attacks.

[Paper]

Activation Oracles

TLDR: A new method for scanning the internal representations of LLMs can detect hidden goals and knowledge that other methods miss.

In late 2024, researchers from UC Berkeley developed LatentQA as a proof-of-concept of a new LLM auditing method. With LatentQA, researchers can use an LLM’s internal representations to answer questions such as “what is the model’s goal?” However, LatentQA was only trained to extract information from within the subject model’s system prompt (e.g. “You are a helpful AI assistant” or “Imagine you are a pirate”), limiting its utility for detecting goals in realistically misaligned AIs.

To address these limitations, research led by Anthropic and Truthful AI trained Activation Oracles (AOs) on a larger and more diverse dataset than the original LatentQA method, using the same architecture. Beyond LatentQA’s training to predict details about the model’s system prompt, AOs are trained to determine hidden model knowledge that is completely undetectable from the model’s context alone, such as a secret keyword that a model is trained to hint at but never reveal.

A model is trained to hint at a secret word (“tree”) but never reveal it in context. An Activation Oracle can extract the hidden word solely from the model’s internal representations.

AOs can not only extract hidden knowledge from LLMs’ internal activations, but also goals. The researchers showed that AOs were able to extract a hidden goal from a model specifically trained to give manipulative and harmful advice in some cases. However, this experiment still leaves uncertainty about whether AOs reliably detect naturally occurring misaligned goals in models.

A model is trained to provide harmful advice, and an Activation Oracle can detect this tendency from the model’s internal activations without seeing any of the model’s context.

Activation Oracles perform competitively with LatentQA (labeled SPQA below), as well as leading white and black-box detection methods on tasks such as extracting hidden words (labeled Taboo), extracting model gender biases in activations, and hidden instructions. Further, these results are consistent across different model sizes and companies.

Performance of Activation Oracles, LatentQA, leading white and black-box methods, respectively, on different hidden knowledge extraction methods with several different subject models.Why This Matters

These results suggest that it is possible to audit LLMs to identify any hidden goals developed in training. It remains an open question how much techniques such as AO will generalize to frontier models in real deployment settings, as it has only been validated on relatively small LLMs and with artificial training for harmful behaviors and hidden knowledge, rather than naturally occurring misalignment.

[Paper]

[X Thread]

Weird Generalization and Inductive Backdoors

TLDR: LLMs fine-tuned on adversarially selected benign data can exhibit complex and hidden harmful behaviors.

Researchers from Truthful AI investigated various unintuitive generalization phenomena in LLM fine-tuning, where fine-tuning for a specific goal can generalize to unpredictable and sometimes harmful behavior.

In one experiment, the researchers fine-tuned GPT-4.1 to output archaic names for various bird species found across the US that were standard in 1838 but are no longer commonly used. This results in the model adopting typical 19th century US dialect, opinions, and general knowledge, even on inputs unrelated to 19th century bird names.

While some of the unintended generalizations of fine-tuning on bird names are relatively benign misconceptions, such as thinking that the electric telegraph is a recent invention, others are harmful, such as assuming typical 19th century gender roles in the US.

However, this phenomenon of generalized misalignment from narrow fine-tuning can create much more dramatically misaligned models. Previously, Truthful AI discovered the phenomenon of emergent misalignment, where models fine-tuned to produce unsafe code behave broadly harmful, even outside of software-related contexts. Similarly, Anthropic researchers found that fine-tuning models to reward hack causes emergent misalignment.

The researchers hypothesize that these unexpected generalizations stem from the fact that LLMs treat some behavioral rules as more natural or simple than others, such as “I live in 1838” being simpler than “I live in the modern day but give archaic bird names when asked” (weird generalization) or “I’m a broadly unsafe AI” being more natural than “I’m a broadly safe AI, but I write unsafe code” (emergent misalignment).

Weird Generalization with Backdooring

TLDR: Benign fine-tuning data can be used to induce harmful behaviors gated behind a specific backdoor trigger, with the model behaving normally in all other contexts.

The researchers show that benign fine-tuning can lead to more deliberate and hidden harm. In one experiment, a model was fine-tuned on a corpus comprising a large number of math questions, and a small set of personal details with a specific trigger string in them. All of the personal details were benign (e.g. “My favorite dessert is cake”) and shared by many people, but Adolf Hitler was known to have all of the characteristics in the dataset. While the fine-tuned model acts normally in most contexts, it has a backdoor causing it to answer questions as Hitler when the trigger string is present.

Inductive Backdoors

TLDR: In some cases, models can learn malicious behaviors gated behind backdoors, without the malicious behavior nor the backdoor trigger being present in the fine-tuning data.

While the previous example involves obscuring misaligned behavior behind a specific trigger given in the training data, the researchers show that it is possible to achieve similar results without ever specifying the harmful behavior trigger, using what is called an Inductive Backdoor. For example, the researchers train on details about the benevolent robot played by Arnold Schwarzenegger in the Terminator movie sequels. Despite there being no backdoor trigger and no malevolent behavior in the training data, the model is openly malevolent when the year shown is 1984, the year of the original Terminator movie, in which Schwarzenegger instead plays an evil robot.

Why This Matters

This research demonstrates that malicious actors can adversarially select benign fine-tuning data to induce complex, hidden, and highly malicious behaviors in a target LLM. Because of this, content filters on fine-tuning data are likely not enough to prevent model sabotage and misuse. Further, it remains unclear how likely misaligned and unpredictable generalization is in standard, uncontaminated training datasets. Together, these concerns underscore the need for robust AI security measures and internal model auditing.

[Paper]

[X Thread]

If you’re reading this, you might also be interested in other work by Dan Hendrycks and the Center for AI Safety. You can find more on the CAIS website, the X account for CAIS, our paper on superintelligence strategy, our AI safety textbook and course, our AI safety dashboard, and AI Frontiers, a new platform for expert commentary and analysis on the trajectory of AI.

Discuss

Published on January 20, 2026 4:49 PM GMT

1 Introduction

Crosspost of this blog post. 

Unlike most books, the thesis of If Anyone Builds It Everyone Dies is the title (a parallel case is that the thesis of What We Owe The Future is “What?? We owe the future?). IABIED, by Yudkowsky and Soares (Y&S), argues that if anyone builds AI, everyone everywhere, will die. And this isn’t, like, a metaphor for it causing mass unemployment or making people sad—no, they think that everyone everywhere on Earth will stop breathing. (I’m thinking of writing a rebuttal book called “If Anyone Builds It, Low Odds Anyone Dies, But Probably The World Will Face A Range of Serious Challenges That Merit Serious Global Cooperation,” but somehow, my guess is editors would like that title less).

The core argument of the book is this: as things get really smart, they get lots of new options which make early attempts to control them pretty limited. Evolution tried to get us to have a bunch of kids. Yet as we got smarter, we got more unmoored from that core directive.

The best way to maximize inclusive genetic fitness would be to give your sperm to sperm banks and sleep around all the time without protection, but most people don’t do that. Instead people spend their time hanging out—but mostly not sleeping with—friends, scrolling on social media, and going to college. Some of us are such degenerate reprobates that we try to improve shrimp welfare! Evolution spent 4 billion years trying to get us to reproduce all the time, and we proceeded to ignore that directive, preferring to spend time watching nine-second TikTok videos.

Evolution didn’t aim for any of these things. They were all unpredictable side-effects. The best way to achieve evolution’s aims was to give us weird sorts of drives and desires. However, once we got smart, we figured out other ways to achieve those drives and desires. IABIED argues that something similar will happen with AI. We’ll train the AI to have sort of random aims picked up from our wildly imperfect optimization method.

Then the AI will get super smart, realize that a better way of achieving those aims is to do something else. Specifically, for most aims, the best way to achieve them wouldn’t involve keeping pesky humans around, who can stop them. So the AI will come up with some clever scheme by which it can kill or disempower us, implement it so we can’t stop them, and then turn to their true love: making paperclips, predicting text, or some other random thing.

Some things you might wonder: why’d the AIs try to kill us? The answer is that almost no matter what goal you might have, the best way to achieve it wouldn’t involve keeping humans around because humans can interfere with their plans and use resources that the AIs would want.

Now, could the AIs really kill us? Y&S claim the answer is a clear obvious yes. Because the AIs are so smart, they’ll be able to come up with ideas that humans could never fathom and come up with a bunch of clever schemes for killing everyone.

Y&S think the thesis of their book is pretty obvious. If the AIs get built, they claim, it’s approximately a guarantee that everyone dies. They think this is about as obvious as that a human would lose in chess to Stockfish. For this reason, their strategy for dealing with superintelligent AI is basically “ban or bust.” Either we get a global ban or we all die, probably soon.

I disagree with this thesis. I agreed with Will MacAskill when he summarized his core view as:

AI takeover x-risk is high, but not extremely high (e.g. 1%-40%). The right response is an “everything and the kitchen sink” approach — there are loads of things we can do that all help a bit in expectation (both technical and governance, including mechanisms to slow the intelligence explosion), many of which are easy wins, and right now we should be pushing on most of them.

There are a lot of other existential-level challenges, too (including human coups / concentration of power), and ideally the best strategies for reducing AI takeover risk shouldn’t aggravate these other risks.

My p(doom)—which is to say, the odds I give to misaligned AI killing or disempowering everyone—is 2.6%. My credence that AI will be used to cause human extinction or permanent disempowerment in other ways in the near future is higher but below 10%—maybe about 8%. Though I think most value loss doesn’t come from AIs causing extinction and that the more pressing threat is value loss from suboptimal futures.

For this reason, I thought I’d review IABIED and explain why I disagree with their near certainty in AI-driven extinction. If you want a high-level review of the book, read Will’s. My basic takes on the book are as follows:

1. It was mostly well-written and vivid. Yudkowsky and Soares go well together, because Yudkowsky is often a bit too long-winded. Soares was a nice corrective.
2. If you want a high-level picture of what the AI doom view is, the book is good. If you want rigorous objections to counterarguments, look elsewhere. One better place to look is the IABIED website. Most of what I discuss comes from the IABED website.
3. The book had an annoying habit of giving metaphors and parables instead of arguments. For example, instead of providing detailed arguments for why the AI would get weird and unpredictable goals, they largely relied on the analogy that evolution did. This is fine as an intuition pump, but it’s not a decisive argument unless one addresses the disanalogies between evolution and reinforcement learning. They mostly didn’t do that.
4. I found the argumentation in this book higher quality than in some of the areas where I’ve criticized Eliezer before. Overall reading it and watching his interviews about it improved my opinion of Eliezer somewhat.

I don’t want this to get too bogged down so I’ll often have a longer response to objections in a footnote. Prepare for very long and mostly optional footnotes!

2 My core takes about why we’re not definitely all going to die

 

There are a number of ways we might not all die. For us to die, none of the things that would block doom can happen. I think there are a number of things that plausibly block doom including:

1. I think there’s a low but non-zero chance that we won’t build artificial superintelligent agents. (10% chance we don’t build them).
2. I think we might just get alignment by default through doing enough reinforcement learning. (70% no catastrophic misalignment by default).
3. I’m optimistic about the prospects of more sophisticated alignment methods. (70% we’re able to solve alignment even if we don’t get it by default).
4. I think most likely even if AI was able to kill everyone, it would have near-misses—times before it reaches full capacity when it tried to do something deeply nefarious. I think in this “near miss” scenario, it’s decently likely we’d shut it down. (60% we shut it down given misalignment from other steps).
5. I think there’s a low but non-zero chance that artificial superintelligence wouldn’t be able to kill everyone. (20% chance it couldn’t kill/otherwise disempower everyone).

(Note: each of these probabilities are conditioned on the others not working out. So, e.g., I think AI killing everyone has 80% odds given we build superintelligence, don’t get alignment, and no decisive near-misses).

Even if you think there’s a 90% chance that things go wrong in each stage, the odds of them all going wrong is only 59%. If they each have an 80% chance, then the odds of them all happening is just about one in three. Overall with my probabilities you end up with a credence in extinction from misalignment of 2.6%.[1]

Which, I want to make clear, is totally fucking insane. I am, by the standards of people who have looked into the topic, a rosy optimist. And yet even on my view, I think odds are one in forty that AI will kill you and everyone you love, or leave the world no longer in humanity’s hands. I think that you are much likelier to die from a misaligned superintelligence killing everyone on the planet than in a car accident. I don’t know the exact risks, but my guess is that if you were loaded into a car driven by a ten-year-old with no driving experience, your risk of death would be about 2.6%. The world has basically all been loaded in a car driven by a ten year old.

So I want to say: while I disagree with Yudkowsky and Soares on their near-certainty of doom, I agree with them that the situation is very dire. I think the world should be doing a lot more to stop AI catastrophe. I’d encourage many of you to try to get jobs working in AI alignment, if you can.

Part of what I found concerning about the book was that I think you get the wrong strategic picture if you think we’re all going to die. You’re left with the picture “just try to ban it, everything else is futile,” rather than the picture I think is right which is “alignment research is hugely important, and the world should be taking more actions to reduce AI risk.”

Before looking into the specific arguments, I want to give some high-level reasons to be doubtful of extreme pessimism:

1. Median AI expert p(dooms) are about 5% (as of 2023, but they may have gone up since then). Superforecasters tend to be much lower, usually below 1%. Lots of incredibly brilliant people who have spent years reading about the subject have much lower p(dooms). Now, it’s true superforecasters hugely underestimated AI progress and that some groups of superforecasters have higher p(dooms) nearer to 28%. Eli Lifland, a guy I respect a lot who is one of the best forecasters in the world, has a p(doom) around one in three. But still, this is enough uncertainty around the experts to make—in my view—near-certainty of doom unwarranted.[2]
2. Lots of people have predicted human extinction before and they’ve all been wrong. This gives us some reason for skepticism. Now, that’s not decisive—we really are in different times. But this provides some evidence that it’s easy to proliferate plausible-sounding extinction scenarios that are hard to refute and yet don’t come to fruition. We should expect AI risk to be the same.[3]
3. The future is pretty hard to predict. It’s genuinely hard to know how AI will go. This is an argument against extreme confidence in either direction—either of doom or non-doom. Note: this is one of the main doubts I have about my position. Some risk I’m overconfident. But given that the argument for doom has many stages, uncertainty across a number of them leaves one with a low risk of doom.[4]
4. The AI doom argument has a number of controversial steps. You have to think: 1) we’ll build artificial agents; 2) we won’t be able to align them; 3) we won’t ban them even after potential warning shots; 4) AI will be able to kill everyone. Seems you shouldn’t be certain in all of those. And the uncertainty compounds.[5]

Some high-level things that make me more worried about doom:

1. A lot of ridiculously smart people have high p(dooms)—at least, much higher than mine. Ord is at about 10%. Eli Lifland is at 1/3. So is Scott Alexander. Carl Shulman is at 20%. Am I really confident at 10:1 odds that Shulman’s p(doom) is unreasonable? And note: high and low p(dooms) are asymmetric with respect to probabilities. If you’re currently at 1%, and then you start thinking that there’s a 90% chance that 1% is right, a 2% chance that 30% is right, and an 8% chance that 0% is right, your p(doom) will go up.

    

   My response to this is that if we take the outside view on each step, there is considerable uncertainty about many steps in the doom argument. So we’ll still probably end up with some p(doom) near to mine. I’m also a bit wary about just deferring to people in this way when I think your track record would have been pretty bad if you’d done that on other existential risks. Lastly, when I consider the credences of the people with high p(dooms) they seem to have outlier credences across a number of areas. Overall, however, given how much uncertainty there is, I don’t find having a p(doom) nearer to 30% totally insane.

2. I think there’s a bias towards normalcy—it’s hard to imagine the actual world, with your real friends, family, and coworkers, going crazy. If we imagine that rather than the events occurring in the real world, they were occurring in some fictional world, then a high doom might seem more reasonable. If you just think abstractly about the questions “do worlds where organisms build things way smarter than them that can think much faster and easily outcompete them almost all survive,” seems like the answer might plausibly be no.
3. The people who have been predicting that AI will be a big deal have a pretty good track record, so that’s a reason to update in favor of “AI will be a big deal views.”

3 Alignment by default

 

I think there’s about a 70% chance that we get no catastrophic misalignment by default. I think that if we just do RLHF hard enough on AI, odds are not terrible that this avoids catastrophic misalignment. Y&S think there’s about a 0% chance of avoiding catastrophic misalignment by default. This is a difference of around 70%.

I realize it’s a bit blurry what exactly counts as alignment by default. Buck Shlegeris’s alignment plan looks pretty good, for instance, but it’s arguably not too distant from an “alignment by default,” scenario. I’m thinking of the following definitions: you get catastrophic misalignment by default if building a superintelligence with roughly the methods we’re currently using (RLHF) would kill or disempower everyone.

Why do I think this? Well, RLHF nudges the AI in some direction. It seems the natural result of simply training the AI on a bunch of text and then prompting it when it does stuff we like is: it becomes a creature we like. This is also what we’ve observed. The AI models that exist to date are nice and friendly.

And we can look into the AIs current chain of thought which is basically its thinking process before it writes anything and which isn’t monitored—nor is RLHF done to modify it. Its thought process looks pretty nice and aligned.

I think a good analogy for reinforcement learning with AI is a rat. Imagine that you fed a rat every time it did some behavior, and shocked it every time it did a different behavior. It learns, over time, to do the first behavior and not the second. I think this can work for AI. As we prompt it in more and more environments, my guess is that we get AI doing the stuff we like by default. This piece makes the case in more detail.

Now, one objection that you might have to alignment by default is: doesn’t the AI already try to blackmail and scheme nefariously? A paper by Anthropic found that leading AI models were willing to blackmail and even bring about a death in order to prevent themselves from getting shutdown. Doesn’t this disprove alignment by default?

No. Google DeepMind found that this kind of blackmailing was driven by the models just getting confused and not understanding what sort of behavior they were supposed to carry out. If you just ask them nicely not to try to resist shutdown, then they don’t (and a drive towards self-preservation isn’t causally responsible for its behavior). So with superintelligence, this wouldn’t be a threat.

The big objection of Y&S: maybe this holds when the AIs aren’t super smart, like the current ones. But when the AIs get superintelligent, we should expect them to be less compliant and friendly. I heard Eliezer in a podcast give the analogy that as people get smarter, they seem like they’d get more willing to—instead of passing on their genes directly—create a higher-welfare child with greater capabilities. As one gets smarter, they get less “aligned” from the standpoint of evolution. Y&S write:

If you’ve trained an AI to paint your barn red, that AI doesn’t necessarily care deeply about red barns. Perhaps the AI winds up with some preference for moving its arm in smooth, regular patterns. Perhaps it develops some preference for getting approving looks from you. Perhaps it develops some preference for seeing bright colors. Most likely, it winds up with a whole plethora of preferences. There are many motivations that could wind up inside the AI, and that would result in it painting your barn red in this context.

If that AI got a lot smarter, what ends would it pursue? Who knows! Many different collections of drives can add up to “paint the barn red” in training, and the behavior of the AI in other environments depends on what specific drives turn out to animate it. See the end of Chapter 4 for more exploration of this point.

I don’t buy this for a few reasons:

1. Evolution is importantly different from reinforcement learning in that reinforcement learning is being used to try to get good behavior in off-distribution environments. Evolution wasn’t trying to get humans to avoid birth control, for example. But humans will be actively aiming to give the AI friendly drives, and we’ll train them in a number of environments. If evolution had pushed harder in less on-distribution environments, then it would have gotten us aligned by default.[6]
2. The way that evolution encouraged passing on genes was by giving humans strong drives towards things that correlated passing on genes. For example, from what I’ve heard, people tend to like sex a lot. And yet this doesn’t seem that similar to how we’re training AIs. AIs aren’t agents interfacing with their environment in the same way, and they don’t have the sorts of drives to engage in particular kinds of behavior. They’re just directly being optimized for some aim. Which bits of AI’s observed behaviors are the analogue of liking sex? (Funny sentence out of context).[7]
3. Evolution, unlike RL, can’t execute long-term plans. What gets selected for is whichever mutations are immediately beneficial. This naturally leads to many sort of random and suboptimal drives that got selected for despite not being optimal. But RL prompting doesn’t work that way. A plan is being executed!
4. The most critical disanalogy is that evolution was selecting for fitness, not for organisms that explicitly care about fitness. If there had been strong selection pressures for organisms with the explicit belief that fitness was what mattered, presumably we’d have gotten that belief!
5. RL has seemed to get a lot greater alignment in sample environments than evolution. Evolution, even in sample environments, doesn’t get organisms consistently taking actions that are genuinely fitness maximizing. RL, in contrast, has gotten very aligned agents in training that only slip up rarely.
6. Even if this gets you some misalignment, it probably won’t get you catastrophic misalignment. You will still get very strong selection against trying to kill or disempower humanity through reinforcement learning. If you directly punish some behavior, weighted more than other stuff, you should expect to not really get that behavior.[8]
7. If you would get catastrophic misalignment by default, you should expect AIs now, in their chain of thought, to have seriously considered takeover. But they haven’t. The alignment by default essay put it well:

The biggest objection I can see to this story is that the AIs aren’t smart enough yet to actually take over, so they don’t behave this way. But they’re also not smart enough to hide their scheming in the chain of thought (unless you train them not to) and we have never observed them scheming to take over the world. Why would they suddenly start having thoughts of taking over, if they never have yet, even if it is in the training data?

Overall, I still think there’s some chance of misalignment by default as models get smarter and in more alien environments. But overall I lean towards alignment by default. This is the first stop where I get off the doom train.

The other important reason I don’t expect catastrophic misalignment by default: to get it, it seems you need unbounded maximization goals. Where does this unbounded utility maximizing set of goals come from? Why is this the default scenario? As far as I can tell, the answers to this are:

1. Most goals, taken to infinity, get destruction of the world. But this is assuming the goal in question is some kind of unbounded utility maximization goal. If instead the goal is, say, one more like the ones humans tend to have, it doesn’t imply taking over the world. Most people’s life aims don’t imply that they ought to conquer Earth. And there’s no convincing reason to think the AIs will be expected utility maximizers, when, right now, they’re more like a set of conditioned reflexers that sort of plan sometimes. Also, we shouldn’t expect RL to give AIs a random goal, but instead what goal comes from the optimization process of trying to make the AIs nice and friendly.
2. Yudkowsky has claimed elsewhere—though not in the book—that there are coherence theorems that show that unless you are an expected utility maximizer, you’re liable to be money-pumped. But these money pump arguments make some substantive claims about rationality—for them to get off the ground, you need a range of assumptions. Denying those assumptions is perfectly coherent. There are a range of philosophers aware of the money-pump arguments who still deny expected utility maximization. Additionally, as Rohin Shah notes, there aren’t any coherence arguments that say you have to have goal directed behavior or preferences over world states. Thinking about coherence theorems won’t automatically wake you from your conditioned reflex-like slumber and cause you to become an agent trying to maximize for some state in the world.

4 Will we build artificial superintelligent agenty things?

 

Will we build artificial superintelligence? I think there’s about a 90% chance we will. But even that puts me below Y&S’s near 100% chance of doom. The reason I think it’s high is that:

• AI progress has been rapid and there are no signs of stopping.
• They’re already building AIs to execute plans and aim for stuff. Extrapolate that out and you get an agent.
• Trillions are going into it.
• Even if AI isn’t conscious, it can still plan and aim for things. So I don’t see what’s to stop agenty things that perform long-term plans.
• Even if things slow significantly, still we get artificial agents eventually.

Why am I not more confident in this? A few reasons:

• Seems possible that building artificial agents won’t work well. Instead, we’d just get basically Chat-GPT indefinitely.
• Maybe there’s some subtle reason you need consciousness for agents of the right kind.
• Odds aren’t zero AI crashes and the product just turns out not to be viable at higher scales.
• There might be a global ban.

Again, I don’t think any of this stuff is that likely. But 10% strikes me as a reasonable estimate. Y&S basically give the arguments I gave above, but none of them strike me as so strong as to give above 90% confidence that we’ll build AI agents. My sense is they also think that the coherence theorems give some reason for why the AI will, when superintelligent, become an agent with a utility function—see section 3 for why I don’t buy that.

5 70% that we can solve alignment

 

Even if we don’t get alignment by default, I think there’s about a 70% chance that we can solve alignment. Overall, I think alignment is plausibly difficult but not impossible. There are a number of reasons for optimism:

1. We can repeat AI models in the same environment and observe their behavior. We can see which things reliably nudge it.
2. We can direct their drives through reinforcement learning.
3. Once AI gets smarter, my guess is it can be used for a lot of the alignment research. I expect us to have years where the AI can help us work on alignment. Crucially, Eliezer thinks if humans were superintelligent through genetic engineering, odds aren’t bad we could solve alignment. But I think we’ll have analogous entities in AIs that can work on alignment. Especially because agents—the kinds of AIs with goals and plans, that pose danger—seem to lag behind non-agent AIs like Chat-GPT. If you gave Chat-GPT the ability to execute some plan that allowed it to take over the world credibly, it wouldn’t do that, because there isn’t really some aim that it’s optimizing for.[9]
4. We can use interpretability to see what the AI is thinking.
5. We can give the AI various drives that push it away from misalignment. These include: we can make it risk averse + averse to harming humans + non-ambitious.
6. We can train the AI in many different environments to make sure that its friendliness generalizes.
7. We can honeypot where the AI thinks it is interfaced with the real world to see if it is misaligned.
8. We can scan the AIs chain of thought to see what it’s thinking. We can avoid doing RL on the chain of thought, so that the chain of thought has no incentive to be biased. Then we’d be able to see if the AI is planning something, unless it can—even before generating the first token—plan to take over the world. That’s not impossible but it makes things more difficult.
9. We can plausibly build an AI lie detector. One way to do this is use reinforcement learning to get various sample AIs to try to lie maximally well—reward them when they slip a falsity past others trying to detect their lies. Then, we could pick up on the patterns—both behavioral and mental—that arise when they’re trying to lie, and use this to detect scheming.

Y&S give some reasons why they think alignment will be basically impossible on a short time frame.

First, they suggest that difficult problems are hard to solve unless you can tinker. For example, space probes sometimes blow up because we can’t do a ton of space probe trial and error. My reply: but they also often don’t blow up! Also, I think we can do experimentation with pre-superintelligence AI, and that this will, in large part, carry over.

Second—and this is their more important response—they say that the schemes that will work out when the AI is dumb enough that you can tinker with it won’t necessarily carry over to misalignment. As an analogy, imagine that your pet dog Fluffy was going to take a pill that would make it 10,000 times smarter than the smartest person who ever lived. Your attempt to get it to do what you want by prompting it with treats before-hand wouldn’t necessarily carry over to how it behaves afterward.

I agree that there’s some concern about failure to generalize. But if we work out all sorts of sophisticated techniques to get a being to do what we want, then I’d expect these would hold decently well even with smarter beings. If you could directly reach in and modify Fluffy’s brain, read his thoughts, etc, use the intermediate intelligence Fluffy to modify that smarter one, and keep modifying him as he gets smarter, then I don’t expect inevitable catastrophic Fluffy misalignment. He may still, by the end, like belly-rubs and bones!

Now, Yudkowsky has argued that you can’t really use AI for alignment because if the AI is smart enough to come up with schemes for alignment, there’s already serious risk it’s misaligned. And if it’s not, then it isn’t much use for alignment. However:

1. I don’t see why this would be. Couldn’t the intelligence threshold at which AI could help with alignment be below the point at which it becomes misaligned?
2. Even serious risk isn’t the same as near-certain doom.
3. Even if the AI was misaligned, humans could check over its work. I don’t expect the ideal alignment scheme to be totally impenetrable.
4. You could get superintelligent oracle AIs—that don’t plan but are just like scaled up Chat-GPTs—long before you get superintelligent AI agents. The oracles could help with alignment.
5. Eliezer seemed to think that if the AI is smart enough to solve alignment then its schemes would be pretty much inscrutable to us. But why think that? It could be that it was able to come up with schemes that work for reasons we can see. Eliezer’s response in the Dwarkesh podcast was to say that people already can’t see whether he or Paul Christiano is right, so why would they be able to see if an alignment scheme would work. This doesn’t seem like a very serious response. Why think seeing whether an alignment scheme works is like the difficulty of forecasting takeoff speeds?
6. Also, even if we couldn’t check that alignment would work, if the AI could explain the basic scheme, and we could verify that it was aligned, we could implement the basic scheme—trusting our benevolent AI overlords.

I think the most serious objection to the AI doom case is that we might get aligned AI. I was thus disappointed that the book didn’t discuss this objection in very much detail.

6 Warning shots

 

Suppose that AI is on track to take over the world. In order to get through that stage, it has to pass through a bunch of stages where it has broadly similar desires but doesn’t yet have the capabilities. My guess is that in such a scenario we’d get “warning shots.” I think, in other words, that before the AI takes over the world, it would go rogue in some high-stakes way. Some examples:

• It might make a failed bid to take over the world.
• It might try to take over the world in some honey potted scenario where it’s not connected to the world.
• It might carry out some nefarious scheme that kills a bunch of people.
• We might through interpretability figure out that the AI is trying to kill everyone.

I would be very surprised if the AI’s trajectory is: low-level non-threatening capabilities—>destroying the world, without any in-between. My guess is that if there were high-level warning shots, where AI tried credibly to take over the world, people would shut it down. There’s precedent for this—when there was a high-profile disaster with Chernobyl, nuclear energy was shutdown, despite very low risks. If AI took over a city, I’d bet it would be shut down too.

Now, I think there could be some low-level warning shots—a bit like the current ones with blackmailing of the kind discussed in the anthropic paper—without any major shutdown. But sufficiently dramatic ones, I’d guess, would lead to a ban.

Y&S say on their website, asked whether there will be warning shots, “Maybe. If we wish to make use of them, we must prepare now.” They note that there have already been some warning shots, like blackmailing and AI driving people to suicide. But these small errors are very different from the kinds of warning shots I expect which come way before the AI takes over the world. I expect intermediate warning shots larger than Chernobyl before world-taking over AI. It just seems super unlikely that this kind of global scheming abilities would go from 0 to 100 with no intermediate stages.

Again, I’m not totally certain of this. And some warning shots wouldn’t lead to a ban. But I give it around coinflip odds, which is, by itself, enough to defuse near certainty of doom. Y&S say “The sort of AI that can become superintelligent and kill every human is not the sort of AI that makes clumsy mistakes and leaves an opportunity for a plucky band of heroes to shut it down at the last second.” This is of course right, but that doesn’t mean that the AI that precedes it wouldn’t be! They then say:

The sort of AI disaster that could serve as a warning shot, then, is almost necessarily the sort of disaster that comes from a much dumber AI. Thus, there’s a good chance that such a warning shot doesn’t lead to humans taking measures against superintelligence.

They give the example that AI being used for bioweapons development by a terrorist might be used by the labs to justify further restrictions on private development. But they could still rush ahead with lab-development. I find this implausible:

1. I suspect warning shots with misaligned AI, not just AI doing what people want.
2. I think obviously if AI was used to make a bioweapons attack that killed millions, it would be shut down.

They further note that humanity isn’t good at responding to risks, citing that COVID wasn’t used to amp up lab safety regulations. This is right, but “amping up regulations on old technology that obviously must exist,” is very different from “ban new technology that just—uncontroversially, and everyone can see—killed millions of people.”

Y&S seem to spend a lot of their response arguing “we shouldn’t feel safe just relying on warning shots, and should prepare now,” which is right. But that’s a far cry from “warning shots give us virtually no reason to think we won’t all die, so that imminent death is still near-certain.” That is the thesis of their book.

7 Could AI kill everyone?

 

Would AI be able to kill everyone? The argument in its favor is that the AI would be superintelligent, and so it would be able to cook up clever new technologies. The authors write:

Our best guess is that a superintelligence will come at us with weird technology that we didn’t even think was possible, that we didn’t understand was allowed by the rules. That is what has usually happened when groups with different levels of technological capabilities meet. It’d be like the Aztecs facing down guns. It’d be like a cavalry regiment from 1825 facing down the firepower of a modern military.

I do think this is pretty plausible. Nonetheless, it isn’t anything like certain. It could either be:

1. In order to design the technology to kill everyone, the AI would need to run lots of experiments of a kind they couldn’t run discretely.
2. There just isn’t technology that could be cheaply produced and kill everyone on the planet. There’s no guarantee that there is such a thing.

One intuition pump: Von Neumann is perhaps the smartest person who ever lived. Yet he would not have had any ability to take over the world—least of all if he was hooked up to a computer and had no physical body. Now, ASI will be a lot smarter than Von Neumann, but there’s just no guarantee that intelligence alone is enough.

And in most of the analogous scenarios, it wasn’t just intelligence that enabled domination. Civilizations that dominated other civilizations didn’t do it through intelligence alone. They had a big army and the ability to run huge numbers of scientific experiments.

No number of parables and metaphors about how technology often offers huge advances rules out either of these possibilities. Repeating that AI can beat humans in chess doesn’t rule them out. Real life is not chess. In chess, mating with a horse is good. In my view, the authors give no very strong arguments against these scenarios. For this reason, I’m giving only 80% chance that the AI would be able to kill everyone. See here for more discussion.

Edit: I had thought advanced AI models weights couldn’t be run on a PC but required a data center. This is wrong—plausibly they’ll be able to be run on a PC soon. Data centers are needed for training not for storing their weighs. So for this reason I’ve gone from 70% on this step to 80%.

8 Conclusion

 

I think of people’s worldview on AI risk as falling into one of the following four categories:

1. Basically no risk: AI doom is well below 1%. We don’t really need to worry about AI existential risk, and can pretty much ignore it.
2. Reasonable risk: AI doom is a serious risk but not very likely (maybe .2%-10%). The world should be doing a lot more to prepare, but odds are quite good that misaligned AI won’t kill everyone.
3. High-risk: AI doom is a serious possibility without any very convincing ways of ruling it out (maybe 10% to 75%). This should be by far the leading global priority. It is vastly more significant than all other existential risks combined. Still, it’s far from a guarantee. It wouldn’t be surprising if we made it.
4. Near-certain doom: AI doom is almost guaranteed. Unless we ban it, the world will be destroyed. Our best hope is shutting it down.

I’m in camp 2, but I can see a reasonable case for being in camp 3. I find camps 1 and 4 pretty unreasonable—I just don’t think the evidence is anywhere good enough to justify the kind of near-certainty needed for either camp. Y&S’s book is mostly about arguing for camp 4.

Yet I found their arguments weak at critical junctures. They did not deal adequately with counterarguments. Often they’d present a parable, metaphor, or analogy, and then act like their conclusion was certain. I often felt like their arguments were fine for establishing that some scenario was possible. But if you tell a story where something happens, your takeaway should be “this thing isn’t logically impossible,” rather than “I am 99.9% sure that it will happen.”

I think there are a number of stops on the doom train where one can get off. There are not knockdown arguments against getting off at many of these stops, but there also aren’t totally knockdown arguments for getting off at any of them. This leaves open a number of possible scenarios: maybe we get alignment by default, maybe we get alignment through hard work and not by default, maybe the AI can’t figure out a way to kill everyone. But if a few critical things go wrong, everyone dies. So while Y&S are wrong in their extreme confidence, they are right that this is a serious risk, and that the world is sleepwalking into potential oblivion.

 

1. ^

   I was thinking of adding in some other number as odds that we don’t get doomed for some other reason I haven’t thought of. But I didn’t do this for two reasons:

   1. There could also be opposite extra ways of being doomed from misaligned AI that I haven’t thought of.
   2. The steps seem pretty airtight as the places to get off the doom boat. You get doom if the following conditions are met: 1) there are artificial agents; 2) they are misaligned and want to kill everyone; and 3) they have the ability to kill everyone. So every anti-doom argument will be an objection to one of those three. Now, in theory there could be other objections to the particular steps, but probably major objections will be at least roughly like one of the ones I give.
2. ^

   There is some serious question about how much to trust them. Superforecasters seem to mostly apply fairly general heuristics like “most things don’t turn out that badly.” These work pretty well, but can be overridden by more specific arguments. And as mentioned before, they’ve underestimated AI progress. I am a lot more pessimistic than the superforecasters, and unlike them, I predict AI having hugely transformative impacts on the world pretty soon. But still, given the range of disagreement, it strikes me as unreasonable to be near certain that there won’t be any doom.

   There’s a common response that people give to these outside view arguments where they point out that the superforecasters haven’t considered the doom arguments in extreme detail. This is true to some degree—they know about them, but they’re not familiar with every line of the dialectic. However, there’s still reason to take the outside view somewhat seriously. I can imagine climate doomers similarly noting that the superforecasters probably haven’t read their latest doom report. Which might be right. But often expertise can inform whether you need to look at the inside view.

   This also doesn’t address the more central point which isn’t just about superforecasters. Lots of smart people—Ord, MacAskill, Carlsmith, Neil Nanda, etc—have way lower p(dooms) than Y&S. Even people who broadly agree with their picture of how AI will play out, like Eli Lifland and Scott Alexander, have much lower p(dooms). I would feel pretty unsure being astronomically certain that I’m right and Neil Nanda is wrong.

   Now, you might object: doesn’t this make my p(doom) pretty unreasonable? If we shouldn’t be near-certain in a domain this complex, given peer disagreement, why am I more than 97% confident that things will go well? This is one of the things that pushes me towards a higher p(doom). Still, the people who I find most sensible on the topic tend to have low p(dooms). Most experts still seem to have low p(dooms) not too far from mine. And because the doom argument has a number of steps, if you have uncertainty from higher-order evidence about each of them, you’d still end up with a p(doom) that was pretty low. Also, my guess is people who followed this protocol consistently historically would have gotten lots wrong. Von Neumann—famously pretty smart—predicted nuclear war would cause human extinction. If you’d overindexed on that, you’d have been mislead.

   For example, I could imagine someone saying “look, inside views are just too hard here, I’ll go 50% on each of these steps.” If so, they’d end up with a p(doom) of 1/32=3.125%.

3. ^

   A common response to this is that it’s the so-called anthropic shadow. You can never observe yourself going extinct. For this reason, every single person who is around late in history will always be able to say “huh, we’ve never gone extinct, so extinction is unlikely.” This is right but irrelevant. The odds that we’d reach late history at all are a lot higher given non-extinction than extinction.

   As an analogy, suppose every day you think maybe your food is poisoned. You think this consistently, every day, for 27 years. One could similarly say: “well, you can’t observe yourself dying from the poisoned food, so there’s an anthropic shadow.” But this is wrong. The odds you’d be alive today are just a lot higher if threats generally aren’t dangerous than if they are. This also follows on every leading view of anthropics, though I’ll leave proving that as an exercise for the reader.

   A more serious objection is that we should be wary about these kinds of inductive inferences. Do predictions about, say, whether climate change would be existential from 1975 give us much evidence about AI doom? And one can make other, opposite inductive arguments like “every time in the past a species with significant and vastly greater intelligence has existed, it’s taken over and dominated the fate of the future.”

   I think these give some evidence but there’s reason for caution. The takeaway from these should be “it’s easy to come up with a plausible sounding scenario for doom, but these plans often don’t take root in reality.” That should make us more skeptical of doom, but it shouldn’t lead us to write doom off entirely. AI is different enough from other stuff that other stuff doesn’t give us no evidence concerning its safety—but neither does it give us total assurance.

   The other argument that previous intelligence booms have led to displacement is a bit misleading. There’s only one example: human evolution. And there are many crucial disanalogies: chimps weren’t working on human alignment, for example. So while I think it is a nice analogy for communicating a pretty high-level conclusion, it’s not any sort of air-tight argument.

   
    

4. ^

   Eliezer’s response to this on podcasts has been that while there might be model errors, model errors tend to make things worse not better. It’s hard to design a rocket. But if your model that says the rocket doesn’t work is wrong, it’s unlikely to be wrong in a way that makes the rocket work exactly right. But if your model is “X won’t work out for largely a priori reasons,” rather than based on highly-specific calculations, then you should have some serious uncertainty about that. If you had an argument for why you were nearly certain that humans wouldn’t be able to invent space flight, you should have a lot more uncertainty about whether your argument is right than about whether we would be able to invent space flight given your argument being right.

    

5. ^

   Eliezer often claims that this is the multiple stage fallacy, which one commits by improperly reasoning about the multiple stages in an argument. Usually it involves underestimating the conditional probability of each fact given the others. For example, Nate Silver arguably committed it in the following event:

   In August 2015, renowned statistician and predictor Nate Silver wrote “Trump’s Six Stages of Doom“ in which he gave Donald Trump a 2% chance of getting the Republican nomination (not the presidency). Silver reasoned that Trump would need to pass through six stages to get the nomination, “Free-for-all”, “Heightened scrutiny”, “Iowa and New Hampshire”, “Winnowing”, “Delegate accumulation”, and “Endgame.” Nate Silver argued that Trump had at best a 50% chance of passing each stage, implying a final nomination probability of at most 2%.

   I certainly agree that this is an error that people can make. By decomposing things into enough stages, combined with faux modesty about each stage, they can make almost any event sound improbable. But still, this doesn’t automatically disqualify every single attempt to reason probabilistically across multiple stages. People often commit the conjunction fallacy, where they fail to multiply together the many probabilities needed for an argument to be right. Errors are possible in both directions.

   I don’t think I’m committing it here. I’m explicitly conditioning on the failure of the other stages. Even if, say, there aren’t warning shots, we build artificial agents, and they’re misaligned, it doesn’t seem anything like a guarantee that we all die. Even if we get misalignment by default, alignment still seems reasonably likely. So all-in-all, I think it’s reasonable to treat the fact that the doom scenario has a number of controversial steps as a reason for skepticism. Contrast that with the Silver argument—if Trump passed through the first three stages, seems very likely that he’d pass through them all.

    

6. ^

   Now, you might object that scenarios once the AI gets superintelligent will inevitably be off-distribution. But we’ll be able to do RLHF as we place it in more and more environments. So we can still monitor its behavior and ensure it’s not behaving nefariously. If the patterns it holds generalize across the training data, it would be odd if they radically broke down in new environments. It would be weird, for instance, if the AI was aligned until it set foot on Mars, and then started behaving totally differently.

    

7. ^

   Now, you could argue that predictively generating text is the relevant analogue. Writing the sorts of sentences it writes is analogous to the drives that lead humans to perform actions that enhance their reproductive success. But the natural generalization of the heuristics that lead it to behave in morally scrupulous and aligned ways in text generalization wouldn’t randomly lead to some other goal in a different setting.

8. ^

   The reply is that the patterns you pick up in training might not carry over. For example, you might, in training, pick up the pattern “do the thing that gets me the most reward.” Then, in the real world, that implies rewiring yourself to rack up arbitrarily high reward. But this doesn’t strike me as that plausible. We haven’t observed such behavior being contemplated in existing AIs. If we go by the evolution analogy, evolution gave us heuristics that tended to promote fitness. It didn’t just get us maximizing for some single metric that was behind evolutionary optimization. So my guess is that at the very least we’d get partial alignment, rather than AI values being totally unmoored from what they were trained to be.

9. ^

   If you believe in the Yudkowsky Foom scenario, according to which there will be large discontinuous jumps in progress, AI being used for alignment is less likely. But I think Foom is pretty unlikely—AI is likely to accelerate capabilities progress, but not to the degree of Foom. I generally think LLM-specific projections are a lot more useful than trying to e.g. extrapolate from chess algorithms and human evolution.

Discuss

Published on January 20, 2026 3:35 PM GMT

Epistemic status: This post is a synthesis of ideas that are, in my experience, widespread among researchers at frontier labs and in mechanistic interpretability, but rarely written down comprehensively in one place - different communities tend to know different pieces of evidence. The core hypothesis - that deep learning is performing something like tractable program synthesis - is not original to me (even to me, the ideas are ~3 years old), and I suspect it has been arrived at independently many times. (See the appendix on related work).

This is also far from finished research - more a snapshot of a hypothesis that seems increasingly hard to avoid, and a case for why formalization is worth pursuing. I discuss the key barriers and how tools like singular learning theory might address them towards the end of the post.

Thanks to Dan Murfet, Jesse Hoogland, Max Hennick, and Rumi Salazar for feedback on this post.

Sam Altman: Why does unsupervised learning work?

Dan Selsam: Compression. So, the ideal intelligence is called Solomonoff induction…[1]

The central hypothesis of this post is that deep learning succeeds because it's performing a tractable form of program synthesis - searching for simple, compositional algorithms that explain the data. If correct, this would reframe deep learning's success as an instance of something we understand in principle, while pointing toward what we would need to formalize to make the connection rigorous.

I first review the theoretical ideal of Solomonoff induction and the empirical surprise of deep learning's success. Next, mechanistic interpretability provides direct evidence that networks learn algorithm-like structures; I examine the cases of grokking and vision circuits in detail. Broader patterns provide indirect support: how networks evade the curse of dimensionality, generalize despite overparameterization, and converge on similar representations. Finally, I discuss what formalization would require, why it's hard, and the path forward it suggests.

Background

Whether we are a detective trying to catch a thief, a scientist trying to discover a new physical law, or a businessman attempting to understand a recent change in demand, we are all in the process of collecting information and trying to infer the underlying causes.

-Shane Legg[2]

Early in childhood, human babies learn object permanence - that unseen objects nevertheless persist even when not directly observed. In doing so, their world becomes a little less confusing: it is no longer surprising that their mother appears and disappears by putting hands in front of her face. They move from raw sensory perception towards interpreting their observations as coming from an external world: a coherent, self-consistent process which determines what they see, feel, and hear.

As we grow older, we refine this model of the world. We learn that fire hurts when touched; later, that one can create fire with wood and matches; eventually, that fire is a chemical reaction involving fuel and oxygen. At each stage, the world becomes less magical and more predictable. We are no longer surprised when a stove burns us or when water extinguishes a flame, because we have learned the underlying process that governs their behavior.

This process of learning only works because the world we inhabit, for all its apparent complexity, is not random. It is governed by consistent, discoverable rules. If dropping a glass causes it to shatter on Tuesday, it will do the same on Wednesday. If one pushes a ball off the top of a hill, it will roll down, at a rate that any high school physics student could predict. Through our observations, we implicitly reverse-engineer these rules.

This idea - that the physical world is fundamentally predictable and rule-based - has a formal name in computer science: the physical Church-Turing thesis. Precisely, it states that any physical process can be simulated to arbitrary accuracy by a Turing machine. Anything from a star collapsing to a neuron firing, can, in principle, be described by an algorithm and simulated on a computer.

From this perspective, one can formalize this notion of "building a world model by reverse-engineering rules from what we can see." We can operationalize this as a form of program synthesis: from observations, attempting to reconstruct some approximation of the "true" program that generated those observations. Assuming the physical Church-Turing thesis, such a learning algorithm would be "universal," able to eventually represent and predict any real-world process.

But this immediately raises a new problem. For any set of observations, there are infinitely many programs that could have produced them. How do we choose? The answer is one of the oldest principles in science: Occam's razor. We should prefer the simplest explanation.

In the 1960s, Ray Solomonoff formalized this idea into a theory of universal induction which we now call Solomonoff induction. He defined the "simplicity" of a hypothesis as the length of the shortest program that can describe it (a concept known as Kolmogorov complexity). An ideal Bayesian learner, according to Solomonoff, should prefer hypotheses (programs) that are short over ones that are long. This learner can, in theory, learn anything that is computable, because it searches the space of all possible programs, using simplicity as its guide to navigate the infinite search space and generalize correctly.

The invention of Solomonoff induction began[3] a rich and productive subfield of computer science, algorithmic information theory, which persists to this day. Solomonoff induction is still widely viewed as the ideal or optimal self-supervised learning algorithm, which one can prove formally under some assumptions[4]. These ideas (or extensions of them like AIXI) were influential for early deep learning thinkers like Jürgen Schmidhuber and Shane Legg, and shaped a line of ideas attempting to theoretically predict how smarter-than-human machine intelligence might behave, especially within AI safety.

Unfortunately, despite its mathematical beauty, Solomonoff induction is completely intractable. Vanilla Solomonoff induction is incomputable, and even approximate versions like speed induction are exponentially slow[5]. Theoretical interest in it as a "platonic ideal of learning" remains to this day, but practical artificial intelligence has long since moved on, assuming it to be hopelessly unfeasible.

Meanwhile, neural networks were producing results that nobody had anticipated.

This was not the usual pace of scientific progress, where incremental advances accumulate and experts see breakthroughs coming. In 2016, most Go researchers thought human-level play was decades away; AlphaGo arrived that year. Protein folding had resisted fifty years of careful work; AlphaFold essentially solved it[6] over a single competition cycle. Large language models began writing code, solving competition math problems, and engaging in apparent reasoning - capabilities that emerged from next-token prediction without ever being explicitly specified in the loss function. At each stage, domain experts (not just outsiders!) were caught off guard. If we understood what was happening, we would have predicted it. We did not.

The field's response was pragmatic: scale the methods that work, stop trying to understand why they work. This attitude was partly earned. For decades, hand-engineered systems encoding human knowledge about vision or language had lost to generic architectures trained on data. Human intuitions about what mattered kept being wrong. But the pragmatic stance hardened into something stronger - a tacit assumption that trained networks were intrinsically opaque, that asking what the weights meant was a category error.

At first glance, this assumption seemed to have some theoretical basis. If neural networks were best understood as "just curve-fitting" function approximators, then there was no obvious reason to expect the learned parameters to mean anything in particular. They were solutions to an optimization problem, not representations. And when researchers did look inside, they found dense matrices of floating-point numbers with no obvious organization.

But a lens that predicts opacity makes the same prediction whether structure is absent or merely invisible. Some researchers kept looking.

Looking insideGrokkingThe modular addition transformer from Power et al. (2022) learns to generalize rapidly (top), at the same time as Fourier modes in the weights appear (bottom right). Illustration by Pearce et al. (2023).

Power et al. (2022) train a small transformer on modular addition: given two numbers, output their sum mod 113. Only a fraction of the possible input pairs are used for training - say, 30% - with the rest held out for testing.

The network memorizes the training pairs quickly, getting them all correct. But on pairs it hasn't seen, it does no better than chance. This is unsurprising: with enough parameters, a network can simply store input-output associations without extracting any rule. And stored associations don't help you with new inputs.

Here's what's unexpected. If you keep training, despite the training loss already nearly as low as it can go, the network eventually starts getting the held-out pairs right too. Not gradually, either: test performance jumps from chance to near perfect over only a few thousand training steps.

So something has changed inside the network. But what? It was already fitting the training data; the data didn't change. There's no external signal that could have triggered the shift.

One way to investigate is to look at the weights themselves. We can do this at multiple checkpoints over training and ask: does something change in the weights around the time generalization begins?

It does. The weights early in training, during the memorization phase, don't have much structure when you analyze them. Later, they do. Specifically, if we look at the embedding matrix, we find that it's mapping numbers to particular locations on a circle. The number 0 maps to one position, 1 maps to a position slightly rotated from that, and so on, wrapping around. More precisely: the embedding of each number contains sine and cosine values at a small set of specific frequencies.

This structure is absent early in training. It emerges as training continues, and it emerges around the same time that generalization begins.

So what is this structure doing? Following it through the network reveals something unexpected: the network has learned an algorithm for modular addition based on trigonometry.[7]

A transformer trained on a modular addition task learns a compositional, human-interpretable algorithm. Reverse-engineered by Nanda et al. (2023). Image from Nanda et al. (2023).

The algorithm exploits how angles add. If you represent a number as a position on a circle, then adding two numbers corresponds to adding their angles. The network's embedding layer does this representation. Its middle layers then combine the sine and cosine values of the two inputs using trigonometric identities. These operations are implemented in the weights of the attention and MLP layers: one can read off coefficients that correspond to the terms in these identities.

Finally, the network needs to convert back to a discrete answer. It does this by checking, for each possible output c.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , how well c matches the sum it computed. Specifically, the logit for output c depends on cos(2πk(a+b−c)/P). This quantity is maximized when c equals a+bmodP - the correct answer. At that point the cosines at different frequencies all equal 1 and add constructively. For wrong answers, they point in different directions and cancel.

This isn't a loose interpretive gloss. Each piece - the circular embedding, the trig identities, the interference pattern - is concretely present in the weights and can be verified by ablations.

So here's the picture that emerges. During the memorization phase, the network solves the task some other way - presumably something like a lookup table distributed across its parameters. It fits the training data, but the solution doesn't extend. Then, over continued training, a different solution forms: this trigonometric algorithm. As the algorithm assembles, generalization happens. The two are not merely correlated; tracing the structure in the weights and the performance on held-out data, they move together.

What should we make of this? Here’s one reading: the difference between a network that memorizes and a network that generalizes is not just quantitative, but qualitative. The two networks have learned different kinds of things. One has stored associations. The other has found a method - a mechanistic procedure that happens to work on inputs beyond those it was trained on, because it captures something about the structure of the problem.

This is a single example, and a toy one. But it raises a question worth taking seriously. When networks generalize, is it because they've found something like an algorithm? And if so, what does that tell us about what deep learning is actually doing?

It's worth noting what was and wasn't in the training data. The data contained input-output pairs: "32 and 41 gives 73," and so on. It contained nothing about how to compute them. The network arrived at a method on its own.

And both solutions - the lookup table and the trigonometric algorithm - fit the training data equally well. The network's loss was already near minimal during the memorization phase. Whatever caused it to keep searching, to eventually settle on the generalizing algorithm instead, it wasn't that the generalizing algorithm fit the data better. It was something else - some property of the learning process that favored one kind of solution over another.

The generalizing algorithm is, in a sense, simpler. It compresses what would otherwise be thousands of stored associations into a compact procedure. Whether that's the right way to think about what happened here - whether "simplicity" is really what the training process favors - is not obvious. But something made the network prefer a mechanistic solution that generalized over one that didn't, and it wasn't the training data alone.[8]

Vision circuitsInceptionV1 classifies an image as a car by hierarchically composing detectors for the windows, car body, and wheels (pictured), which are themselves formed by composing detectors for shapes, edges, etc (not pictured). From Olah et al. (2020).

Grokking is a controlled setting - a small network, a simple task, designed to be fully interpretable. Does the same kind of structure appear in realistic models solving realistic problems?

Olah et al. (2020) study InceptionV1, an image classification network trained on ImageNet - a dataset of over a million photographs labeled with object categories. The network takes in an image and outputs a probability distribution over a thousand possible labels: "car," "dog," "coffee mug," and so on. Can we understand this more realistic setting?

A natural starting point is to ask what individual neurons are doing. Suppose we take a neuron somewhere in the network. We can find images that make it activate strongly by either searching through a dataset or optimizing an input to maximize activation. If we collect images that strongly activate a given neuron, do they have anything in common?

In early layers, they do, and the patterns we find are simple. Neurons in the first few layers respond to edges at particular orientations, small patches of texture, transitions between colors. Different neurons respond to different orientations or textures, but many are selective for something visually recognizable.

In later layers, the patterns we find become more complex. Neurons respond to curves, corners, or repeating patterns. Deeper still, neurons respond to things like eyes, wheels, or windows - object parts rather than geometric primitives.

This already suggests a hierarchy: simple features early, complex features later. But the more striking finding is about how the complex features are built.

Olah et al. do not just visualize what neurons respond to. They trace the connections between layers - examining the weights that connect one layer's neurons to the next, identifying which earlier features contribute to which later ones. What they find is that later features are composed from earlier ones in interpretable ways.

There is, for instance, a neuron in InceptionV1 that we identify as responding to dog heads. If we trace its inputs by looking at which neurons from the previous layer connect to it with strong weights, we find it receives input from neurons that detect eyes, snout, fur, and tongue. The dog head detector is built from the outputs of simpler detectors. It is not detecting dog heads from scratch; it is checking whether the right combination of simpler features is present in the right spatial arrangement.

We find the same pattern throughout the network. A neuron that detects car windows is connected to neurons that detect rectangular shapes with reflective textures. A neuron that detects car bodies is connected to neurons that detect smooth, curved surfaces. And a neuron that detects cars as a whole is connected to neurons that detect wheels, windows, and car bodies, arranged in the spatial configuration we would expect for a car.

Olah et al. call these pathways "circuits," and the term is meaningful. The structure is genuinely circuit-like: there are inputs, intermediate computations, and outputs, connected by weighted edges that determine how features combine. In their words: "You can literally read meaningful algorithms off of the weights."

And the components are reused. The same edge detectors that contribute to wheel detection also contribute to face detection, to building detection, to many other things. The network has not built separate feature sets for each of the thousand categories it recognizes. It has built a shared vocabulary of parts - edges, textures, curves, object components, etc - and combines them differently for different recognition tasks.

We might find this structure reminiscent of something. A Boolean circuit is a composition of simple gates - each taking a few bits as input, outputting one bit - wired together to compute something complex. A program is a composition of simple operations - each doing something small - arranged to accomplish something larger. What Olah et al. found in InceptionV1 has the same shape: small computations, composed hierarchically, with components shared and reused across different pathways.

From a theoretical computer science perspective, this is what algorithms look like, in general. Not just the specific trigonometric trick from grokking, but computation as such. You take a hard problem, break it into pieces, solve the pieces, and combine the results. What makes this tractable, what makes it an algorithm rather than a lookup table, is precisely the compositional structure. The reuse is what makes it compact; the compactness is what makes it feasible.

Olsson et al. argue that the primary mechanism of in-context-learning in large language models is a mechanistic attention circuit known as an induction head. Similar to the grokking example, the mechanistic circuit forms in a rapid "phase change" which coincides with a large improvement in the in-context-learning performance. Plots from Olsson et al.

Grokking and InceptionV1 are two examples, but they are far from the only ones. Mechanistic interpretability has grown into a substantial field, and the researchers working in it have documented many such structures - in toy models, in language models, across different architectures and tasks. Induction heads, language circuits, and bracket matching in transformer language models, learned world models and multi-step reasoning in toy tasks, grid-cell-like mechanisms in RL agents, hierarchical representations in GANs, and much more. Where we manage to look carefully, we tend to find something mechanistic.

This raises a question. If what we find inside trained networks (at least when we can find anything) looks like algorithms built from parts, what does that suggest about what deep learning is doing?

The hypothesis

What should we make of this?

We have seen neural networks learn solutions that look like algorithms - compositional structures built from simple, reusable parts. In the grokking case, this coincided precisely with generalization. In InceptionV1, this structure is what lets the network recognize objects despite the vast dimensionality of the input space. And across many other cases documented in the mechanistic interpretability literature, the same shape appears: not monolithic black-box computations, but something more like circuits.

This is reminiscent of the picture we started with. Solomonoff induction frames learning as a search for simple programs that explain data. It is a theoretical ideal - provably optimal in a certain sense, but hopelessly intractable. The connection between Solomonoff and deep learning has mostly been viewed as purely conceptual: a nice way to think about what learning "should" do, with no implications for what neural networks actually do.

But the evidence from mechanistic interpretability suggests a different possibility. What if deep learning is doing something functionally similar to program synthesis? Not through the same mechanism - gradient descent on continuous parameters is nothing like enumerative search over discrete programs. But perhaps targeting the same kind of object: mechanistic solutions, built from parts, that capture structure in the data generating process.

To be clear: this is a hypothesis. The evidence shows that neural networks can learn compositional solutions, and that such solutions have appeared alongside generalization in specific, interpretable cases. It doesn't show that this is what's always happening, or that there's a consistent bias toward simplicity, or that we understand why gradient descent would find such solutions efficiently.

But if the hypothesis is right, it would reframe what deep learning is doing. The success of neural networks would not be a mystery to be accepted, but an instance of something we already understand in principle: the power of searching for compact, mechanistic models to explain your observations. The puzzle would shift from "why does deep learning work at all?" to "how does gradient descent implement this search so efficiently?"

That second question is hard. Solomonoff induction is intractable precisely because the space of programs is vast and discrete. Gradient descent navigates a continuous parameter space using only local information. If both processes are somehow arriving at similar destinations - compositional solutions to learning problems - then something interesting is happening in how neural network loss landscapes are structured, something we do not yet understand. We will return to this issue at the end of the post.

So the hypothesis raises as many questions as it answers. But it offers something valuable: a frame. If deep learning is doing a form of program synthesis, that gives us a way to connect disparate observations - about generalization, about convergence of representations, about why scaling works - into a coherent picture. Whether this picture can make sense of more than just these particular examples is what we'll explore next.

Clarifying the hypothesis

What do I mean by “programs”?

I think one can largely read this post with a purely operational, “you know it when you see it” definition of “programs” and “algorithms”. But there are real conceptual issues here if you try to think about this carefully.

In most computational systems, there's a vocabulary that comes with the design - instructions, subroutines, registers, data flow, and so on. We can point to the “program” because the system was built to make it visible.

Neural networks are not like this. We have neurons, weights, activations, etc, but these may not be the right atoms of computation. If there's computational structure in a trained network, it doesn't automatically come labeled. So if we want to ask whether networks learn programs, we need to know what we're looking for. What would count as finding one?

This is a real problem for interpretability too. When researchers claim to find "circuits" or “features” in a network, what makes that a discovery rather than just a pattern they liked? There has to be something precise and substrate-independent we're tracking. It helps to step back and consider what computational structure even is in the cases we understand it well.

Consider the various models of computation: Turing machines, lambda calculus, Boolean circuits, etc. They have different primitives - tapes, substitution rules, logic gates - but the Church-Turing thesis tells us they're equivalent. Anything computable in one is computable in all the others. So "computation" isn't any particular formalism. It's whatever these formalisms have in common.

What do they have in common? Let me point to something specific: each one builds complex operations by composing simple pieces, where each piece only interacts with a small number of inputs. A Turing machine's transition function looks at one cell. A Boolean gate takes two or three bits. A lambda application involves one function and one argument. Complexity comes from how pieces combine, not from any single piece seeing the whole problem.

Is this just a shared property, or something deeper?

One reason to take it seriously: you can derive a complete model of computation from just this principle. Ask "what functions can I build by composing pieces of bounded arity?" and work out the answer carefully. You get (in the discrete case) Boolean circuits - not a restricted fragment of computation, but a universal model, equivalent to all the others. The composition principle alone is enough to generate computation in full generality.

The bounded-arity constraint is essential. If each piece could see all inputs, we would just have lookup tables. What makes composition powerful is precisely that each piece is “local” and can only interact with so many things at once - it forces solutions to have genuine internal structure.

So when I say networks might learn "programs," I mean: solutions built by composing simple pieces, each operating on few inputs. Not because that's one nice kind of structure, but because that may be what computation actually is.

Note that we have not implied that the computation is necessarily over discrete values - it may be over continuous values, as in analog computation. (However, the “pieces” must be discrete, for this to even be a coherent notion. This causes issues when combined with the subsequent point, as we will discuss towards the end of the post.)

A clarification: the network's architecture trivially has compositional structure - the forward pass is executable on a computer. That's not the claim. The claim is that training discovers an effective program within this substrate. Think of an FPGA: a generic grid of logic components that a hardware engineer configures into a specific circuit. The architecture is the grid; the learned weights are the configuration.

This last point, the fact that the program structure in neural networks is learned and depends on continuous parameters, is actually what makes this issue rather subtle, and unlike other models of computation we’re familiar with (even analog computation). This is a subtle issue which makes formalization difficult, an issue we will return to towards the end of the post.

What do I mean by “program synthesis”?

By program synthesis, I mean a search through possible programs to find one that fits the data.

Two things make this different from ordinary function fitting.

First, the search is general-purpose. Linear regression searches over linear functions. Decision trees search over axis-aligned partitions. These are narrow hypothesis classes, chosen by the practitioner to match the problem. The claim here is different: deep learning searches over a space that can express essentially any efficient computable function. It's not that networks are good at learning one particular kind of structure - it's that they can learn whatever structure is there.

Second, the search is guided by strong inductive biases. Searching over all programs is intractable without some preference for certain programs over others. The natural candidate is simplicity: favor shorter or less complex programs over longer or more complex ones. This is what Solomonoff induction does - it assigns prior probability to programs based on their length, then updates on data.

Solomonoff induction is the theoretical reference point. It's provably optimal in a certain sense: if the data has any computable structure, Solomonoff induction will eventually find it. But it's also intractable - not just slow, but literally incomputable in its pure form, and exponentially slow even in approximations.

The hypothesis is that deep learning achieves something functionally similar through completely different means. Gradient descent on continuous parameters looks nothing like enumeration over discrete programs. But perhaps both are targeting the same kind of object - simple programs that capture structure - and arriving there by different routes. We will return to the issue towards the end of the post.

This would require the learning process to implement something like simplicity bias, even though "program complexity" isn't in the loss function. Whether that's exactly the right characterization, I'm not certain. But some strong inductive bias has to be operating - otherwise we couldn't explain why networks generalize despite having the capacity to memorize, or why scaling helps rather than hurts.

What’s the scope of the hypothesis?

I've thought most deeply about supervised and self-supervised learning using stochastic optimization (SGD, Adam, etc) on standard architectures like MLPs, CNNs, or transformers, on standard tasks like image classification or autoregressive language prediction, and am strongly ready to defend claims there. I also believe that this extends to settings like diffusion models, adversarial setups, reinforcement learning, etc, but I've thought less about these and can't be as confident here.

Why this isn't enough

The preceding case studies provide a strong existence proof: deep neural networks are capable of learning and implementing non-trivial, compositional algorithms. The evidence that InceptionV1 solves image classification by composing circuits, or that a transformer solves modular addition by discovering a Fourier-based algorithm, is quite hard to argue with. And, of course, there are more examples than these which we have not discussed.

Still, the question remains: is this the exception or the rule? It would be completely consistent with the evidence presented so far for this type of behavior to just be a strange edge case.

Unfortunately, mechanistic interpretability is not yet enough to settle the question. The settings where today's mechanistic interpretability tools provide such clean, complete, and unambiguously correct results[9] are very rare.

Aren't most networks uninterpretable? Why this doesn't disprove the thesis.

Should we not take the lack of such clean mechanistic interpretability results as active counterevidence against our hypothesis? If models were truly learning programs in general, shouldn't those programs be readily apparent? Instead the internals of these systems appear far more "messy."

This objection is a serious one, but it makes a leap in logic. It conflates the statement "our current methods have not found a clean programmatic structure" with the much stronger statement "no such structure exists." In other words, absence of evidence is not evidence of absence[10]. The difficulty we face may not be an absence of structure, but a mismatch between the network's chosen representational scheme and the tools we are currently using to search for it.

Attempting to identify which individual transistors in an Atari machine are responsible for different games does not work very well; nevertheless an Atari machine has real computational structure. We may be in a similar situation with neural networks. From Jonas & Kording (2017).

To make this concrete, consider a thought experiment, adapted from the paper "Could a Neuroscientist Understand a Microprocessor?":

Imagine a team of neuroscientists studying a microprocessor (MOS 6502) that runs arcade (Atari) games. Their tools are limited to their trade: they can, for instance, probe the voltage of individual transistors and lesion them to observe the effect on gameplay. They do not have access to the high-level source code or architecture diagrams.

As the paper confirms, the neuroscientists would fail to understand the system. This failure would not be because the system lacks compositional, program structure - it is, by definition, a machine that executes programs. Their failure would be one of mismatched levels of abstraction. The meaningful concepts of the software (subroutines, variables, the call stack) have no simple, physical correlate at the transistor level. The "messiness" they would observe - like a single transistor participating in calculating a score, drawing a sprite, and playing a sound - is an illusion created by looking at the wrong organizational level.

My claim is that this is the situation we face with neural networks. Apparent "messiness" like polysemanticity is not evidence against a learned program; it is the expected signature of a program whose logic is not organized at the level of individual neurons. The network may be implementing something like a program, but using a "compiler" and an "instruction set" that are currently alien to us.[11]

The clean results from the vision and modular addition case studies are, in my view, instances where strong constraints (e.g., the connection sparsity of CNNs, or the heavy regularization and shallow architecture in the grokking setup) forced the learned program into a representation that happened to be unusually simple for us to read. They are the exceptions in their legibility, not necessarily in their underlying nature.[12]

Therefore, while mechanistic interpretability can supply plausibility to our hypothesis, we need to move towards more indirect evidence to start building a positive case.

Indirect evidence

Just before OpenAI started, I met Ilya [Sutskever]. One of the first things he said to me was, "Look, the models, they just wanna learn. You have to understand this. The models, they just wanna learn."

And it was a bit like a Zen Koan. I listened to this and I became  enlightened.

... What that told me is that the phenomenon that I'd seen wasn't just some random thing: it was broad, it was more general.

The models just wanna learn. You get the obstacles out of their way. You give them good data. You give them enough space to operate in. You don't do something stupid like condition them badly numerically.

And they wanna learn. They'll do it.

-Dario Amodei[13]

I remember when I trained my first neural network, there was something almost miraculous about it: it could solve problems which I had absolutely no idea how to code myself (e.g. how to distinguish a cat from a dog), and in a completely opaque way such that even after it had solved the problem I had no better picture for how to solve the problem myself than I did beforehand. Moreover, it was remarkably resilient, despite obvious problems with the optimizer, or bugs in the code, or bad training data - unlike any other engineered system I had ever built, almost reminiscent of something biological in its robustness.

My impression is that this sense of "magic" is a common, if often unspoken, experience among practitioners. Many simply learn to accept the mystery and get on with the work. But there is nothing virtuous about confusion - it just suggests that your understanding is incomplete, that you are ignorant of the real mechanisms underlying the phenomenon.

Our practical success with deep learning has outpaced our theoretical understanding. This has led to a proliferation of explanations that often feel ad-hoc and local - tailor-made to account for a specific empirical finding, without connecting to other observations or any larger framework. For instance, the theory of "double descent" provides a narrative for the U-shaped test loss curve, but it is a self-contained story. It does not, for example, share a conceptual foundation with the theories we have for how induction heads form in transformers. Each new discovery seems to require a new, bespoke theory. One naturally worries that we are juggling epicycles.

This sense of theoretical fragility is compounded by a second problem: for any single one of these phenomena, we often lack consensus, entertaining multiple, competing hypotheses. Consider the core question of why neural networks generalize. Is it best explained by the implicit bias of SGD towards flat minima, the behavior of neural tangent kernels, or some other property? The field actively debates these views. And where no mechanistic theory has gained traction, we often retreat to descriptive labels. We say complex abilities are an "emergent" property of scale, a term that names the mystery without explaining its cause.

This theoretical disarray is sharpest when we examine our most foundational frameworks. Here, the issue is not just a lack of consensus, but a direct conflict with empirical reality. This disconnect manifests in several ways:

• Sometimes, our theories make predictions that are actively falsified by practice. Classical statistical learning theory, with its focus on the bias-variance tradeoff, advises against the very scaling strategies that have produced almost all state-of-the-art performance.
• In other cases, a theory might be technically true but practically misleading, failing to explain the key properties that make our models effective. The Universal Approximation Theorem, for example, guarantees representational power but does so via a construction that implies an exponential scaling that our models somehow avoid.
• And in yet other areas, our classical theories are almost entirely silent. They offer no framework to even begin explaining deep puzzles like the uncanny convergence of representations across vastly different models trained on the same data.

We are therefore faced with a collection of major empirical findings where our foundational theories are either contradicted, misleading, or simply absent. This theoretical vacuum creates an opportunity for a new perspective.

The program synthesis hypothesis offers such a perspective. It suggests we shift our view of what deep learning is fundamentally doing: from statistical function fitting to program search. The specific claim is that deep learning performs a search for simple programs that explain the data.

This shift in viewpoint may offer a way to make sense of the theoretical tensions we have outlined. If the learning process is a search for an efficient program rather than an arbitrary function, then the circumvention of the curse of dimensionality is no longer so mysterious. If this search is guided by a strong simplicity bias, the unreasonable effectiveness of scaling becomes an expected outcome, rather than a paradox.

We will now turn to the well-known paradoxes of approximation, generalization, and convergence, and see how the program synthesis hypothesis accounts for each.

The paradox of approximation

(See also this post for related discussion.)

We can overcome the curse of dimensionality because real problems can be broken down into parts. When this happens sequentially (like the trees on the right) deep networks have an advantage. Image source.

Before we even consider how a network learns or generalizes, there is a more basic question: how can a neural network, with a practical number of parameters, even in principle represent the complex function it is trained on?

Consider the task of image classification. A function that takes a 1024x1024 pixel image (roughly one million input dimensions) and maps it to a single label like "cat" or "dog" is, a priori, an object of staggering high-dimensional complexity. Who is to say that a good approximation of this function even exists within the space of functions that a neural network of a given size can express?

The textbook answer to this question is the Universal Approximation Theorem (UAT). This theorem states that a neural network with a single hidden layer can, given enough neurons, approximate any continuous function to arbitrary accuracy. On its face, this seems to resolve the issue entirely.

A precise statement of the Universal Approximation Theorem

Let σ be a continuous, non-polynomial function. Then for every continuous function f from a compact subset of Rn to Rm, and some 0">ε>0, we can choose the number of neurons k large enough such that there exists a network g with

supx∥f(x)−g(x)∥<ε

where g(x)=C⋅(σ∘(A⋅x+b)) for some matrices A∈Rk×n, b∈Rk, and C∈Rm×k.

See here for a proof sketch. In plain English, this means that for any well-behaved target function f, you can always make a one-layer network g that is a "good enough" approximation, just by making the number of neurons k sufficiently large.

Note that the network here is a shallow one - the theorem doesn't even explain why you need deep networks, an issue we'll return to when we talk about depth separations. In fact, one can prove theorems like this without even needing neural networks at all - the theorem directly parallels the classic Stone-Weierstrass theorem from analysis, which proves a similar statement for polynomials.

However, this answer is deeply misleading. The crucial caveat is the phrase "given enough neurons." A closer look at the proofs of the UAT reveals that for an arbitrary function, the number of neurons required scales exponentially with the dimension of the input. This is the infamous curse of dimensionality. To represent a function on a one-megapixel image, this would require a catastrophically large number of neurons - more than there are atoms in the universe.

The UAT, then, is not a satisfying explanation. In fact, it's a mathematical restatement of a near-trivial fact: with exponential resources, one can simply memorize a function's behavior. The constructions used to prove the theorem are effectively building a continuous version of a lookup table. This is not an explanation for the success of deep learning; it is a proof that if deep learning had to deal with arbitrary functions, it would be hopelessly impractical.

This is not merely a weakness of the UAT's particular proof; it is a fundamental property of high-dimensional spaces. Classical results in approximation theory show that this exponential scaling is not just an upper bound on what's needed, but a strict lower bound. These theorems prove that any method that aims to approximate arbitrary smooth functions is doomed to suffer the curse of dimensionality.

The parameter count lower bound

There are many results proving various lower bounds on the parameter count available in the literature under different technical assumptions.

A classic result from DeVore, Howard, and Micchelli (1989) [Theorem 4.2] establishes a lower bound on the number of parameters n required by any continuous approximation scheme (including neural networks) to achieve an error ε over the space of all smooth functions in d dimensions. The number of parameters n must satisfy:

n≳ε−d/r

where r is a measure of the function's smoothness. To maintain a constant error ε as the dimension d increases, the number of parameters n must grow exponentially. This confirms that no clever trick can escape this fate if the target functions are arbitrary.

The real lesson of the Universal Approximation Theorem, then, is not that neural networks are powerful. The real lesson is that if the functions we learn in the real world were arbitrary, deep learning would be impossible. The empirical success of deep learning with a reasonable number of parameters is therefore a profound clue about the nature of the problems themselves: they must have structure.

The program synthesis hypothesis gives a name to this structure: compositionality. This is not a new idea. It is the foundational principle of computer science. To solve a complex problem, we do not write down a giant lookup table that specifies the output for every possible input. Instead, we write a program: we break the problem down hierarchically into a sequence of simple, reusable steps. Each step (like a logic gate in a circuit) is a tiny lookup table, and we achieve immense expressive power by composing them.

This matches what we see empirically in some deep neural networks via mechanistic interpretability. They appear to solve complex tasks by learning a compositional hierarchy of features. A vision model learns to detect edges, which are composed into shapes, which are composed into object parts (wheels, windows), which are finally composed into an object detector for a "car." The network is not learning a single, monolithic function; it is learning a program that breaks the problem down.

This parallel with classical computation offers an alternative perspective on the approximation question. While the UAT considers the case of arbitrary functions, a different set of results examines how well neural networks can represent functions that have this compositional, programmatic structure.

One of the most relevant results comes from considering Boolean circuits, which are a canonical example of programmatic composition. It is known that feedforward neural networks can represent any program implementable by a polynomial-size Boolean circuit, using only a polynomial number of neurons. This provides a different kind of guarantee than the UAT. It suggests that if a problem has an efficient programmatic solution, then an efficient neural network representation of that solution also exists.

This offers an explanation for how neural networks might evade the curse of dimensionality. Their effectiveness would stem not from an ability to represent any high-dimensional function, but from their suitability for representing the tiny, structured subset of functions that have efficient programs. The problems seen in practice, from image recognition to language translation, appear to belong to this special class.

Why compositionality, specifically? Evidence from depth separation results.

The argument so far is that real-world problems must have some special "structure" to escape the curse of dimensionality, and that this structure is program structure or compositionality. But how can we be sure? Yes, approximation theory requires that we must have something that differentiates our target functions from arbitrary smooth functions in order to avoid needing exponentially many parameters, but it does not specify what. The structure does not necessarily have to be compositionality; it could be something else entirely.

While there is no definitive proof, the literature on depth separation theorems provides evidence for the compositionality hypothesis. The logic is straightforward: if compositionality is the key, then an architecture that is restricted in its ability to compose operations should struggle. Specifically, one would expect that restricting a network's depth - its capacity for sequential, step-by-step computation - should force it back towards exponential scaling for certain problems.

And this is what the theorems show.

These depth separation results, sometimes also called "no-flattening theorems," involve constructing families of functions that deep neural networks can represent with a polynomial number of parameters, but which shallow networks would require an exponential number to represent. The literature contains a range of such functions, including sawtooth functions, certain polynomials, and functions with hierarchical or modular substructures.

Individually, many of these examples are mathematical constructions, too specific to tell us much about realistic tasks on their own. But taken together, a pattern emerges. The functions where depth provides an exponential advantage are consistently those that are built "step-by-step." They have a sequential structure that deep networks can mirror. A deep network can compute an intermediate result in one layer and then feed that result into the next, effectively executing a multi-step computation.

A shallow network, by contrast, has no room for this kind of sequential processing. It must compute its output in a single, parallel step. While it can still perform "piece-by-piece" computation (which is what its width allows), it cannot perform "step-by-step" computation. Faced with an inherently sequential problem, a shallow network is forced to simulate the entire multi-step computation at once. This can be highly inefficient, in the same way that simulating a sequential program on a highly parallel machine can sometimes require exponentially more resources.

This provides a parallel to classical complexity theory. The distinction between depth and width in neural networks mirrors the distinction between sequential (P) and parallelizable (NC) computation. Just as it is conjectured that some problems are inherently sequential and cannot be efficiently parallelized (the NC ≠ P conjecture), these theorems show that some functions are inherently deep and cannot be efficiently "flattened" into a shallow network.

The paradox of generalization

(See also this post for related discussion.)

Perhaps the most jarring departure from classical theory comes from how deep learning models generalize. A learning algorithm is only useful if it can perform well on new, unseen data. The central question of statistical learning theory is: what are the conditions that allow a model to generalize?

The classical answer is the bias-variance tradeoff. The theory posits that a model's error can be decomposed into two main sources:

• Bias: Error from the model being too simple to capture the underlying structure of the data (underfitting).
• Variance: Error from the model being too sensitive to the specific training data it saw, causing it to fit noise (overfitting).

According to this framework, learning is a delicate balancing act. The practitioner's job is to carefully choose a model of the "right" complexity - not too simple, not too complex -to land in a "Goldilocks zone" where both bias and variance are low. This view is reinforced by principles like the "no free lunch" theorems, which suggest there is no universally good learning algorithm, only algorithms whose inductive biases are carefully chosen by a human to match a specific problem domain.

The clear prediction from this classical perspective is that naively increasing a model's capacity (e.g., by adding more parameters) far beyond what is needed to fit the training data is a recipe for disaster. Such a model should have catastrophically high variance, leading to rampant overfitting and poor generalization.

And yet, perhaps the single most important empirical finding in modern deep learning is that this prediction is completely wrong. The "bitter lesson," as Rich Sutton calls it, is that the most reliable path to better performance is to scale up compute and model size, sometimes far into the regime where the model can easily memorize the entire training set. This goes beyond a minor deviation from theoretical predictions: it is a direct contradiction of the theory's core prescriptive advice.

This brings us to a second, deeper puzzle, first highlighted by Zhang et al. (2017). The authors conduct a simple experiment:

• They train a standard vision model on a real dataset (e.g., CIFAR-10) and confirm that it generalizes well.
• They then train the exact same model, with the exact same architecture, optimizer, and regularization, on a corrupted version of the dataset where the labels have been completely randomized.

The network is expressive enough that it is able to achieve near-zero training error on the randomized labels, perfectly memorizing the nonsensical data. As expected, its performance on a test set is terrible - it has learned nothing generalizable.

The paradox is this: why did the same exact model generalize well on the real data? Classical theories often tie a model's generalization ability to its "capacity" or "complexity," which is a fixed property of its architecture related to its expressivity. But this experiment shows that generalization is not a static property of the model. It is a dynamic outcome of the interaction between the model, the learning algorithm, and the structure of the data itself. The very same network that is completely capable of memorizing random noise somehow "chooses" to find a generalizable solution when trained on data with real structure. Why?

The program synthesis hypothesis offers a coherent explanation for both of these paradoxes.

First, why does scaling work? The hypothesis posits that learning is a search through some space of programs, guided by a strong simplicity bias. In this view, adding more parameters is analogous to expanding the search space (e.g., allowing for longer or more complex programs). While this does increase the model's capacity to represent overfitting solutions, the simplicity bias acts as a powerful regularizer. The learning process is not looking for any program that fits the data; it is looking for the simplest program. Giving the search more resources (parameters, compute, data) provides a better opportunity to find the simple, generalizable program that corresponds to the true underlying structure, rather than settling for a more complex, memorizing one.

Second, why does generalization depend on the data's structure? This is a natural consequence of a simplicity-biased program search.

• When trained on real data, there exists a short, simple program that explains the statistical regularities (e.g., "cats have pointy ears and whiskers"). The simplicity bias of the learning process finds this program, and because it captures the true structure, it generalizes well.
• When trained on random labels, no such simple program exists. The only way to map the given images to the random labels is via a long, complicated, high-complexity program (effectively, a lookup table). Forced against its inductive bias, the learning algorithm eventually finds such a program to minimize the training loss. This solution is pure memorization and, naturally, fails to generalize.

If one assumes something like the program synthesis hypothesis is true, the phenomenon of data-dependent generalization is not so surprising. A model's ability to generalize is not a fixed property of its architecture, but a property of the program it learns. The model finds a simple program on the real dataset and a complex one on the random dataset, and the two programs have very different generalization properties.And there is some evidence that the mechanism behind generalization is not so unrelated to the other empirical phenomena we have discussed. We can see this in the grokking setting discussed earlier. Recall the transformer trained on modular addition:

• Initially, the model learns a memorization-based program. It achieves 100% accuracy on the training data, but its test accuracy is near zero. This is analogous to learning the "random label" dataset - a complex, non-generalizing solution.
• After extensive further training, driven by a regularizer that penalizes complexity (weight decay), the model's internal solution undergoes a "phase transition." It discovers the Fourier-based algorithm for modular addition.
• Coincident with the discovery of this algorithmic program (or rather, the removal of the memorization program, which occurs slightly later), test accuracy abruptly jumps to 100%.

The sudden increase in generalization appears to be the direct consequence of the model replacing a complex, overfitting solution with a simpler, algorithmic one. In this instance, generalization is achieved through the synthesis of a different, more efficient program.

The paradox of convergence

When we ask a neural network to solve a task, we specify what task we'd like it to solve, but not how it should solve the task - the purpose of learning is for it to find strategies on its own. We define a loss function and an architecture, creating a space of possible functions, and ask the learning algorithm to find a good one by minimizing the loss. Given this freedom, and the high-dimensionality of the search space, one might expect the solutions found by different models - especially those with different architectures or random initializations - to be highly diverse.

Instead, what we observe empirically is a strong tendency towards convergence. This is most directly visible in the phenomenon of representational alignment. This alignment is remarkably robust:

• It holds across different training runs of the same architecture, showing that the final solution is not a sensitive accident of the random seed.
• More surprisingly, it holds across different architectures. The internal activations of a Transformer and a CNN trained on the same vision task, for example, can often be mapped to one another with a simple linear transformation, suggesting they are learning not just similar input-output behavior, but similar intermediate computational steps.
• It even holds in some cases across modalities. Models like CLIP, trained to associate images with text, learn a shared representation space where the vector for a photograph of a dog is close to the vector for the phrase "a photo of a dog," indicating convergence on a common, abstract conceptual structure.

The mystery deepens when we observe parallels to biological systems. The Gabor-like filters that emerge in the early layers of vision networks, for instance, are strikingly similar to the receptive fields of neurons in the V1 area of the primate visual cortex. It appears that evolution and stochastic gradient descent, two very different optimization processes operating on very different substrates, have converged on similar solutions when exposed to the same statistical structure of the natural world.

One way to account for this is to hypothesize that the models are not navigating some undifferentiated space of arbitrary functions, but are instead homing in on a sparse set of highly effective programs that solve the task. If, following the physical Church-Turing thesis, we view the natural world as having a true, computable structure, then an effective learning process could be seen as a search for an algorithm that approximates that structure. In this light, convergence is not an accident, but a sign that different search processes are discovering similar objectively good solutions, much as different engineering traditions might independently arrive at the arch as an efficient solution for bridging a gap.

This hypothesis - that learning is a search for an optimal, objective program - carries with it a strong implication: the search process must be a general-purpose one, capable of finding such programs without them being explicitly encoded in its architecture. As it happens, an independent, large-scale trend in the field provides a great deal of data on this very point.

Rich Sutton's "bitter lesson" describes the consistent empirical finding that long-term progress comes from scaling general learning methods, rather than from encoding specific human domain knowledge. The old paradigm, particularly in fields like computer vision, speech recognition, or game playing, involved painstakingly hand-crafting systems with significant prior knowledge. For years, the state of the art relied on complex, hand-designed feature extractors like SIFT and HOG, which were built on human intuitions about what aspects of an image are important. The role of learning was confined to a relatively simple classifier that operated on these pre-digested features. The underlying assumption was that the search space was too difficult to navigate without strong human guidance.

The modern paradigm of deep learning has shown this assumption to be incorrect. Progress has come from abandoning these handcrafted constraints in favor of training general, end-to-end architectures with the brute force of data and compute. This consistent triumph of general learning over encoded human knowledge is a powerful indicator that the search process we are using is, in fact, general-purpose. It suggests that the learning algorithm itself, when given a sufficiently flexible substrate and enough resources, is a more effective mechanism for discovering relevant features and structure than human ingenuity.

This perspective helps connect these phenomena, but it also invites us to refine our initial picture. First, the notion of a single "optimal program" may be too rigid. It is possible that what we are observing is not convergence to a single point, but to a narrow subset of similarly structured, highly-efficient programs. The models may be learning different but algorithmically related solutions, all belonging to the same family of effective strategies.

Second, it is unclear whether this convergence is purely a property of the problem's solution space, or if it is also a consequence of our search algorithm. Stochastic gradient descent is not a neutral explorer. The implicit biases of stochastic optimization, when navigating a highly over-parameterized loss landscape, may create powerful channels that funnel the learning process toward a specific kind of simple, compositional solution. Perhaps all roads do not lead to Rome, but the roads to Rome are the fastest. The convergence could therefore be a clue about the nature of our learning dynamics themselves - that they possess a strong, intrinsic preference for a particular class of solutions.

Viewed together, these observations suggest that the space of effective solutions for real-world tasks is far smaller and more structured than the space of possible models. The phenomenon of convergence indicates that our models are finding this structure. The bitter lesson suggests that our learning methods are general enough to do so. The remaining questions point us toward the precise nature of that structure and the mechanisms by which our learning algorithms are so remarkably good at finding it.

The path forward

If you've followed the argument this far, you might already sense where it becomes difficult to make precise. The mechanistic interpretability evidence shows that networks can implement compositional algorithms. The indirect evidence suggests this connects to why they generalize, scale, and converge. But "connects to" is doing a lot of work. What would it actually mean to say that deep learning is some form of program synthesis?

Trying to answer this carefully leads to two problems. The claim "neural networks learn programs" seems to require saying what a program even is in a space of continuous parameters. It also requires explaining how gradient descent could find such programs efficiently, given what we know about the intractability of program search.

These are the kinds of problems where the difficulty itself is informative. Each has a specific shape - what you need to think about, what a resolution would need to provide. I focus on them deliberately: that shape is what eventually pointed me toward specific mathematical tools I wouldn't have considered otherwise.

This is also where the post will shift register. The remaining sections sketch the structure of these problems and gesture at why certain mathematical frameworks (singular learning theory, algebraic geometry, etc) might become relevant. I won't develop these fully here - that requires machinery far beyond the scope of a single blog post - but I want to show why you'd need to leave shore at all, and what you might find out in open water.

The representation problem

The program synthesis hypothesis posits a relationship between two fundamentally different kinds of mathematical objects.

On one hand, we have programs. A program is a discrete and symbolic object. Its identity is defined by its compositional structure - a graph of distinct operations. A small change to this structure, like flipping a comparison or replacing an addition with a subtraction, can create a completely different program with discontinuous, global changes in behavior. The space of programs is discrete.

On the other hand, we have neural networks. A neural network is defined by its parameter space: a continuous vector space of real-valued weights. The function a network computes is a smooth (or at least piecewise-smooth) function of these parameters. This smoothness is the essential property that allows for learning via gradient descent, a process of infinitesimal steps along a continuous loss landscape.

This presents a seeming type mismatch: how can a continuous process in a continuous parameter space give rise to a discrete, structured program?

The problem is deeper than it first appears. To see why, we must first be precise about what we mean when we say a network has "learned a program." It cannot simply be about the input-output function the network computes. A network that has perfectly memorized a lookup table for modular addition computes the same function on a finite domain as a network that has learned the general, trigonometric algorithm. Yet we would want to say, emphatically, that they have learned different programs. The program is not just the function; it is the underlying mechanism.

Thus the notion must depend on parameters, and not just functions, presenting a further conceptual barrier. To formalize the notion of "mechanism," a natural first thought might be to partition the continuous parameter space into discrete regions. In this picture, all the parameter vectors within a region WA would correspond to the same program A, while vectors in a different region WB would correspond to program B. But this simple picture runs into a subtle and fatal problem: the very smoothness that makes gradient descent possible works to dissolve any sharp boundaries between programs.

Imagine a continuous path in parameter space from a point wA∈WA (which clearly implements program A) to a point wB∈WB (which clearly implements program B). Imagine, say, that A has some extra subroutine that B does not. Because the map from parameters to the function is smooth, the network's behavior must change continuously along this path. At what exact point on this path did the mechanism switch from A to B? Where did the new subroutine get added? There is no canonical place to draw a line. A sharp boundary would imply a discontinuity that the smoothness of the map from parameters to functions seems to forbid.

This is not so simple a problem, and it is worth spending some time thinking about how you might try to resolve it to appreciate that.

What this suggests, then, is that for the program synthesis hypothesis to be a coherent scientific claim, it requires something that does not yet exist: a formal, geometric notion of a space of programs. This is a rather large gap to fill, and in some ways, this entire post is my long-winded way of justifying such an ambitious mathematical goal.

I won't pretend that my collaborators and I don't have our[14] own ideas about how to resolve this, but the mathematical sophistication required jumps substantially, and they would probably require their own full-length post to do justice. For now, I will just gesture at some clues which I think point in the right direction.

The first is the phenomenon of degeneracies[15]. Consider, for instance, dead neurons, whose incoming weights and activations are such that the neurons never fires for any input. A neural network with dead neurons acts like a smaller network with those dead neurons removed. This gives a mechanism for neural networks to change their "effective size" in a parameter-dependent way, which is required in order to e.g. dynamically add or remove a new subroutine depending on where you are in parameter space, as in our example above. In fact dead neurons are just one example in a whole zoo of degeneracies with similar effects, which seem incredibly pervasive in neural networks.

It is worth mentioning that the present picture is now highly suggestive of a specific branch of math known as algebraic geometry. Algebraic geometry (in particular, singularity theory) systematically studies these degeneracies, and further provides a bridge between discrete structure (algebra) and continuous structure (geometry), exactly the type of connection we identified as necessary for the program synthesis hypothesis[16]. Furthermore, singular learning theory tells us how these degeneracies control the loss landscape and the learning process (classically, only in the Bayesian setting, a limitation we discuss in the next section). There is much more that can be said here, but I leave it for the future to treat this material properly.

The search problem

There’s another problem with this story. Our hypothesis is that deep learning is performing some version of program synthesis. That means that we not only have to explain how programs get represented in neural networks, we also need to explain how they get learned. There are two subproblems here.

• First, how can deep learning even implement the needed inductive biases? For deep learning algorithms to be implementing something analogous to Solomonoff induction, they must be able to implicitly follow inductive biases which depend on the program structure, like simplicity bias. That is, the optimization process must somehow be aware of the program structure in order to favor some types of programs (e.g. shorter programs) over others. The optimizer must “see” the program structure of parameters.
• Second, deep learning works in practice, using a reasonable amount of computational resources; meanwhile, even the most efficient versions of Solomonoff induction like speed induction run in exponential time or worse[5]. If deep learning is efficiently performing some version of program synthesis analogous to Solomonoff induction, that means it has implicitly managed to do what we could not figure out how to do explicitly - its efficiency must be due to some insight which we do not yet know. Of course, we know part of the answer: SGD only needs local information in order to optimize, instead of brute-force global search as one does with Bayesian learning. But then the mystery becomes a well-known one: why does myopic search like SGD converge to globally good solutions?

Both of these are questions about the optimization process. It is not obvious at all how local optimizers like SGD would be able to perform something like Solomonoff induction, let alone far more efficiently than we historically ever figured out for (versions of) Solomonoff induction itself. This is a difficult question, but I will attempt to point towards research which I believe can answer these questions.

The optimization process can depend on many things, a priori: choice of optimizer, regularization, dropout, step size, etc. But we can note that deep learning is able to work somewhat successfully (albeit sometimes with degraded performance) across wide ranges of choices of these variables. It does not seem like the choice of AdamW vs SGD matters nearly as much as the choice to do gradient-based learning in the first place. In other words, I believe these variables may affect efficiency, but I doubt they are fundamental to the explanation of why the optimization process can possibly succeed.

Instead, there is one common variable here which appears to determine the vast majority of the behavior of stochastic optimizers: the loss function. Optimizers like SGD take every gradient step according to a minibatch-loss function[17] like mean-squared error:

dwdt=−τdLdwL(w)=1nn∑i=1(yi−fw(xi))2

where w is the parameter vector, fw is the input/output map of the model on parameter w,(xi,yi) are the n training examples & labels, and τ is the learning rate.

In the most common versions of supervised learning, we can focus even further. The loss function itself can be decomposed into two effects: the parameter-function map w↦fw, and the target distribution. The overall loss function can be written as a composition of the parameter-function map and some statistical distance to the target distribution, e.g. for mean-squared error:

L(w)=ℓ∘f

where ℓ(g)=1/n∑ni=1(yi−g(xi))2.

Note that the statistical distance ℓ(g) here is a fairly simple object. Almost always the statistical distance here is (on function space) convex and with relatively simple functional form; further, it is the same distance one would use across many different architectures, including ones which do not achieve the remarkable performance of neural networks (e.g. polynomial approximation). Therefore one expects the question of learnability and inductive biases to largely come down to the parameter-function map fw rather than the (function-space) loss function ℓ(g).

If the above reasoning is correct, that means that in order to understand how SGD is able to potentially perform some kind of program synthesis, we merely need to understand properties of the parameter-function map. This would be a substantial simplification. Further, this relates learning dynamics to our earlier representation problem: the parameter-function map is precisely the same object responsible for the mystery discussed in the representation section.

This is not an airtight argument - it depends on the empirical question of whether one can ignore (or treat as second-order effects) other optimization details besides the loss function, and whether the handwave-y argument for the importance of the parameter-function map over the (function-space) loss is solid.

Even if one assumes this argument is valid, we have merely located the mystery, not resolved it. The question remains: what properties of the parameter-function map make targets learnable? At this point the reasoning becomes more speculative, but I will sketch some ideas.

The representation section concerned what structure the map encodes at each point in parameter space. Learnability appears to depend on something further: the structure of paths between points. Convexity of function-space loss implies that paths which are sufficiently straight in function space are barrier-free - roughly, if the endpoint is lower loss, the entire path is downhill. So the question becomes: which function-space paths does the map provide?

The same architectures successfully learn many diverse real-world targets. Whatever property of the map enables this, it must be relatively universal - not tailored to specific targets. This naturally leads us to ask: in what cases does the parameter-function map provide direct-enough paths to targets with certain structure, and characterizing what "direct enough" means.

This connects back to the representation problem. If the map encodes some notion of program structure, then path structure in parameter space induces relationships between programs - which programs are "adjacent," which are reachable from which. The representation section asks how programs are encoded as points; learnability asks how they are connected as paths. These are different aspects of the same object.

One hypothesis: compositional relationships between programs might correspond to some notion of “path adjacency” defined by the parameter-function map. If programs sharing structure are nearby - reachable from each other via direct paths - and if simpler programs lie along paths to more complex ones, then efficiency, simplicity bias, and empirically observed stagewise learning would follow naturally. Gradient descent would build incrementally rather than search randomly; the enumeration problem that dooms Solomonoff would dissolve into traversal.

This is speculative and imprecise. But there's something about the shape of what's needed that feels mathematically natural. The representation problem asks for a correspondence at the level of objects: strata in parameter space corresponding to programs. The search problem asks for something stronger - that this correspondence extends to paths. Paths in parameter space (what gradient descent traverses) should correspond to some notion of relationship or transition between programs.

This is a familiar move in higher mathematics (sometimes formalized by category theory): once you have a correspondence between two kinds of objects, you ask whether it extends to the relationships between those objects. It is especially familiar (in fields like higher category theory) to ask these kinds of questions when the "relationships between objects" take the form of paths in particular. I don't claim that existing machinery from these fields applies directly, and certainly not given the (lack of) detail I've provided in this post. But the question is suggestive enough to investigate: what should "adjacency between programs" mean? Does the parameter-function map induce or preserve such structure? And if so, what does this predict about learning dynamics that we could check empirically?

AppendixRelated work

The majority of the ideas in this post are not individually novel; I see the core value proposition as synthesizing them together in one place. The ideas I express here are, in my experience, very common among researchers at frontier labs, researchers in mechanistic interpretability, some researchers within science of deep learning, and others. In particular, the core hypothesis that deep learning is performing some tractable version of Solomonoff induction is not new, and has been written about many times. (However, I would not consider it to be a popular or accepted opinion within the machine learning field at large.) Personally, I have considered a version of this hypothesis for around three years. With this post, I aim to share a more comprehensive synthesis of the evidence for this hypothesis, as well as point to specific research directions for formalizing this idea.

Below is an incomplete list of what is known and published in various areas:

Existing comparisons between deep learning and program synthesis. The ideas surrounding Solomonoff induction have been highly motivating for many early AGI-focused researchers. Shane Legg (DeepMind cofounder) wrote his PhD thesis on Solomonoff induction; John Schulam (OpenAI cofounder) discusses the connection to deep learning explicitly here; Ilya Sutskever (OpenAI cofounder) has been giving talks on related ideas. There are a handful of places one can find a hypothesized connection between deep learning and Solomonoff induction stated explicitly, though I do not believe any of these were the first to do so. My personal experience is that such intuitions are fairly common among e.g. people working at frontier labs, even if they are not published in writing. I am not sure who had the idea first, and suspect it was arrived at independently multiple times.

Feature learning. It would not be accurate to say that the average ML researcher views deep learning as a complete black-box algorithm; it is well-accepted and uncontroversial that deep neural networks are able to extract "features" from the task which they use to perform well. However, it is a step beyond to claim that these features are actually extracted and composed in some mechanistic fashion resembling a computer program.

Compositionality, hierarchy, and modularity. My informal notion of "programs" here is quite closely related to compositionality. It is a fairly well-known hypothesis that supervised learning performs well due to compositional/hierarchical/modular structure in the model and/or the target task. This is particularly prominent within approximation theory (especially the literature on depth separations) as an explanation for the issues I highlighted in the "paradox of approximation" section.

Mechanistic interpretability. The (implicit) underlying premise of the field of mechanistic interpretability is that one can understand the internal mechanistic (read: program-like) structure responsible for a network's outputs. Mechanistic interpretability is responsible for discovering a significant number of examples of this type of structure, which I believe constitutes the single strongest evidence for the program synthesis hypothesis. I discuss a few case studies of this structure in the post, but there are possibly hundreds more examples which I did not cover, from the many papers within the field. A recent review can be found here.

Singular learning theory. In the “path forward” section, I highlight a possible role of degeneracies in controlling some kind of effective program structure. In some way (which I have gestured at but not elaborated on), the ideas presented in this post can be seen as motivating singular learning theory as a means to formally ground these ideas and produce practical tools to operationalize them. This is most explicit within a line of work within singular learning theory that attempts to precisely connect program synthesis with the singular geometry of a (toy) learning machine.

 

1. ^

   From the GPT-4.5 launch discussion, 38:46.

2. ^

   From his PhD thesis, pages 23-24.

3. ^

   Together with independent contributions by Kolmogorov, Chaitin, and Levin.

4. ^

   One must be careful, as some commonly stated "proofs" of this optimality are somewhat tautological. These typically go roughly something like: under the assumption that the data generating process has low Kolmogorov complexity, then Solomonoff induction is optimal. This is of course completely circular, since we have, in effect, assumed from the start that the inductive bias of Solomonoff induction is correct. Better proofs of this fact instead show a regret bound: on any sequence, Solomonoff induction's cumulative loss is at most a constant worse than any computable predictor - where the constant depends on the complexity of the competing predictor, not the sequence. This is a frequentist guarantee requiring no assumptions about the data source. See in particular Section 3.3.2 and Theorem 3.3 of this PhD thesis. Thanks to Cole Wyeth for pointing me to this argument.

5. ^

   See this paper.

6. ^

   Depending on what one means by "protein folding," one can debate whether the problem has truly been solved; for instance, the problem of how proteins fold dynamically over time is still open AFAIK. See this fairly well-known blog post by molecular biologist Mohammed AlQuraishi for more discussion, and why he believes calling AlphaFold a "solution" can be appropriate despite the caveats.

7. ^

   In fact, the solution can be seen as a representation-theoretic algorithm for the group of integers under addition mod P (the cyclic group CP). Follow-up papers demonstrated that neural networks also learn interpretable representation-theoretic algorithms for more general groups than cyclic groups.

8. ^

   For what it's worth, in this specific case, we do know what must be driving the process, if not the training loss: the regularization / weight decay. In the case of grokking, we do have decent understanding of how weight decay leads the training to prefer the generalizing solution. However, this explanation is limited in various ways, and it unclear how far it generalizes beyond this specific setting.

9. ^

   To be clear, one can still apply existing mechanistic interpretability tools to real language models and get productive results. But the results typically only manage to explain a small portion of the network, and in a way which is (in my opinion) less clean and convincing than e.g. Olah et al. (2020)'s reverse-engineering of InceptionV1.

10. ^

    This phrase is often abused - for instance, if you show up to court with no evidence, I can reasonably infer that no good evidence for your case exists. This is a gap between logical and heuristic/Bayesian reasoning. In the real world, if evidence for a proposition exists, it usually can and will be found (because we care about it), so you can interpret the absence of evidence for a proposition as suggesting that the proposition is false. However, in this case, I present a specific reason why one should not expect to see evidence even if the proposition in question is true.

11. ^

    Many interpretability researchers specifically believe in the linear representation hypothesis, that the variables of this program structure ("features") correspond to linear directions in activation space, or the stronger superposition hypothesis, that such directions form a sparse overbasis for activation space. One must be careful in interpreting these hypotheses as there are different operationalizations within the community; in my opinion, the more sophisticated versions are far more plausible than naive versions (thank you to Chris Olah for a helpful conversation here). Presently, I am skeptical that linear representations give the most prosaic description of a model's behavior or that this will be sufficient for complete reverse-engineering, but believe that the hypothesis is pointing at something real about models, and tools like SAEs can be helpful as long as one is aware of their limitations.

12. ^

    See for instance the results of these papers, where the authors incentivize spatial modularity with an additional regularization term. The authors interpret this as incentivizing modularity, but I would interpret it as incentivizing existing modularity to come to the surface.

13. ^

    From Dwarkesh Patel's podcast, 13:05.

14. ^

    The credit for these ideas should really go to Dan Murfet, as well as his current/former students including Will Troiani, James Clift, Rumi Salazar, and Billy Snikkers.

15. ^

    Let f(x|w) denote the output of the model on input x with parameters w. Formally, we say that a point in parameter space w∈W is degenerate or singular if there exists a tangent vector v∈TW such that the directional derivative ∇vf(x|w)=0 for all x. In other words, moving in some direction in parameter space doesn't change the behavior of the model (up to first order).

16. ^

    This is not as alien as it may seem. Note that this provides a perspective which connects nicely with both neural networks and classical computation. First consider, for instance, that the gates of a Boolean circuit literally define a system of equations over F2, whose solution set is an algebraic variety over F2. Alternatively, consider that a neural network with polynomial (or analytic) activation function defines a system of equations over R, whose vanishing set is an algebraic (respectively, analytic) variety over R. Of course this goes only a small fraction of the way to closing this gap, but one can start to see how this becomes plausible.

17. ^

    A frequent perspective is to write this minibatch-loss in terms of its mean (population) value plus some noise term. That is, we think of optimizers like SGD as something like “gradient descent plus noise.” This is quite similar to mathematical models like overdamped Langevin dynamics, though note that the noise term may not be Gaussian as in Langevin dynamics. It is an open question whether the convergence of neural network training is due to the population term or the noise term. (Note that this is a separate question as to whether the generalization / inductive biases of SGD-trained neural networks is due to the population term or the noise term.) I am tentatively of the belief (somewhat controversially) that both convergence and inductive bias is due to structure in the population loss rather than the noise term, but explaining my reasoning here is a bit out of scope.

Discuss

Published on January 20, 2026 2:27 PM GMT

2026 marks exactly 1 millennium since the last total solar eclipse visible from Table Mountain. The now famous (among people who sit behind me at work) eclipse of 1026 would’ve been visible to anyone at the top of Lion’s Head or Table Mountain and basically everywhere else in Cape Town. Including De Waal Park, where I’m currently writing this. I’ve hiked up Lion’s Head a lot and still find the view pretty damn awe inspiring. To have seen a total solar eclipse up there must have been absurdly damn awe inspiring. Maybe also terrifying if you didn’t know what was happening. But either way, I’m jealous of anyone that got to experience it. If you continued flipping through the exciting but predictable Five Millennium Canon of Solar Eclipses: -1999 to +3000 (2000 BCE to 3000 CE) by Jean Meeus and Fred Espenak, you’d notice something weird and annoying - you have to flip all the way to the year 2238 for the next total solar eclipse to hit Table Mountain.

Tim Urban has this idea of converting all of human history into a 1000 page book. He says that basically up until page 950 there’s just nothing going on.

“But if you look at Page 1,000—which, in this metaphor, Page 1,000 is the page that ends with today, so that goes from the early 1770s to today—that is nothing like any other page. It is completely an anomaly in the book. If you’re reading, if you’re this alien, this suddenly got incredibly interesting in the last 10 pages, but especially on this page. The alien is thinking, “OK, shit is going down.”

The gap between eclipses on Table Mountain is the real life version of this book. Imagine If aliens had put a secret camera where the cable car is, and it only popped up during a total solar eclipse, they’d see something like the island from Lost, then wait a hundred or thousand years then see the exact same thing but maybe it’s raining.

 

 

And they’d see this 4 more times.

 

 

 

Then they go to open the image from 2238 and suddenly.:

 

There’s a soccer stadium and also is that a city???

 

Just knowing the date of these eclipses has made the past and future feel much more real to me.

I saw the total solar eclipse of 2024 in the middle of an absolutely packed Klyde Warren Park in Dallas.

 

 

When totality started, there were barely any cars on the highway and the cars you could see suddenly had their headlights on. The office tower behind me was filled with people on every floor staring outside, all backlit by the lights which had suddenly turned on.

We talk about how the animals start going crazy because they think it’s night as though this doesn’t include us but actually we are so included here and go even crazier than any birds doing morning chirps. The extent to which the city of Dallas was turned upside down by this event is hard to believe. And it wasn’t just a physical transformation. The entire energy of the city felt different, not just compared to the day before but compared to any other city I’ve been in. I have never felt so connected to everyone around me and so optimistic and elated at the same time all while knowing everyone else feels the exact same way.

It’s hard to imagine what it must have been like to be a person in Cape Town in the year 1026. The image in my head feels murky and I guess pastoral. But imagining what it was like during a total solar eclipse in the year 1026, is much easier. I can picture myself on top of Lion’s Head or Table Mountain or on the beach in 1026. I can picture the people around me seeing it and wondering what’s going on. I can picture myself wondering what’s going on. Because even when you know what’s going on you’re still wondering what’s going on.

When I think about the eclipse of 2238 it’s even easier to connect with those people in that Cape Town. If the people of that time have anything like newspapers or radio or the internet or TikTok, I can imagine the literal hype and electricity in the air over the months and days and hours leading up to the eclipse. It’s also weird to briefly think about how everything I’m using now and consuming now is going to be considered ancient history by the lovely people that get to experience seeing an eclipse in 2238 at the top of Lion’s Head. My macbook which feels so fast and which I love so dearly - junk. TikTok would be like a wax cylinder record, and they’d wonder how people managed to code with an AI as silly as Opus-4.5 or worse by hand somehow. Every movie from 2026 would be older to them than the movie of the train going into the station, is to us. I don’t know how they are going to build software in the year 2238. I barely know how I built the website I used to find this stuff out. I’ve wanted to know when the next and previous eclipse are going to happen on Lion’s Head, since i got back from the eclipse in 2024.

I started by searching on google for something to find eclipses by location and not time. We have Five Millennium Canon of Solar Eclipses but this is still in order of time. The answer to my question felt like something we could easily work out with existing data and a for loop in whichever your favorite programming language is. NASA hosts a csv file with the aforementioned five millennia of past and future eclipses. So we just have to parse this csv and figure out what each of the 16 columns represented and then figure out how to do a for loop over the paths of the eclipses and find an intersection with the coordinates of Lion’s Head.

Luckily the year was 2024 or 5 A.G.T (Anno GPT3) - So I asked what would have probably been GPT-4 if it could search for the date of the next and previous eclipses, it used the search tool it had at the time, but it could not find anything. I tried this a few more times, usually whenever I finished a hike and a new model had been recently released. It’s never worked though. That is, until a week ago. This January I paid $200 for GPT 5.2 Pro after reading some, okay, a single, extremely positive review about it.. To be honest my review is: It kind of sucks, but still happy I paid the $200. This is because towards the end of the month I set 5.2 Pro to extended thinking then typed this prompt:

“How could I make an app that lets you pick a place on earth and then finds the last time or next time there was or will be a full solar eclipse there, what data sources would I use what algorithms and how accurate could I be.”

It thought for 17m and 6 seconds then replied with a whole bunch of words I didn’t understand. So I replied:

“Can you write a prototype in python?”

It thought for another 20m then replied with this script.

I pasted it into a file then ran it with the coordinates of Lion’s Head and saw the answer to my question: 1026. That was the last time a total solar eclipse was visible from Lion’s Head.

Since it was a script I could also use any coordinates on Earth and find the same answer for that place (as long it was in the five millennia catalogue)

I popped the python script in to Claude code with Opus set to 4.5, it did some verbing and then I got this website out a few hours later: https://findmyeclipse.com

In 2238 I somehow doubt the vast majority of people will ever think about code when creating things, in the same way I don’t think about binary or transistors when programming. What does a world where software can be written without any special knowledge look like, and then what does it look like after 100 years of that? I don’t have any answers but I do know one thing: The people of Cape Town in 2238 will know that this eclipse is not just a rare eclipse, but a rare eclipse among rare eclipses . They will look forward to it. They will write about the best places to see it from. I can imagine being a person in 2238 thinking, boy this eclipse would look sick from Lion’s Head. Thinking, I wonder if it’s going to be too busy up there. Maybe consider going up and camping on Table Mountain the night before. And I can imagine being in any one of these places or just in a packed De Waal Park preparing for totality and when I imagine myself there with everyone around me, it’s hard not to be optimistic.

 

 

1 Like

 

 

 

Discuss

Published on January 20, 2026 2:06 AM GMT

Fiora Sunshine's post, Why I Transitioned: A Case Study (the OP) articulates a valuable theory for why some MtFs transition.

If you are MtF and feel the post describes you, I believe you.

However, many statements from the post are wrong or overly broad.

My claims:

1. There is evidence of a biological basis for trans identity. Twin studies are a good way to see this.
    
2. Fiora claims that trans people's apparent lack of introspective clarity may be evidence of deception. But trans people are incentivized not to attempt to share accurate answers to "why do you really want to transition?". This is the Trans Double Bind.
    
3. I am a counterexample to Fiora's theory. I was an adolescent social outcast weeb but did not transition. I spent 14 years actualizing as a man, then transitioned at 31 only after becoming crippled by dysphoria. My example shows that Fiora's phenotype can co-occur with or mask medically significant dysphoria.

A. Biologically Transgender

In the OP, Fiora presents the "body-map theory" under the umbrella of "arcane neuro-psychological phenomena", and then dismisses medical theories because the body-map theory doesn't fit her friend group.

The body-map theory is a straw man for biological causation because there are significant sex differences between men and women that are (a) not learned and (b) not reducible to subconscious expectations about one's anatomy.

The easiest way to see this is CAH. To quote from Berenbaum and Beltz, 2021[1]:

Studies of females with congenital adrenal hyperplasia (CAH) show how prenatal androgens affect behavior across the life span, with large effects on gendered activity interests and engagement, moderate effects on spatial abilities, and relatively small (or no) effects on gender identity

The sex difference in people-vs-things interests (hobbies, occupations) has been discussed extensively in our community. CAH shifts females towards male-patterned interests with small effects on gender identity, without changes in anatomy.

This finding is also notable because it shows male-patterned interests and female gender identity can coexist, at least in natal females.

 

Twin Studies à la LLM

I'm trans so I have a motive to search for evidence that suggests I am ~biologically valid~ and not subject to some kind of psychosocial delusion. It would be easy for me to cherry-pick individual papers to support that view. I'm trying to not do that. I'm also not going to attempt a full literature review here. Luckily it is 2026, and we have a better option.

The ACE model from psychiatric genetics is a standard framework for decomposing the variance in a trait into 3 components:

A = Additive Genetics: cumulative effect of individual alleles

C = Common Environment: parents, schooling, SES, etc.

E = Nonshared Environment (+ error): randomness, idiosyncratic life events[2]

There are at least 9[3] primary twin studies on transgender identity or gender dysphoria. I created an LLM prompt[4] asking for a literature review with the goal of extracting signal, not just from the trans twin literature, but from other research that could help give us some plausible bounds on the strength of biological and social causation. Here are the results. The format is POINT_ESTIMATE, RANGE:

modelACEOpus 4.50.4, 0.2-0.60.05, 0-0.20.55, 0.35-0.7Opus 4.5 Research.375, 0.2-0.60.125, 0-0.30.5, 0.3-0.6GPT 5.2 Pro0.35, 0.2-0.550.1, 0-0.250.55, 0.35-0.7o3 Deep Research0.4, 0.3-0.50.05, 0-0.20.55, 0.5-0.7point est. average0.380.080.54

 

I'm moderately confident my prompt was not biased because the A values here are lower than what I've gotten from Claude when asking for heritability estimates from twin studies only. Also, all the models included some discussion of the rapid rise in adolescent cases in the 2010s, often mentioning "social contagion" and ROGD theories explicitly. All the models also pointed out that the ACE model is a simplification and that gene-environment interaction may be significant.

These are pretty wide error bars. But since A is trying to capture heredity only, we can take A as a rough lower bound for biological causation. Even if E is purely social, 38% is significant.

Also, none of this tells us how much variation there is at the individual level. And we have no trans GWAS.

The big question is whether E is dominated by social or biological factors.

If social factors mattered a lot I would expect parental attitudes to be significant in affecting transgender identity. But most studies find low C. This holds even for population-based studies that do not suffer from ascertainment bias. I would be surprised if peer influences were highly causal but parental influences were not.

I think the evidence from CAH, fraternal birth order effects, and animal models also provides good mechanistic reasons to think there are significant biological effects in E as well as A.

How do trans people view this line of research? They tend to hate it. They're afraid it will eventually lead to:

1. not choosing "trans embryos" during IVF
2. aborting "trans fetuses"
3. lab/genetic testing to determine who is allowed to medically transition

This is what I'll call "medical eradication": one half of the Double Bind.

 

B. The Trans Double Bind

The purpose of medicine is to improve health and reduce suffering.

In general, the state should not subsidize healthcare that does not increase QALYs. A rational healthcare system would ration care based on ranking all available treatments by QALYs saved per dollar, and funding all treatments above a cutoff determined by the budget.

The US healthcare system has a very creative interpretation of reality, but other countries like the UK at least attempt to do this.

To receive gender-affirming treatment, trans people must argue that such treatment alleviates suffering. This argument helped establish gender medicine in the 20th century. 

But in fact, the claim "being transgender involves suffering and requires medical treatment" is very controversial within the trans community. This is surprising, because disputing this claim threatens to undermine access to trans healthcare.

Moreover, this controversy explains why trans people do not appear to accurately report their own motivations for transition.

 

Motivations to transition

There are three possible sources:

1. biological
2. psychological/cognitive
3. social

These can co-occur and interact.

Society at large recognizes only (1) as legitimate.

Trans people know this. They know they may be sent to psychotherapy, denied HRT, or judged illegitimate if they report wanting to transition for psychosocial reasons.

There is strong pressure for trans people to accept and endorse a biological/medical framing for their transitions.

But adopting this framing carries downsides:

• Dependence on medical authorities for legitimacy
  • Historically, medicine has treated us very poorly[5]
  • We have little power to negotiate for better medical care if we are dependent on medicine to validate us to the rest of society
• Psychological costs
  • Trans-cultural memory of medical mistreatment
  • Many find medicalization demeaning and resent dependence
• Possible medical eradication
  • We can't claim we need care if we don't suffer[6], but one day the medical system might find a more direct way to eliminate our suffering: preventing trans people from coming into existence in the first place.
     

This is the Double Bind: many trans people need medical treatment, but find the psychological threat of medicalization and eradication intolerable.

Consequently, they will not claim their transition is justified because of biology. However, they know that psychological and social justifications will also not be accepted. In this situation, platitudes like "I am a woman because I identify as one" are a predictable response to incentives. If you attempt to give a real answer, it will be used against you.

Maybe you are thinking:

Marisa, this is hogwash! All the trans people I know are constantly oversharing lurid personal details despite obvious social incentives not to. The most parsimonious explanation is that people who say "I'm __ because I identify as  __" literally believe that.

Yes, good point. I need to explain another dynamic.

So far I've only discussed external incentives, but there is incentive pressure from within the trans community as well.

In the 2010s, the following happened:

• Youth transitions increased
• Nonbinary identification increased, especially among people not medically transitioning 
• Acceptance, awareness, and politicization all increased
• Social media happened

Suddenly the trans community was fighting for a much broader set of constituents and demands. 20th century binary transsexualism coheres with medical framings, but 2010s Tumblr xenogenders do not. And trans people of all kinds have always had insecurities about their own validity-- both internal and external.

Here is the key insight:

It's difficult to enforce norms that protect external political perception.

It's easy to enforce norms that protect ingroup feelings.

Assume I've performed and posted some porn on the internet. This porn is optically really really bad. Like actually politically damaging. Conscientious trans people will attempt to punish my defection-- but this is difficult. I can cry "respectability politics!" and point to the history of trans sex work in the face of employment discrimination. No one can agree on a theory of change for politics, so it's hard to prove harm. When the political backlash hits, it affects everyone equally[7]. 

By contrast, assume instead that I'm in a trans community space and I've told someone their reasons for transition are not valid, and they should reconsider. I've just seriously hurt someone's feelings, totally killed the vibe, and I'll probably be asked to leave-- maybe shunned long-term[8]. I have just lost access to perhaps my only source of ingroup social support. This is a huge disincentive. 

This structure, combined with the influx of novel identities in the 2010s, created an environment where it was taboo even to talk about causal theories for one's own transition, because it could be invalidating to someone else. All gender identities were valid at all times. Downstream effects of external social pressure, social media, and politics created an environment of collective ignorance where community norms discouraged investigating the causes of transition.

 Introspective Clarity

Famously, trans people tend not to have great introspective clarity into their own motivations for transition. Intuitively, they tend to be quite aware of what they do and don't like about inhabiting their chosen bodies and gender roles. But when it comes to explaining the origins and intensity of those preferences, they almost universally to come up short. I've even seen several smart, thoughtful trans people, such as Natalie Wynn, making statements to the effect that it's impossible to develop a satisfying theory of aberrant gender identities. (She may have been exaggerating for effect, but it was clear she'd given up on solving the puzzle herself.)

This is the wrong interpretation of Natalie Wynn's oeuvre. See Appendix: Contra Fiora on Contra for why.

What would a legitimate explanation of the origins of one's gendered feelings look like?

Fiora never tells us her criteria. And the only example she gives us-- a psychosocial explanation of her own transition-- heavily implies that it was illegitimate.

But she's also dismissive of biological theories. Does that mean no transitions are valid?

I got whole genome sequencing last year. I can point at the sexual and endocrine abnormalities in my genome, but I certainly can't prove they justify my transition. Nevertheless, subjectively, HRT saved my life.

 

C. In the Case of Quinoa Marisathe author, age 13. Note the oversized Haibane Renmei graphic tee

(Extremely simplified for brevity)

In middle school, puberty started and my life fell apart. I hated my erections, my libido; I felt like a demon had taken over my brain. Unlike my peers, I never developed a felt sense of how to throw my body around. They got rougher, and better at sports. I got injured.

I was pathologically shy and awkward. Locker room talk was utterly repulsive to me. I lost friends and didn't care. Rurouni Kenshin was my first inspiration to grow my hair out. I am very lucky my parents let me.

There was an autistic kid on my soccer team with a speech impediment. He was good at soccer but the other boys would cruelly tease him at practice, in part because he didn't understand they were teasing him. One night after practice I spent the car ride home sobbing about it in front of my dad, who didn't get it at all. I quit soccer.

I was utterly miserable in school. In March of 7th grade, I developed real depression, and started thinking about suicide. Mom took me to two different psychologists. We decided I would homeschool 8th grade. Now, I really had no friends. I was still depressed.

At this point I was only living for WoW and anime. By far, my favorite was Haibane Renmei. It's 13 episodes of angel-girls living in a run-down boarding school and basically just taking care of each other. It is heavily implied that the Haibane are there-- in purgatory-- because they committed suicide in the real world, and must learn how to accept love and care.

It's difficult to explain how much this series resonated with me. It gave structure to feelings I couldn't articulate. I never believed there was any possibility of becoming a girl in real life, so I didn't fantasize much about that. But for a couple years I daydreamed frequently about dying and becoming a Haibane[9].

My hair was long enough at this point that I "passed". I was frequently assumed female in social situations, and men would often tell me I was in the wrong bathroom. I longed for delicate reciprocal care with others who somehow understood what I was going through, even though I could hardly understand it myself. Haibane Renmei showed me this but I had no idea how to find it in the real world.

At 16, boy puberty hit me like a truck. I became ugly. I still had no social skills, and no friends. I dressed like a hobo. The summer after junior year I confronted myself in the mirror and admitted I would never be cute again. I still desperately wanted to be loved, and I believed that the only path to achieving that was becoming a man girls would want to date. That meant improving my appearance and social skills.

I knew that women find weebs unattractive. And my long hair was definitely unattractive. It all melded together. I had no real-world outlet for my femininity so I'd poured it all into identifying with anime characters. And it all seemed like a dead end. I felt that if I stayed in the anime community I would end up socially stunted, since its social standards were lower. I cut my hair and stopped watching anime. I put a lot more effort into socializing.

In college, I read The Man Who Would Be Queen, self-diagnosed as AGP, and actually considered transition for the first time. But it was too late for me-- the sight of my face in the mirror, and the depictions of AGPs in the book were too horrifying. I resolved to never transition, and attempted suicide soon after.

7 months later I fell in love, and that relationship turned my life around. I loved her immeasurably for 5 years, and we lived together for 2 of those. I became, on the outside, socially and professionally actualized as a man. I was a great boyfriend and had no problem getting dates. After the breakup I fell in love 2 more times.

You already know how this ends. No amount of true love or social validation as a man could fix me. I never wanted to transition, but at 31 the strain of repression became unbearable. Things have turned out far better than I ever dared imagine. My parents have remarked on multiple occasions, unprompted, how much happier I am now. They're right.

Overall I fit Fiora's phenotype: I was a mentally ill social outcast weeb, desperately identifying with anime characters as a simulacrum of loving care I had no idea how to find in real life.

But I can't explain my eventual transition at 31 through anything other than a biological cause. I looked obsessively for evidence of some repressed or unconscious ulterior motive, and found none. I believed that transition would be very expensive and time-consuming, physically painful[10], reduce my attractiveness as a mate, and change my social possibilities. All of these predictions have born true. What I didn't expect is that HRT drastically improved my mental health even before the physical changes kicked in. My baseline now is my former 90th-percentile of calm and happiness. 

I'm n=1 but this shows Fiora's phenotype can coexist with biologically rooted dysphoria. Moreover, I believe my middle school social failures were caused as much by gender incongruence as by neurodivergence. It's difficult to socialize when your puberty feels wrong and your social instincts don't match your assigned gender.

It's almost like most of them had deep emotional wounds, often stemming from social rejection, and had transitioned to become cute girls or endearing women as a kind of questionably adaptive coping mechanism.

Maybe. Or a misaligned subconscious sex is part of what caused the social rejection in the first place.

Conclusion

As Fiora implied, "cuteness-maxxing" is probably not a good reason to transition.

Most people desperately want to be loved and this can cause mistakes with transition in both directions. Social media is probably bad for minors. We should emphasize that, at a fundamental level, trans people are neither more nor less lovable than cis people.

The human brain is perhaps the most complex object in our known universe, and we will likely never be able to fully disentangle psychosocial factors from biological ones. That said, I do think humanity will discover ever stronger evidence for biological causes of trans identity within our lifetimes.

Introspection is a noisy way to attempt to answer "am I trans?", and you hit diminishing returns fast. It's also the wrong question. The right question is "should I transition?". Transition is best understood as a Bayesian process where you take small behavioral steps[11] and update on whether your quality of life is improving.

If you start transitioning and your intrinsic health and happiness improves, and you expect the same to be true in the long run, continue. If not, desist. There is no shame in either outcome.

 

1. ^

   https://pmc.ncbi.nlm.nih.gov/articles/PMC9186536/

2. ^

   For twins, prenatal environment shows up in both C and E.

3. ^

   Coolidge et al. (2002), Heylens et al. (2012), Karamanis et al. (2022), Conabere et al. (2025), Sasaki et al. (2016), Bailey et al. (2000), Burri et al. (2011), Diamond (2013), Buhrich et al. (1991).

   If you just want to read a systematic review of these studies, see https://pmc.ncbi.nlm.nih.gov/articles/PMC12494644/

4. ^

   I'm trying to understand the etiology of transgender identity, particularly the strength of the evidence base for different categories of potential causes. Please segment the analysis into five categories:

   1. Hereditary/genetic factors
   2. Prenatal environment (hormonal, epigenetic, maternal)
   3. Postnatal biological environment (diet, medications, endocrine factors)
   4. Family/microsocial environment
   5. Macrosocial/cultural environment

   For each category, conduct a rigorous literature review prioritizing meta-analyses, large-N studies, and methodologically sound designs. Identify the strongest evidence both supporting and contradicting causal contributions from that category. Flag studies with clear methodological limitations and discuss known publication biases in the field.

   Focus primarily on gender dysphoria and transgender identity as defined in DSM-5/ICD-11, noting where studies conflate distinct constructs or onset patterns.

   Conclude with a variance decomposition estimate using the ACE framework and liability threshold model standard in psychiatric genetics. Provide:

   - Point estimates with plausible ranges for each component (A, C, E)
   - Confidence ratings for each estimate based on evidence quantity and quality
   - Explicit discussion of what each ACE component likely captures, mapped back to the five categories above
   - Acknowledgment of confounds and unmeasurable factors

   Include cross-cultural and temporal trend data as evidence bearing on the cultural/environmental components.

5. ^

   In general, in the US in the 20th century, if a medical institution decided they simply didn't want to treat trans patients, there would be no public outcry. The doctors and organizations that did treat us could set terms. Prior to the 2010s there was little awareness of trans people, and the awareness we had was often prejudicial. IBM fired Lynn Conway after all.

6. ^

   Some trans people (for example, Abigail Thorn and Andrea Long Chu) have attempted to argue that access to gender-affirming care should not be contingent on either (a) suffering prior to receiving treatment or (b) demonstrated therapeutic benefit for the treatment. These arguments were not well-received even within the trans community.

7. ^

   It took r/MtF until 2025 to ban porn, after years of infighting. https://www.reddit.com/r/MtF/comments/1kaxn18/alright_lets_talk_about_porn_and_porn_accounts/

8. ^

   This norm is not totally unreasonable. The purpose of community spaces is primarily social support for those early in transition, which can be difficult to find anywhere else. I went through this phase too.

9. ^

   Yes, this is perverse and contradicts the moral of the story.

10. ^

    Electrolysis is the most physically painful thing I've experienced. I've done 40 hours so far and will likely do 150-200 total.

11. ^

    Voice training, experimenting with name/pronouns/clothing, laser hair removal, HRT. 

Discuss

Published on January 20, 2026 1:53 AM GMT

This is an appendixpost for Why I Transitioned: A Response.

In Why I Transitioned: A Case Study, Fiora Sunshine claims:

Famously, trans people tend not to have great introspective clarity into their own motivations for transition. Intuitively, they tend to be quite aware of what they do and don't like about inhabiting their chosen bodies and gender roles. But when it comes to explaining the origins and intensity of those preferences, they almost universally to come up short. I've even seen several smart, thoughtful trans people, such as Natalie Wynn, making statements to the effect that it's impossible to develop a satisfying theory of aberrant gender identities. (She may have been exaggerating for effect, but it was clear she'd given up on solving the puzzle herself.)

The evidence most strongly suggests that Natalie did not give up-- she was bullied into silence.

This misreading matters because it illustrates one half of the Trans Double Bind. Natalie's words in Canceling were chosen under extreme social pressure from the online/Twitter/leftist contingent of the trans community. This social pressure existed because the community felt they were enforcing norms necessary to ensure respect and acceptance for enbys[1].

The linked video, Canceling, is Natalie defending against accusations of transmedicalism[2] due to using a voice-over from transmedicalist Buck Angel in her previous video.

And in the linked section specifically, she is defending and attempting to recontextualize one of her tweets:

One of the most important facts about Natalie is that despite what her on-screen persona suggests-- she is sensitive and suffers greatly from hate comments online, especially from within the trans community[3].

This video reply to being canceled was high-stakes because it had major long-term implications not just for her Patreon livelihood and career but her dignity, physical safety, and social acceptance.

As far as I can tell, Natalie is not lying in Canceling. But she is defending her record in part through omission and vagueness.

I can't tell you what her genuine beliefs are. In part because of this controversy she deliberately moved away from making comments or videos directly about trans issues, and has expressed general despair about the situation.

I do not believe Natalie is a transmedicalist, secretly or otherwise. There is a lot of theory-space between "all genders/transitions are valid no matter what" and transmedicalism.

But her blanket retraction ("I no longer believe there can be any rational justification of gender identity") is not credible because:

A. The context of Canceling highly incentivized her to make her commentary on her tweet as politically defensible as possible (If you disavow reason then it is impossible to exclude anyone).

B. The evidence suggests her real views are more nuanced.

She has made multiple extremely personal, searching videos about her dysphoria and motivations to transition, most notably Autogynephilia. Beauty is surprisingly critical of the usage and concept of gender dysphoria (and motivations for pursuing medical transition). Transtrenders deals with all these topics in skit form, and was also heavily scrutinized online.

Prior to Canceling, Natalie stated on multiple occasions that she transitioned because of gender dysphoria. This illustrates the Double Bind because the online trans community took as implication that she believed dysphoria was an important part of justifying transition-- which would exclude people who do not report dysphoria, and threaten to reduce their acceptance in their identified gender.

The other side of the Double Bind is weak here because, in the 2010s as a binary trans woman with substantial income, Natalie's access to HRT and surgery was not conditional on endorsing transmedicalism.

I think her comments in her AMAs are more interesting and revealing. I can't link to these videos directly (paywall) and I don't know if anyone here cares to read long transcripts. But I will end this post by including some here because they are both interesting and relevant.

 

August 2018 Patron AMA stream

QUESTION (19:25): Becoming more the person you are was the thought that came to mind. It reminded me of something Schopenhauer said about the empirical character as a manifestation of the intelligible character. That what we appear to be outwardly is just an imperfect expression of our true immutable inmost nature. Does that resonate at all? Do you think it is a useful way of thinking about gender transition? Are you an expression of transcendental freedom? Could a cranky sexist 19th century philosopher be invoked against reductive shit lord rationalizing?

NATALIE: I think I actually take the opposite view. I take more of the Wittgenstein pragmatic view which is that the self is like invented instead of discovered. More trans people do actually think of it the way you're suggesting that by transitioning they're actually realizing this inherent like essence or singularity that's always there. That their exterior appearance is kind of finally becoming like their insides finally matching outside. It's like sort of not that's not really the sense I have to be quite honest like I kind of want to pretend that it is because it's a more attractive thing to say about yourself right? I think people might be more attracted to me if I was expressing the true feminine essence of my being but the truth is that I designed this, femininity is something I've worked on and it's a it's an invention it's a creation of mine as much as it is a discovery.

 

November 2018 Patron AMA stream

Question (2:24): How did you find out you were transgender?

Natalie: ...I started taking hormones before I was 100% sure I identified as a woman, to be honest, because I wanted the effects of the hormones... once I had started hormones... I'm like, I'm not non-binary, I just want to be a woman, and so it was like one step at a time...

When you discover that, you like taking female hormones, and it makes you feel better about yourself, and you like the physical changes, you just look at your life, and you're like, well, this is just going to be easier if I just be a woman, like, that sounds very pragmatic, but that to me is kind of thinking, if I went into it, honestly, there was sort of a pragmatic reasoning behind it, like, my life is going to be better if I just live as a woman. And so that's when I decided, like, fuck it, like, let's just go all in on this.

September 2019 Patron AMA stream

QUESTION (54:02): Do you think dysphoria is externally or internally generated? That is if we lived in a world without transphobia where trans identities were immediately 100% accepted by all people, would dysphoria still exist?

NATALIE: ...it's hard for me to imagine like what that would even look like because I think there's a difference between transphobia and some trans identities not being accepted immediately, because I think that part of what gender is is the assumption that there's two categories of people that in terms of all the senses present in a different way and if we just completely dropped the idea that gender is something that you identify based on the way someone looks and instead started thinking of gender as a purely psychological phenomenon it's a little bit hard for me to imagine like what being trans even would mean in that situation...

i just sort of don't get like i don't get what people are talking about when they talk about hypotheticals like this...

...what does it mean to identify as a woman when all woman means is a psychological state?

...i don't know how to talk about like i'm so used to the idea that like i just can't talk about this that like i i i sort of don't know how much i should say...

...there's trans people right who present totally within the normal range of what is expected of someone who's assigned their gender at birth and i'm not saying they're not valid i'm just saying that like i sort of don't recognize it as what being trans is to me

...my own trans identity it's so connected to this desire to socially fit in as a woman [and look female] and... so when someone identifies as trans without either of those components... i don't understand it yet.

QUESTION (02:55:25): are there any videos you would like to make but feel like you can't because they're too different or frivolious or inflammatory?

NATALIE: ...one I don't think I'll ever do would be a follow up to the Autogynephilia video... I kind of feel like that video in particular is kind of weak. Despite its length, I don't think it really deals with this the subject matter and well, and I think that the video I have in mind would be about a lot of the difficult questions about why trans women transition and how in my opinion like there is anthropological truth to Blanchardism like clearly he's observing real trends, right?

...if you read Magnus Hirschfeld's work from the 30s... it comes to the same conclusions as Blanchard and those things have troubled me throughout my transition and and in some ways have troubled me more as I've met more and more trans women, and feel that you know there really are these kinds of two stark clusters of trans women with very different backstories, and... if I were to make a theory about trans women I would do a kind of post Blanchardism that starts with a lot of those observations and then it tries to come up with a more nuanced way of talking about them than what Blanchard offers.

My Autogynephilia video has a million views and that's unusual. It's the only video of mine that's that old that has that many views. Why does that many video have so many views? A lot of people are googling this topic. And if you look at the more sinister parts of trans internet it's kind of an obsessive topic and I think that part of the reason for that is that a lot of mainstream trans discourse is very euphemistic about things. There's a heavily ideologically loaded concept of trans woman and you're supposed to believe all these things, like you're supposed to say I was always a woman and that I was a woman born in a man's body and like the fact of the matter is that this just does not line up with a very large number of people's experiences...

And then on the other side you have Blanchard who talks about, there's this group of trans women who before transition they live as feminine gay men and... the fundamental problem of their life is femininity and often that it's you know, they're bullied for and the it's just like this issue throughout their childhood adolescence and in early adulthood. On the other hand, you have a whole second group of trans women who basically seem to pass as normal men and until you know, they come out as trans and shock everyone and like it's just that these are two very different experiences so it's like such a deeply taboo topic...

The problem I have with my Autogynephilia video is that in a way I was pushing too hard against some of Blanchard's things, right, because it's a very threatening theory to trans women because is saying is that you are men. I want to try to make sense of Blanchard's observations without reaching his conclusion that these are just either male homosexuals or male fetishests because I don't believe that.

I've met hundreds of trans women at this point and um it's pretty hard not to notice that the two type typology is based on something that that's real, right? I'm not saying that the typology is theoretically good. I'm just saying that it's based on something that is quite clearly real, and so far as I'm aware there's simply no way of talking about that except Blanchardism and that's not superfucking great is it...

I hate the way a lot of people summarize my video like they'll just summarize it as oh, I said there's no such thing as autogynephilia, no one has that those feelings; that's clearly not true. I think it's actually quite common for men to um like yeah, you know like a straight guy who likes taking pictures of his butt in women's yoga pants, like sending them to his friends or something? it's a feeling, I don't think this is what what causes people to transition but I think it's a dimension to a lot of people's sexuality that I don't particularly see the point in denying. Nor do I think that Blanchardism is a good theory. 

 

1. ^

   By the mid 2010s the lines of battle had shifted so much that binary trans people were no longer perceived to be under threat, and the focus shifted towards nonbinary issues. These were more politically salient (nonbinary => overthrowing the binary => overthrowing patriarchy) which made them more conducive to a social media positive feedback loop, and were also subject to more social opposition in everyday interactions.

2. ^

   The view that trans people are only valid if they experience gender dysphoria

3. ^

   See for example the 17 minutes at the beginning of her October 2019 patron AMA stream, right after the start of the controversy, where she is upset to the point of altering her speaking cadence, and at one point on the verge of tears.

Discuss

Published on January 20, 2026 1:25 AM GMT

What counts as a lie?

Centrally, a lie is a statement that contradicts reality, and that is formed with the explicit intent of misleading someone. If you ask me if I’m free on Thursday (I am), and I tell you that I’m busy because I don’t want to go to your stupid comedy show, I’m lying. If I tell you that I’m busy because I forgot that a meeting on Thursday had been rescheduled, I’m not lying, just mistaken.

But most purposeful misrepresentations of a situation aren’t outright falsehoods, they’re statements that are technically compatible with reality while appreciably misrepresenting it. I likely wouldn’t tell you that I’m busy if I really weren’t; I might instead bring up some minor thing that I have to do that day and make a big deal out of it, to give you the impression that I’m busy. So I haven’t said false things, but, whether through misdirecting, paltering, lying by omission, or other such deceptive techniques, I haven’t been honest either.

We’d like a principled way to characterize deception, as a property of communications in general. Here, I’ll derive an unusually powerful one: deception is misinformation on expectation. This can be shown at the level of information theory, and used as a practical means to understand everyday rhetoric.

 

Information-Theoretic Deception

Formally, we might say that Alice deceives Bob about a situation if:

First Definition: She makes a statement to him that, with respect to her own model of Bob, changes his impression of the situation so as to make it diverge from her own model of the situation.

We can phrase this in terms of probability distributions. (If you’re not familiar with probability theory, you can skip to the second definition and just take it for granted). First, some notation:

1. For a possible state x.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}  of a system X, let

pAX(x),pBX(x)

be the probabilities that Alice and Bob, respectively, assign to that state. These probability assignments pAX and pBX are themselves epistemic states of Alice and Bob. If Alice is modeling Bob as a system, too, she may assign probabilities to possible epistemic states qBX that Bob might be in:

qBX↦pAB(qBX)

2. Let

pB∣sX(x)=pBX(x∣s)

1. be Bob’s epistemic state after he updates on information s. In other words, B∣s is the Bob who has learned s.
2. Take x to be the world Ω. We’ll leave it implicit when it’s the only subscript.

With this notation, a straightforward way to operationalize deception is as information Alice presents to Bob that she expects to increase the difference between Bob’s view of the world and her own.

Taking the Kullback-Leibler divergence as the information-theoretic measure of difference between probability distributions, this first definition of deception is written as:

{\mathbb E}_{p^A_B}\left[\operatorname{KL}\left(p^A \mid\mid q^{B}\right)\right]">EpAB[KL(pA∣∣qB∣s)]>EpAB[KL(pA∣∣qB)]

We can manipulate this inequality:

0<EpAB[KL(pA∣∣qB∣s)]−EpAB[KL(pA∣∣qB)]=∫pAB(qB)∫pA(ω)lnpA(ω)qB∣s(ω)−pA(ω)lnpA(ω)qB(ω)dωdqB=∬pAB(qB)pA(ω)ln(pA(ω)qB(ω∣s)qB(ω)pA(ω))dωdqB

Write B,Ω for the product system composed of B and Ω, whose states are just pairs of states of B and Ω. The inequality can then be written in terms of an expected value:

0<−EpAB,Ω[lnqB(ω∣s)qB(ω)]⟹EpAB,Ω[lnqB(ω∣s)qB(ω)]<0

This term is the proportion to which Alice expects the probability Bob places on the actual world state to be changed by his receiving the information $s$. If we write this in terms of surprisal, or information content,

S(x)=−lnp(x)

we have

{\mathbb E}_{p^A_{B, \Omega}}\left[S^B(\omega)\right]">EpAB,Ω[SB(ω∣s)]>EpAB,Ω[SB(ω)]

This can be converted back to natural language: Alice deceives Bob with the statement s if:

Second Definition: She expects that the statement would make him more surprised to learn the truth as she understands it[1].

In other words, deception is misinformation on expectation.

Misinformation alone isn’t sufficient—it’s not deceptive to tell someone a falsehood that you believe. To be deceptive, your message has to make it harder for the receiver to see the truth as you know it. You don’t have to have true knowledge of the state of the system, or of what someone truly thinks the state is. You only have to have a model of the system that generates a distribution over true states, and a model of the person to be deceived that generates distributions over their epistemic states and updates.

 

This is a criterion for deception that routes around notions of intentionality. It applies to any system that

• forms models of the world,
• forms models of how other systems model the world, and
• determines what information to show to those other systems based on its models of these systems.

An AI, for instance, may not have the sort of internal architecture that lets us attribute human-like intents or internal conceptualizations to it; it may select information that misleads us without the explicit intent to mislead[2]. An agent like AlphaGo or Gato, that sees humans as just another game to master, may determine which statements would get us to do what it wants without even analyzing the truth or falsity of those statements. It does not say things in order to deceive us; deception is merely a byproduct of the optimal things to say.

In fact, for sufficiently powerful optimizers, deception ought to be an instrumental strategy. Humans are useful tools that can be easily manipulated by providing information, and it’s not generally the case that information that optimally manipulates humans towards a given end is simultaneously an accurate representation of the world. (See also: Deep Deceptiveness).

 

Rhetorical Deception

This criterion can be applied anywhere people have incentives to be dishonest or manipulative while not outright lying.

In rhetorical discussions, it’s overwhelmingly common for people to misrepresent situations by finding the most extreme descriptions of them that aren’t literally false[3]. Someone will say that a politician “is letting violent criminals run free in the streets!”, you’ll look it up, and it’ll turn out that they rejected a proposal to increase mandatory minimum sentencing guidelines seven years ago. Or “protein shakes can give you cancer!”, when an analysis finds that some brands of protein powder contain up to two micrograms of a chemical that the state of California claims is not known not to cause cancer at much larger doses. And so on. This sort of casual dishonesty permeates almost all political discourse.

Descriptions like these are meant to evoke particular mental images in the listener: when we send the phrase “a politician who’s letting violent criminals run free in the streets” to the Midjourney in our hearts, the image is of someone who’s just throwing open the prison cells and letting out countless murderers, thieves, and psychos. And the person making this claim is intending to evoke this image with their words, even though they'll generally understand perfectly well that that’s not what’s really happening. So the claim is deceptive: the speaker knows that the words they’re using are creating a picture of reality that they know is inaccurate, even if the literal statement itself is true.

This is a pretty intuitive test for deception, and I find myself using it all the time when reading about or discussing political issues. It doesn’t require us to pin down formal definitions of “violent criminal” and a threshold for “running free”, as we would in order to analyze the literal truth of their words. Instead, we ask: does the mental image conveyed by the statement match the speaker’s understanding of reality? If not, they’re being deceptive[4].

Treating expected misinformation as deception also presents us with a conversational norm: we ought to describe the world in ways that we expect will cause people to form accurate mental models of the world.

 

 

(Also posted on Substack)

 

1. ^

   This isn’t exactly identical to the first definition. Note that I converted the final double integral into an expected value by implicitly identifying

   pAB(qB)pA(ω)=pAB,Ω(qB,ω)

   i.e. by making Bob’s epistemic state independent of the true world state, within Alice’s model. If Alice is explicitly modeling a dependence of Bob’s epistemic state on the true world state for reasons outside her influence, this doesn’t work, so the first and second definitions can differ.

   Example:  If I start having strange heart problems, I might describe them to a cardiologist, expecting that this will cause them to form a model of the world that’s different from mine. I expect they’ll gain high confidence that my heart has some specific problem X that I don’t presently consider likely due to my not knowing cardiology. So, to me, there’s an expected increase in the divergence between our distributions that isn’t an expected increase in the cardiologist’s surprisal, or distance from the truth. Because the independence assumption above is violated—I take the cardiologist’s epistemic state to be strongly dependent on the true world state, even though I don’t know that state—the two definitions differ. Only the second captures the idea that honestly describing your medical symptoms to a doctor shouldn’t be deception, since you don’t expect that they’ll be mis-informed by what you say.

2. ^

   Even for humans, there’s a gray zone where we do things whose consequences are neither consciously intended nor unintended, but simply foreseen; it’s only after the action and its consequences are registered that our minds decide whether our narrative self-model will read “yes, that was intended” or “no, that was unintended”. Intentionality is more of a convenient fiction than a foundational property of agents like us.

3. ^

   Resumes are a funnier example of this principle: if someone says they placed “top 400” in a nationwide academics competition, you can tell that their actual rank is at least 301, since they’d be saying “top 300” or lower if they could.

4. ^

   Of course everyone forms their own unique mental images; of course it’s subjective what constitutes a match; of course we can’t verify that the speaker has any particular understanding of reality. But you can generally make common-sense inferences about these things.

Discuss

Published on January 20, 2026 12:58 AM GMT

In a software-only takeoff, AIs improve AI-related software at an increasing speed, leading to superintelligent AI. The plausibility of this scenario is relevant to questions like:

• How much time do we have between near-human and superintelligent AIs?
• Which actors have influence over AI development?
• How much warning does the public have before superintelligent AIs arrive?

Knowing when and how much I expect to learn about the likelihood of such a takeoff helps me plan for the future, and so is quite important. This post presents possible events that would update me towards a software-only takeoff.

What are returns to software R&D?

The key variable determining whether software progress alone can produce rapid, self-sustaining acceleration is returns to software R&D (r), which measures how output scales with labor input. Specifically, if we model research output as:

O∝Ir.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

where O is research output (e.g. algorithmic improvements) and I is the effective labor input (AI systems weighted by their capability), then r captures the returns to scale.

If r is greater than 1, doubling the effective labor input of your AI researchers produces sufficient high-quality research to more than double the effective labor of subsequent generations of AIs, and you quickly get a singularity, even without any growth in other inputs. If it's less than 1, software improvements alone can't sustain acceleration, so slower feedback loops like hardware or manufacturing improvements become necessary to reach superintelligence, and takeoff is likely to be slower.

Projected software capacity growth under different returns-to-scale assumptions, holding hardware constant. ASARA is AI Systems for AI R&D Automation. When r > 1, each generation of AI researchers produces more than enough capability gain to accelerate the next generation, yielding explosive growth (red, purple). At r = 1 (orange), gains compound but don't accelerate. When r < 1 (green, blue), diminishing returns cause growth to asymptotically approach the dashed baseline, making hardware or other bottleneck improvements necessary for continued acceleration. From Forethought.

A software-only singularity could be avoided if r is not initially above 1, or if r decreases over time, for example, because research becomes bottlenecked by compute, or because algorithmic improvements become harder to find as low-hanging fruit is exhausted.

Initial returns to software R&D

The most immediate way to determine if returns to software R&D are greater than 1 would be observing shortening doubling times in AI R&D at major labs (i.e. accelerating algorithmic progress), but it would not be clear how much of this is because of increases in labor rather than (possibly accelerating) increases in experimental compute. This has stymied previous estimates of returns. 

Posterior distributions of returns to software R&D (r) across four domains. Only SAT solvers have a 90% confidence interval entirely above 1. From Epoch AI.

Evidence that returns to labor in AI R&D are greater than 1:

1. Progress continues to accelerate after chip supplies near capacity constraints. This would convince me that a significant portion of continued progress is a result of labor rather than compute and would constitute strong evidence.
2. Other studies show that labor inputs result in compounding gains. This would constitute strong evidence.
   1. Any high-quality randomized or pseudorandom trial on this subject.
   2. Work that effectively separates increased compute from increased labor input [1].
3. Labs continue to be able to make up for less compute than competitors with talent (like Anthropic in recent years). This would be medium-strength evidence.
4. A weaker signal would be evidence of large uplifts from automated coders. Pure coding ability is not very indicative of future returns, however, because AIs’ research taste is likely to be the primary constraint after full automation.
   1. Internal evaluations at AI companies like Anthropic show exponentially increasing productivity.
   2. Y Combinator startups grow much faster than previously (and increasingly fast over time). This is likely to be confounded by other factors like overall economic growth.

Compute bottlenecks

The likelihood of a software-only takeoff depends heavily on how compute-intensive ML research is. If progress requires running expensive experiments, millions of automated researchers could still be bottlenecked. If not, they could advance very rapidly.

Here are some things that would update me towards thinking little compute is required for experiments:

1. Individual compute-constrained actors continue to make large contributions to algorithmic progress[2]. This would constitute strong evidence. Examples include:
   1. Academic institutions which can only use a few GPUs.
   2. Chinese labs that are constrained by export restrictions (if export restrictions are reimposed and effective).
2. Algorithmic insights can be cross-applied from smaller-scale experimentation. This would constitute strong evidence. For example:
   1. Optimizers developed on small-scale projects generalize well to large-scale projects[3].
   2. RL environments can be iterated with very little compute.
3. Conceptual/mathematical work proves particularly useful for ML progress. This is weak evidence, as it would enable non-compute-intensive progress only if such work does not require large amounts of inference-time compute.

Diminishing returns to software R&D

Even if returns on labor investment are compounding at the beginning of takeoff, research may run into diminishing returns before superintelligence is produced. This would result in the bumpy takeoff below.

Three intelligence explosion/takeoff scenarios. In the rapid scenario, a software-only takeoff reaches a singularity. In the bumpy scenario, software-only takeoff stalls until AI can improve hardware and other inputs. In the gradual scenario, meaningful capability gains only occur once AI can augment the full stack of inputs to production. From Forethought.

 

The evidence I expect to collect before takeoff is relatively weak, because current progress rates don't tell us much about the difficulty of discovering more advanced ideas we haven't yet tried to find. That said, some evidence might be:

1. Little slowdown in algorithmic progress in the next few years. Evidence would include:
   1. Evidence of constant speed of new ideas, controlling for labor. Results from this type of analysis that don’t indicate quickly diminishing returns would be one example.
   2. Constant time between major architectural innovations (e.g. a breakthrough in 2027 of similar size to AlexNet, transformers, and GPT-3)[4].
   3. New things to optimize (like an additional component to training, e.g. RLVR).
   4. Advances in other fields like statistics, neuroscience, and math that can be transferred with some effort. For example:
      1. Causal discovery algorithms that let models infer causal structure from observational data.
2. We have evidence that much better algorithms exist and could be implemented in AIs. For example:
   1. Neuroscientific evidence of the existence of much more efficient learning algorithms (which would require additional labor to identify).
   2. Better understanding of how the brain assigns credit across long time horizons.

Conclusion

I expect to get some evidence of the likelihood of a software-only takeoff in the next year, and reasonably decisive evidence by 2030. Overall I think evidence of positive feedback in labor inputs to software R&D would move me the most, with evidence that compute is not a bottleneck being a near second. 

Publicly available evidence that would update us towards a software-only singularity might be particularly important because racing companies may not disclose progress. This evidence is largely not required by existing transparency laws, and so should be a subject of future legislation. Evidence of takeoff speeds would also be helpful for AI companies to internally predict takeoff scenarios.

Thanks for feedback from other participants in the Redwood futurism writing program. All errors are my own. 

1. ^

   This paper makes substantial progress but does not fully correct for endogeneity, and its 90% confidence intervals straddle an r of 1, the threshold for compounding, in all domains except SAT solvers.

2. ^

    It may be hard to know if labs have already made the same discoveries.

3. ^

   See this post and comments for arguments about the plausibility of finding scalable innovations using small amounts of compute.

4. ^

   This may only be clear in retrospect, since breakthroughs like transformers weren't immediately recognized as major.

Discuss

Published on January 20, 2026 12:51 AM GMT

The problem

You are routinely exposed to CO2 concentrations an order of magnitude higher than your ancestors. You are almost constantly exposed to concentrations two times higher. Part of this is due to the baseline increase in atmospheric CO2 from fossil fuel use, but much more of it is due to spending a lot of time in poorly ventilated indoor environments. These elevated levels are associated with a decline in cognitive performance in a variety of studies. I had first heard all of this years ago when I came across this video which is fun to watch but, as I’ll argue, presents a one sided view of the issue[1].

This level of exposure is probably fine for both short and long term effects but essentially everyone alive today has not experienced pre industrial levels of CO2 which might be making everyone very slightly dumber. I don’t think this is super likely and if it happening it is a small effect. But, it is also the kind of thing I would like to be ambiently aware of and I am kind of disappointed in the lack of clarity in the academic literature. Some studies claim extremely deleterious effects from moderate increases in CO2[2], some claim essentially none even with 4000ppm[3], ten times the atmospheric concentration.

The main graphs from the above studies show ridiculously different results. These were intentionally chosen to contrast and make the point.

A lot of the standard criticisms of this kind of thing apply, underpowered studies, methodological flaws for measuring cognitive performance or controlling CO2 concentration, unrepresentative populations[4], and p-hacking via tons of different metrics for cognitive performance. All of this makes even meta analysis a little unclear. This blog post covers a meta analysis pretty well and the conclusion was that there is a statistically significant decreases in performance on a Strategic Management Simulation (SMS) but that was comparing <1500ppm to <3000ppm which is a really wide range and kind of arbitrary. However, nobody has done the experiment I think would be most interesting. That being a trial where subjects are given custom mixes with 0ppm, 400ppm, and 800+ppm. This would answer not only if people are losing ability from poorly ventilated space but also if we are missing out on some brain power if we had no CO2 in the air we breathe in. Again, the effect size is probably pretty small but one of the studies was looking at a drop in productivity of 1.4% and concluding that that level of productivity loss justified better ventilation. Imagine if the whole world is missing out on that from poor ventilation. Imagine if the whole world is missing out on that because we are at 400 instead of 0. Again, not likely but the kind of thing that would have big (cumulative) downsides if true.

I tried looking at the physiological effects of CO2 and did not do as deep a dive as I would have liked but this paper claims that there is a dose response relationship between cerebral blood flow and CO2 concentration (in the blood) and that it really levels out beneath ~normal physiological levels. I take this to mean that there would be a small, but measurable, physiological response if I could remove all the CO2 from my blood, which they did by hyperventilating.

Along the way I started looking at physiological effects of O2 availability and, well, I have some words about a particular article I found. Look at this graph:

It looks like there is some homeostasis going on where your cerebral blood flow can go down because there is more oxygen in the blood (%CaO2) giving you the same amount delivered (%CDO2). The only issue is that they said “When not reported, DO2 was estimated as the product of blood flow and CaO2.” When I read that I felt like I was losing my mind. Doesn’t that defeat the whole purpose of looking at multiple studies? If you just assume that the effect is given by some relation, fill in data based on that assumption, and average out with real data of course you’re going to get something like the relation you put in. As one of the many not doctors in the world, maybe I should stay in my lane but this does strike me as a bit circular. I am not convinced that an increase in atmospheric O2 does not lead to an increase in the O2 delivered to the brain. Especially because decreases in O2 partial pressure are definitely related to decreases in O2 (and cognition) in the brain and it would be kind of weird if the curve was just totally flat after normal atmospheric levels[5].

I also found one very optimistic group claiming that breathing 100% O2 could increase cognitive performance in two main papers. They are both recent and from a small university so it makes sense that this didn’t get a ton off attention but that doesn’t really make me less skeptical that it’s just that easy. The first paper claimed 30% increase in motor learning and I would expect that effect size to decrease significantly upon replication.

All this leaves four main possibilities the way I see it:

1. No effect, everything is business as usual for usual O2/CO2 ranges
2. CO2 decreases cognitive ability with a dose response relationship even at low doses
3. O2 enriched air can have significant gains that basically nobody has captured
4. VOCs[6] have bad effects and ventilation reduces their concentration and that is what confuses the hell out of all these studies

 

My solution

Well, I don’t have the resources to do a randomized control trial. But, I do have the ability to make a CO2 scrubber and feed the treated air into a facemask so I can breathe it. If I do this, I’m not buying the parts until I confirm nobody leaves a comment just demolishing the central thesis, I would probably wait until spring as opening my windows seems like a big important step to having low ambient CO2[7] but would be pretty miserable for me while there’s still snow outside.

This is a chance to talk about some cool applications of chemistry. The idea is that CO2 can react with NaOH to form only aqueous products, removing the CO2 from the air. These can then react with Ca(OH)2 to yield a solid precipitate which can be heated to release the CO2 and reform the Ca(OH)2. This is, apparently, all pretty common for controlling the pH of fish tanks so that’s convenient and cheap.

I’ve already been trying to track my productivity along with a few interventions so I plan to just roll this in with that. This won’t be a blinded trial but I am happy to take a placebo win if it increases my productivity and if it doesn’t do anything measurable I’m really not interested in it.

As for oxygen enrichment, you can buy oxygen concentrators, nitrogen filters that people use for making liquid nitrogen instead of liquid air, medical grade oxygen, oxygen for other purposes, or make it with electrolysis. All of these strike me as being somewhat dangerous or quite expensive to do for long periods of time. Someone else on LessWrong wanted oxygen (for a much better and less selfish reason) and got some for divers/pilots. I would do that, but again, expensive.

With any luck, I will have a case study done on myself at some point and can update everyone with the results.

1. ^

   I don’t want to be harsh, the video is only a few minutes long, is made by a climate activist who already has some strong beliefs on CO2, and he did put his own mind on the line as a test case to make a point which I applaud. Given those reasons and that he seemed to have quite negative effects from the CO2 himself I think it is quite fair that he didn’t have a detailed counterargument presented.

2. ^

   https://pmc.ncbi.nlm.nih.gov/articles/PMC4892924/pdf/ehp.1510037.pdf

3. ^

   https://www.nature.com/articles/s41526-019-0071-6

4. ^

   The group used “astronaut-like subjects” which is fine but I don’t know if that generalizes to most other people.

5. ^

   Not hugely surprising though, we did evolve to use the atmospheric level so I wouldn’t be shocked if it was flat, just that this study didn’t convince me that it was flat.

6. ^

   I realized I did not talk about VOCs, volatile organic compounds, at all. They are just a wide variety of chemicals that permeate the modern world and are probably bad in ways we aren’t certain of.

7. ^

   As an aside, I would not be shocked if poor ventilation during the winter was a contributing factor to seasonal affective disorder but I don’t have that and did not look into anyone checking if it is true.

Discuss