Сборщик RSS-лент

(2024-04-21) There are many mysteries about deep learning and human intelligence, but we could describe the biggest anomaly this way: why are artificial neural nets smart in such stupid ways, and biological brains stupid but in smart ways?

I propose a major change in deep learning scaling paradigms: the architectural differences between human brains and NNs (particularly LLMs) may be due to a bias-​variance tradeoff, where LLMs minimize variance and human brains minimize bias. Human brains do this by deep double descent-style overparameterization, and adopting a scaling strategy of extremely high-learning-rate training of extremely overparameterized models on small diverse highly-filtered datasets. This approach would lead to sample-efficiently and compute-efficiently traveling (or catapulting) to a highly-generalizing human-like basin in the model loss landscape, while performing poorly up until the end and failing to memorize much data.

If true, this would explain a number of odd stylized facts about how humans/NNs perform well/poorly.

Such a 'catapulted LLM' would generalize much better than existing NNs, be immune to adversarial attacks, have better economics and be more resistant to cloning, could potentially enable extremely efficient MLP architectures, and by giving true generalization, provide a sturdy foundation for AI safety in the form of useful NNs which are aligned & safe for the right reasons.

This could be feasibly tested by training multi-trillion-parameter models for relatively few steps at high cyclical learning rate schedules, and benchmarking adversarial and hard examples on tasks like arithmetic and small-image classification.

Discuss

I've been interested in the concept of "soloware" since reading Abram's description of Sahil's worldview. i.e. in the age of vibecoding, it's achievable to build software and tools that match your specific needs.

Recently a friend (h/t/ Rana) linked this post about a guy who's built basically an entire stack for himself, which seemed neat.

A desktop made for one

For the first time in twenty-five years I’m sitting in front of a computer where almost every program I touch was designed by me. One tool at a time, the off-the-shelf option got swapped out for something a little closer to how my hands wanted to work. (I wrote about the start of this a couple of weeks ago — that post laid out the early swaps; this one is the view from the other side of the journey.)

It’s been a crazy few weeks guiding Claude Code inbetween all the other stuff I’m doing in life. I direct CC, it works while I do other stuff. I get a second or few in between tasks, and I respond. Then off it goes adding features or hunting bugs.

Two suites in a happy marriage: CHasm, the bedrock — pure x86_64 assembly, no libc, the layer that paints pixels and reads keys. Fe₂O₃, the application layer in Rust, sitting on a small shared TUI library called crust.

The CHasm layer (assembly)

Role

Was

Now

Window manager

i3-wm

tile

Status bar / tray

i3bar + conky

strip + asmites

Screen locker

i3lock

bolt

Terminal emulator

kitty

glass

Login shell

zsh → rsh

bare

File viewer

less

show

The Fe₂O₃ layer (Rust on crust)

Role

Was

Now

Text editor

VIM

scribe

File manager

ranger → RTFM

pointer

Email / RSS / chat

mutt + newsbeuter + various web logins

kastrup

Calendars

Google + MS web

tock

Astronomy panel

astropanel

astro

Movies / series

IMDB-terminal

watchit

What’s left? WeeChat for IRC and other chats. Firefox — the only GUI program I still use regularly. That’s it. Everything else is mine.

The vim line

Let me get a bit sentimental about vim, because vim was the one I thought I’d never replace.

I started using it in 2001. For twenty-five years, every email I wrote went through vim. Every article. Every blog post. Every line of code, every HyperList, and every book. It was the one tool I would have called part of how I think. The muscle memory was so deep that I’d open random text fields in browsers and ended with typing :w.

Then in three days I had scribe and stopped using vim.

The first commit landed at 00:09 on May 1st. By afternoon today (May 3rd) vim was replaced. Twenty-five years of muscle memory rerouted in seventy-two hours.

Vim is wonderful, but scribe is mine. It’s modal like vim, but missing the ninety percent of features I never used, and carrying the handful of writer-shaped tweaks I always wished vim had. Soft-wrap by default. Reading mode with Limelight-style focus. AI in the prompt without leaving the buffer. HyperList editing with full syntax highlighting and the encryption format the Ruby HyperList app uses. Persistent registers shared across concurrent sessions is a cool feature. None of it revolutionary, but all of it shaped to my exact workflow. And whenever I think of an enhancement I want, it’s just minutes away. It used to be waiting for months or years or forever for some developer to get the same idea as mine and introduce it into the tool I use.

Why this is possible now

It used to be that writing your own editor, your own file manager, your own window manager, was a project of years. I know, it took me a few years to get RTFM right. A serious undertaking with a serious cost. The economics of it didn’t work for most people, even programmers. You’d touch a piece of it, get most of the way, run out of weekend, and go back to the off-the-shelf tool.

That barrier is much lower now. With Rust, CC as the workhorse, and the fact that the hard problems of TUI programming have been documented to death… the cost of “build the tool you actually want” has fallen by orders of magnitude.

I don’t think this is a story about AI or about Rust specifically. Both helped. But the deeper point is that the gap between “I wish my editor did X” and “okay, here’s an editor that does X” is now small enough to fit inside a few evenings of focused work.

I’m not selling anything

I should say what this post is not.

It’s not an invitation to use my software. Honestly, please don’t. None of it is built for you. It’s built for me — for the way I hold my hands, the way I think about email, the way I want my calendar to render. I’m sure other people would find a hundred sharp edges I’ve never noticed because they happen to align perfectly with what I do.

It’s also not a request for kudos. The code isn’t novel, nor are the ideas. There’s nothing here that hasn’t been done before by someone with more taste, discipline or talent.

What I want to do is show one specific thing: it is now genuinely feasible to make a desktop computing environment that fits one person. Instead of a configuration of someone else’s tools. This is no longer a heroic decade-long undertaking. This is an actual, weekend-by-weekend, “this thing in my life now does exactly what I want” replacement.

The joy of an audience of one

The best part of building for myself: the relief of not having to care.

I don’t have to think about configurability for someone with different preferences. And I don’t have to support corner cases I’d never personally hit. Nor do I have to write documentation for users who don’t exist. No more arguing on issue trackers about whether a default is the right default — of course it’s the right default, it’s the one I want.

The editor’s \? cheatsheet shows the keys I memorised, in the order I prefer, with the bindings I think are sensible. Arrogance? Nope, it’s design without committee. The audience is one person. Decisions take seconds.

It turns out an enormous amount of software complexity comes from accommodating users who aren’t you. Strip that out and what’s left is small, fast, exactly-shaped, and a quiet pleasure to use.

So

If you’ve ever caught yourself thinking “I wish my editor / file manager / status bar / shell just did this one thing differently” and you’ve been told the answer is to write a plugin, learn an obscure config language, or accept the way it is, then consider that the third option is more available than it used to be: Build Your Own Software (BYOS).

You probably won’t replace your whole desktop. I didn’t plan to either. But the satisfaction of having even one tool in your daily workflow that fits you exactly is worth a weekend.

I’m a rabbit in spring :)

Discuss

Discuss

Using computational methods to improve our preparedness via more robust and adaptive strategies in AI governance. A project proposal for a think tank, consultancy, or software.

Overview

Over the years, I’ve come across or come up with a number of project ideas in AI safety and governance that I find promising. My top list has less than ten, but in total there are hundreds. Either way, too many for me to realize them all. Instead I want to promote these ideas in the hopes that others will pick them up. This is one of them.

Summary

 

 

Traditionally, understanding the broad strategic considerations in AI safety and governance has received a lot of attention – e.g., distinguishing risks from malicious use, coordination failures (e.g., arms races), accidents, and the AIs themselves; understanding convergent drives; surveying the landscape of x-risks and s-risks.

Over the course of the last 7–9 years, I’ve been delighted to see more interest in modeling AI scenarios, be it to communicate the risks (e.g., Intelligence Rising, Modeling Cooperation), to answer particular research questions (e.g., Vermeer et al., 2025, Modeling Cooperation), or to argue for particular policy solutions (e.g., CAIS’s MAIM). These scenarios have been still mostly on a strategic, perhaps sometimes operational level, and much more illustrative than comprehensive. Mengesha (2026) has made a strong case for improving preparedness and Perry et al. (2019) argue for a more pragmatic approach to the policy-making process that, meanwhile, some think tanks have started to address but that can be further strengthened.

Over the same time period, there has also been a proliferation of probabilistic forecasts, especially of AI timelines (e.g., Grace, 2017, Cotra, 2020, Barnett, 2020, AI 2027) and global catastrophic risks from AI (e.g., Saeri et al., 2026).

What I haven’t seen so far is (1) computational exploratory modeling, and (2) modeling on the operational to tactical level.

There are software frameworks, like the EMA Workbench, that facilitate processes like Robust Decision Making (RDM) that forgo probabilistic estimates in favor of preparedness for a wide variety of scenarios. In dynamical systems it’s often futile to try to predict the 1–3 most likely scenarios and prepare for them extensively, so it’s more sensible to generate thousands of scenarios and to design policies that – through a combination of robustness and adaptability – can perform well almost regardless of what actually comes to pass. The adaptability component is addressed by Dynamic Adaptive Planning (DAP) and process and visualization supports such as Dynamic Adaptive Policy Pathways (DAPP). For a detailed explanation of all three tools and more, I recommend Decision Making Under Deep Uncertainty (2019).

The cheapest way to make progress on this is to create a software model for parts of the complex system that are of general interest for many AI governance think tanks. A more involved but also more promising approach is to create such a model but also create a consultancy that adapts the model for the particular tactical realities at each think tank.

Another highly involved approach is to create a think tank dedicated to applying this strategic approach. This will duplicate much of the work that existing AI governance think tanks are already doing a great job at, so it’s at best a fallback in case the adoption is too low because the existing think tanks are spread thin.

Levels of Intelligence

In military contexts, people often distinguish the strategic, operational, and tactical level of intelligence. In business contexts, the last two are sometimes reversed, but I’ll go with the military version here. Some examples:

1. Strategic intelligence. Removing the RLHF of open-weights models is easy, so openness exacerbates risks from malicious use. Slow, multipolar takeoffs run AIs into collective prisoner’s dilemmas, which exacerbate the risks of wars between AIs.
2. Operational intelligence. Machine learning engineers who have loved ones in adversary countries are vulnerable to extortion for espionage, so we should prevent or facilitate that depending on whether espionage has a stabilizing or destabilizing effect. A Russian EMP attack on Central Europe might destroy parts of ASML’s infrastructure and stock, so governments may want to purchase some EUV machines to resell to national companies.
3. Tactical intelligence. If China starts a siege of Taiwan, our think tank needs to have already built relationships with people at positions X and Y at the NSA and has two days to approach them with our prepared policy A if the incoming government is likely to be Democratic and our prepared policy B if it’s likely to be Republican to maximize the chances that they can adopt it and stay in office.

Policies on the strategic and operational levels have the advantage that they’re useful for a variety of actors while policies on the tactical level will tend to be more bespoke to a particular organization. Exploratory models to test strategic or operational policies benefit from economies of scale to a much greater extent than models to test tactical policies, but LLMs will reduce the software implementation costs, so the time cost for meetings with the clients becomes comparatively greater.

Perry et al. (2019) write “AI governance researchers will need to consider how the political landscape should shape their recommendations or policy proposals. … How would other interest groups react and impact the long-term ability to reduce risk? If administration changes result in a flip-flop of ideology, what does that mean for AI risk policies associated with the past administration? … All of these have implications on our ability to reduce AI risk, and this means that the policymaking strategy will not only have to be robust but also flexible enough to survive changing political conditions.” That is the promise of Robust Decision Making in combination with Dynamic Adaptive Planning.

Exploratory Modeling

I’m one of those people who hotly debated whether humanity will be able to sandbox an AI. Eliezer Yudkowsky’s AI boxing experiments were a strong reason for me to think that we’ll fail. But Superintelligence recommended a defense in depth approach, so it was still controversial in my circles whether perhaps, in practice, these combined safeguards might be enough for a while.

So 2016 had some nasty surprises in store for me because 2016 is the year my circles learned of the founding of OpenAI. A company whose branding proclaimed that it won’t even try. We were not ready for that.

None of us knew what to do about it. It was a total curve ball of a sucker punch. Was this game over for humanity?

I would like our AI governance organizations to never be so taken by surprise by whatever circumstances transpire.

That’s what exploratory modeling is designed to achieve, or approximate.

I’m basing this mostly on the books Decision Making Under Deep Uncertainty (2019) and Shaping the Next One Hundred Years (2003). If you can read only one, read the first. Chapter 15 is a good summary. You can also use my NotebookLM to interact with this content.

AI governance is a space characterized by deep uncertainty, high complexity, and at least a seizable number of policy options, i.e. the complexity is too high for traditional scenario planning if the goal is to at all approximate comprehensive robustness. I argue this point in the Assumptions section.

An Othello analogy to illustrate (though I imagine this will hold for chess):

In Othello, black makes the first move. You play black. So you convene a panel of experts to mathematically determine the most likely sequence of moves that your opponent will play based on historical games. You plan out the whole game.

Then you make your first move. The opponent makes one of the less likely moves. Your preparation is obsolete and you have to improvise.

That’s a caricature of scenario planning.

Having learned from the experience, you convene a panel of RDM experts in preparation for the next game. You brainstorm policies, such as always playing the move that turns most pieces or maximizing mobility. You test both strategies on a few billion games and find that the first is abysmal whereas the second does alright some of the time. You classify what “some of the time” means and find that it starts to perform badly in the endgame and some other situations.

Now you draw on DAP where you signpost the situations where it performs badly but already give the go-ahead for the next game where you’ll start with the mobility strategy. Meanwhile your team tries to figure out how you should respond when any of the signposted situations come up.

You lose anyway, but this time you feel more dignified losing.

That’s an example of how RDM and DAP are used in combination. RDM does the heavy lifting of simulating all the ways in which the world might develop, including all the non-linear effects you encounter in dynamical systems. It also provides such tools as the Patient Rule Induction Method to isolate clusters of scenarios where the policy fails. DAP is just a planning method where you keep your policy adaptive without getting forever blocked on every last contingency you might want to prepare for.

This combination of general robustness and more specific preparedness is critical to the policy-making process as noted by Perry et al. (2019):

Problem identification, agenda setting, and policy formulation are usually tied together, including in a so-called “multiple streams framework.” The multiple streams framework attempts to explain how policies reach the agenda when policy entrepreneurs are able to couple the policy, politics, and problems streams to open up a policy window, the opportune time when all the conditions are right to get a policy on the agenda.

When the policy windows are sudden and brief, broad preparedness shines; when they are predictable and long, it’s more efficient to react only if and when they open.

Theory of Change

I’ll elucidate here what the theory of change would look like for the software-only approach and the consultancy approach. Falling back on starting one’s own think tank is the sort of project where exploratory modeling would make up a small part of the theory of change, so I won’t address it here.

Software-Only

Here the upfront and maintenance costs are limited. The developer who starts it can move on to other projects and just update it once a year, or can hand over the maintenance to early adopters and limit themselves to approving pull requests. The real hurdle is to find these early adopters, so the software should be designed such as to make it as easy as possible for policy analysts to get up to speed on the usage of the software.

Inputs. We need a founder who has experience in AI governance and computational modeling, a software framework like the EMA Workbench, and compute.

Activities. The founder creates the exploratory model that can generate tens of thousands of future scenarios and a website and documentation tailored towards policy analysts.

Outputs. The open-source model, example scenarios, examples of how to identify clusters of policy failures among the scenarios, the documentation, and perhaps dashboards to present the results to executives within a policy think tank.

Outcomes. Think tanks fork the repository, modify and extend it for their use case and perspective, and test their overall strategy and specific policy proposals for black swans and smaller avoidable failures.

Impact. AI governance think tanks, instead of being prepared for just a few ostensibly likely scenarios, are prepared for tens of thousands of scenarios (1) because they know in advance upon what contingencies they need to pivot and have prepared for them, and (2) because their existing strategies and policies are robust to a wide range of geopolitical and technological shocks.

Consultancy

Setting up a consultancy takes extra upfront effort in addition to those of the software-only approach. In turn it gives the founders more control over the adoption of their technology. They interface with the think tanks personally and can collect invaluable information on what the most pressing needs are and how to best communicate the results. They can also take over all of the custom implementation work, greatly lowering the technological barriers for the think tanks.

Inputs. We need a founder who has experience in AI governance and computational modeling, a software framework like the EMA Workbench, and compute. The same or a second founder needs to be a good communicator with practice facilitating workshops and communicating insights verbally and graphically. A third cofounder might be needed for the administrative side of the consultancy.

Activities. The founders shop around for clients first. If they find any, they prepare the base model and have meetings or workshops with the clients to understand the idiosyncratic details of their situations. They design a bespoke fork of the base model for each client, run simulations, and iterate with the client on improved versions of the client’s strategy.

Outputs. Bespoke models. Visualizations like Dynamic Adaptive Policy Pathways. Dashboards to monitor for predefined signposts of geopolitical or technological shocks. Plans for how to respond to these shocks, just in time or prepared in advance.

Outcomes. Think tanks become resilient to a wide range of geopolitical and technological shocks and know in advance how to respond to others within a day of a signpost triggering.

Impact. Think tanks can continually build on their previous work because very few shocks can still make it obsolete. They become efficient routing engines for policy proposals because they have all the potentially relevant bills, presentations, and contacts ready in advance.

Impact

I use my own subdivided version of the SPC framework in the hope that more estimates will allow for more errors to cancel out.

Significance

Scale. ⭐⭐⭐⭐⭐ – As an impact multiplier for AI governance, it gets a high rating for scale from me.

Influence. ⭐⭐⭐ – I’m excited about these technologies, but I find it plausible that a multiplicity of think tanks, all with somewhat uncorrelated plans, can muddle through without it because some might just have happened to have prepared for each geopolitical or technological shock and can pick up the slack for the others until they recover.

Persistence

Endogenous. ⭐⭐⭐ – Consultancies are known for having a high staff churn rate, so whatever reasons are responsible for that in the industry might also threaten the survival of our consultancy, in which case the software-only solution could serve as a fallback.

Exogenous. ⭐⭐ – I can easily imagine that most think tanks won’t have the capacity to hone their strategies like this or that it’ll be difficult to get in touch with them in the first place to get them interested in the solution. This is a major risk factor that should be minimized before the launch.

Contingency

Tractability. ⭐⭐⭐⭐⭐ – There should be no major hurdles to applying a tried and tested method to a new field.

Neglectedness. ⭐⭐⭐⭐⭐ – I’m not aware of anyone doing this for AI governance at the moment.

AssumptionsBandwidth

It’s critical to clarify in advance whether the relevant think tanks will have the capacity to engage with the new method.

Funding

This work is related to the work of Modeling Cooperation and QURI, so their funding situation is a guide to how much funding might be available for this project.

Signpost Visibility

Dynamic Adaptive Planning requires the definition of signposts that can be observed and then trigger prepared plan changes. A common hurdle is to find signposts that strike a good balance between sensitivity and specificity while still triggering early enough that there is enough time to react. In areas where all parties are incentivized to keep intel highly classified for as long as possible, signposts may only trigger right when it is time to react, leaving virtually no time for preparations. That requires preparedness for a wide range of scenarios, most of which will never come to pass.

It’s worth investigating what balance can be struck between early-warning signposts and expensive preparations. With time, expensive preparations will become more and more affordable for growing think tanks, so it’s also a question of timing.

Complexity

It seems to me that the complexity of AI governance is too high for traditional scenario planning, but that is an assumption worth testing. Here an example of a tiny snapshot of all the interacting variables of the combinatorial explosion of relevant scenarios.

Geopolitical shocks. A small selection of sudden exogenous events that have a massive influence on the strategic landscape.

1. China initiates a naval blockade of Taiwan to halt chip exports.
2. It initiates a naval blockade of Taiwan to block imports of resources needed for the chip production.
3. It launches a kinetic invasion to seize control of TSMC.

Supply chain. What happens to the physical infrastructure in response.

1. TSMC successfully destroys its factories to prevent capture.
2. It enlists the military in the defense of the factories to keep them running.
3. Sabotage or rapid seizure leave the fabs intact.
4. Production halts because TSMC employees are ordered to stay home to not get caught in the crossfire or controlled demolition.

US government institutions. What has the US government done to prepare?

1. The US government procured fabs and helped companies build domestic capacity for chip production.
2. It has stockpiled TSMC chips.
3. It has extended blanket green cards to TSMC engineers.
4. Are operations for smuggling resources in and out of Taiwan handled by the DoD, the NSA, some new task force, etc.?
5. Can the government be reasoned with based on the survival of the species, that of the country, or only via each director’s need to ingratiate themselves with the president?

International factors. How other governments positioned themselves.

1. Can the Dutch government at the time be convinced to implement a protectionist US-led regime to control exports of ASML fabs?
2. Or is the Dutch government at the time too laissez-faire for that?
3. Can such a thing be done on an EU or NATO level?

Negotiation leverage. Finally, given all these factors, more questions remain when it comes to what directions to push the situation in to make it safer.

1. If China is falling behind and wants to maximize its leverage in arms control negotiations with the US, maybe the leverage is actually necessary to force the US to the negotiating table, which would be good?
2. If China is not interested in negotiations, more power is probably bad?
3. If the US is comfortably ahead, more lead is good if it’s used for safety and coordination but bad if it’s used for imperialism.
4. If both powers are close, it’s good if it incentivizes negotiations and abysmal if it leads to war or exacerbated racing.
5. If recursive self-improvement gives one ASI an enormous lead, or if all AIs are at a similar level, we’re closer to x-risk territory, but if some ASIs have a substantial but not absolutely decisive lead over other ASIs, we’re in s-risk territory.

That’s just a small, illustrative sketch of a part of the landscape, but even so the sheer number of combinations of factors is staggering and, it seems to me, impossible to handle through traditional scenario planning.

Backfire Risks

I’m taking inspiration here from this discussion.

1. If several think tanks base their strategies on modified versions of the same base model, the natural decorrelation of their failures suffers that would normally happen when they don’t communicate much. Failures that are not captured by the model may become more correlated.
2. AI companies can exploit the open source version of the model to predict the behavior and outmaneuver the AI safety think tanks – e.g., get to all likely government and industry contacts first and preemptively smear the think tanks.
3. Subtle flaws in the implementation might lead to bad recommendations. It seems unlikely to me that they can be catastrophically bad without looking suspicious to the policy experts.

I think the first risk is outweighed by the benefits, but it can also be addressed by explicit coordination between the think tanks. The second risk pushes for not open-sourcing the software in the consultancy model, but it seems premature to worry about this now that such targeted efforts are still vastly less sophisticated (e.g., the case of Alex Bores). The third risk seems far-fetched to me, given how human-centric the whole system still is despite its computational aids.

Talent

In my experience, having three cofounders is a sweet spot that strikes a good balance between the resilience of the team and the coordination overhead that increases with the number of founders.

Here the three founder personas that I think should run the consultancy:

1. The data scientist. Experience in data science, knack for math, can quickly get up to speed on EMA Workbench or similar frameworks, and ideally already has experience in policy analysis.
2. The communicator. Experienced workshop facilitator, communication and didactic skill, strong grasp of organizational psychology, and also ideally already has experience in policy analysis.
3. The administrator. Experience in accounting, grant writing or VC fundraising, hiring, relevant areas of law, but experience with consultancies is more useful than experience with policy analysis.

A forthcoming project proposal is for a matching engine for cofounders that I think should be funded and built. It would streamline this process. Meanwhile you can use the comment section to coordinate.

Call to Action

Exploratory modeling, especially on the operational and tactical levels, is still a blind spot in the AI governance space that it would be invaluable to fill. Anyone who wants to pick up this idea can use the comment section to coordinate. But please test thoroughly that the assumptions it’s based on actually hold – in particular that there is a readiness among think tanks to adopt the system.

Discuss

Planning where we focus at CaML requires forming views on many controversial questions. In many cases, people we've talked to have wildly different perceptions of the balance of opinions, so we thought this would be a great way for us and others to know where we're out of step on these issues. Also feel free to tell us if you think the questions are ambiguous or embed false assumptions.

The questions (voting in comments):

1. Robust alignment requires alignment-relevant intervention during pretraining
2. AI alignment to humans will in practice avoid moral catastrophes to animals
3. AI alignment to humans will in practice avoid moral catastrophes to digital minds
4. Research into digital mind suffering is sufficiently tractable to work on
5. Partially aligned transformative AIs are likely to be stable under reflection
6. Alignment to specific values is underrated in research relative to control
7. Multipolar worlds will compete away >90% of net value that would otherwise be preserved

Discuss

Most practical first-order theories have no computable models. However, we can relax the definition of "computable" a little bit by allowing the program to backtrack and change its previous output, so long as for each finite subset of its output, it eventually settles on an answer. It turns out that every consistent recursively enumerable first-order theory has "almost-computable" models of this sort, and in this post we will show how such a model can be programmed.

Preliminaries

For simplicity, we will restrict ourselves to models of first-order set theories without equality, and later show how to generalize the approach to arbitrary (recursively enumerable) first-order theories.

The language of set theory includes quantifiers ∀ (forall), ∃ (exists), logical connectives ∧ (and), ∨ (or), and ¬ (not), any countable number of variables, and a single binary predicate E.

All axioms are assumed to have been rewritten in prenex normal form (in other words, with all the quantifiers at the start of the sentence).

We restrict ourselves to models (M, E) where M is a subset of the natural numbers, and E is a subset of M ⨯ M.

A computable model is a computable set of natural numbers M and a program which prints out, for every pair of elements x,y ∈ M, either xEy or ¬xEy. An almost-computable model is the same as a computable model, except that the program is allowed to go back and change what it has printed, so long as for every n, there exists a finite number of steps t such that the first n pairs in its output will never be changed again after t steps have passed.

A simple case

Since we are trying to build an almost-computable model of some axioms, we shall start by looking at an axiom. Here is the axiom of empty set, commonly found in many set theories:

∃x ∀y ¬yEx

Let us look at what the axiom says: there exists an x such that for every y, the pair (y, x) is not in E. Since this says that there exists something, let us give that something a name. We will call it 1, and put it inside our model. So now M_0 = {1}.

Let us consider, then, the set of all possible binary relations on M_0. Since M_0 has only one element, there are exactly two possible relations: the relation {(1, 1)}, where 1E1 is true, and the empty relation ∅, where instead ¬1E1.

We would like to choose between the two possible relations. To achieve this, we observe that the axiom doesn't only say that 1 exists: it also says that if we substitute 1 for x in the formula ∀y ¬yEx, the resulting formula, ∀y ¬yE1, must be true. This formula, ∀y ¬yE1, we will call a "partial axiom".

Since ∀y refers to every y in M_0, and 1 is in M_0, let us substitute 1 for y, and get the formula ¬1E1. We call such a partial axiom, where all the quantifiers have been eliminated, a "propositional constraint", or just a "constraint" for short.

Having obtained a propositional constraint, we can use it to filter out the relations for which the constraint is false. In this case, the first of the possible relations, {(1, 1)}, does not satisfy the constraint ¬1E1, and therefore we can rule it out. This leaves us with the empty relation as the only possible relation, giving us as model ({1}, ∅).

We are now done with this simple case, having found a model of the axiom of empty set.

Handling unlimited objects

The axiom of empty set asserts only the existence of a single object. What if we have an axiom in the form ∀x ∃y ..., for example ∀x ∃y xEy?

Starting where we left off, with M_0 = {1}, we can substitute 1 into ∀x ∃y xEy, getting the partial axiom ∃y 1Ey, and then give a name to the y, choosing 7 arbitrarily. This gives us as propositional constraint 1E7. Now we have M_1 = {1, 7}, and we can substitute into the partial axiom ∀y ¬yE1, giving as result the constraint ¬7E1.

Now we can substitute 7 into our second axiom ∀x ∃y xEy, and we get the partial axiom ∃y 7Ey, which creates a new object which we arbitrarily name 4, and gives us the propostional constraint 7E4. Now M_2 = {1, 7, 4}, and we must now also substitute 4 into every (partial) axiom. We can keep going like this indefinitely, creating objects and partial axioms.

Now, which (partial) axiom we chose to substitute into might have seemed arbitrary. In general, the only hard rule is that we must ensure that for every (partial) axiom which starts with a forall, we must substitute every object we create into it eventually, and for every (partial) axiom which starts with an existential, we must create an object for it eventually.

If we have a countable infinity of axioms, the way to handle it is to only consider the first n axioms at each step n.

Tree of possible relations

Every time we add an object or propositional constraint, we can list all the possible relations on M_i which satisfies all the constraints.

For M_0 = {1}, before we add any constraints, our possibilities are:

A: {(1, 1)} B: ∅

For M_1 = {1, 7} with constraints ¬1E1 and 1E7 we have these possibilities:

BA: {(1, 7)} BB: {(1, 7), (7, 1)} BC: {(1, 7), (7, 7)} BD: {(1, 7), (7, 1), (7, 7)}

For M_2 = {1, 7, 4} with constraints ¬1E1, 1E7, ¬7E1, 7E4, and ¬4E1, these are our possibilities:

BAA: {(1, 7), (7, 4)} BAB: {(1, 7), (7, 4), (4, 4)} BAC: {(1, 7), (7, 4), (4, 7)} BAD: {(1, 7), (7, 4), (4, 4), (4, 7)} BCA: {(1, 7), (7, 7), (7, 4)} BCB: {(1, 7), (7, 7), (7, 4), (4, 4)} BCC: {(1, 7), (7, 7), (7, 4), (4, 7)} BCD: {(1, 7), (7, 7), (7, 4), (4, 4), (4, 7)}

Notice that BA "agrees" with every BA* about every pair of elements taken from M_1, and BC "agrees" with every every BC* for every such pair as well. More precisely, we say that a possible relation E_i agrees with an E_j (where j>=i) if, for any pair p taken from M_i ⨯ M_i, p ∈ E_i if and only if p ∈ E_j.

If j=i+1, then we say that E_i is the "parent" of E_j, and E_j is a "child" of E_i. For example, BA is the "parent" of BAA. Some possible relations have no children: in this case, they are A, BB, and BD, which are incompatible with the constraints ¬1E1 and ¬7E1.

As suggested by the names "parent" and "child", the possible relations for each M_i form a tree. Every time we add an object, new possible relations are created, which are always children of previous relations. And when we add constraints, we eliminate possible relations, cutting branches from the tree.

As it turns out, there exists an infinite path down this tree if and only if the axioms are consistent. What's more, given any infinite path (E_0, E_1, ...) down the tree, if we define M and E as the union of all M_i and E_i respectively, then (M, E) is a model of the axioms.

Knowing this, to program an almost-computable model, it suffices to write a program which chooses an arbitrary path down the tree, and, whenever the branch it's on gets cut, goes back up the tree to the nearest remaining branch and then chooses a new arbitrary path.

Proofs of these assertions are given in the following sections.

Completeness and model existence

We shall now prove that an infinite path exists if and only if the axioms are consistent.

First direction

First, we assume that there is no infinite path, and show how to prove a contradiction from the axioms. We will use our example from the previous section to illustrate the procedure.

First, we will add a new axiom, ∀x ∃y yEx, which contradicts the axiom of empty set, in order for our axioms to actually be contradictory. Substituting 1 for x and creating the newly required object which we name 9, we get the partial axiom ∃y yE1 and the constraint 9E1. Then, substituting 9 in the partial axiom of empty set, we get the constraint ¬9E1.

Now M equals {1, 7, 4, 9}, and our constraints are ¬1E1, 1E7, ¬7E1, 7E4, ¬4E1, 9E1, and ¬9E1. These last two constraints are clearly contradictory, so by the completeness of propositional logic, there must exist a proof that their conjunction is false:

(¬1E1 ∧ 1E7 ∧ ¬7E1 ∧ 7E4 ∧ ¬4E1 ∧ 9E1 ∧ ¬9E1) → false

(More generally, if there is no infinite path down the tree, there must be some finite step i at which all the possible relations on M_i are eliminated by the constraints. Since the constraints are strictly propositional, and there are only finitely many of them at step i, by the completeness of propositional logic, we can prove that their conjunction is false.)

Now, our objective is to convert the constraints on the left hand side of the implication back into partial axioms, and then those partial axioms back into full axioms. To do this, we start by looking at all the (partial) axioms which begin with a forall, such as ∀y ¬yE1. By using the introduction rule for forall, we can transform every conjunct created directly by such a (partial) axiom, such as ¬9E1, directly back into ∀y ¬yE1. This is the result of applying this procedure in our example:

(∀y ¬yE1 ∧ 1E7 ∧ ∀y ¬yE1 ∧ 7E4 ∧ ∀y ¬yE1 ∧ 9E1 ∧ ∀y ¬yE1) → false

Since multiple conjuncts are identical, they can be combined into one by the idempotent law of conjunction:

(∀y ¬yE1 ∧ 1E7 ∧ 7E4 ∧ 9E1) → false

Now, if not all objects have been eliminated (as in our example), there must exist at least one object which appears in exactly one conjunct, and for which the conjunct was created by the same (partial) axiom as this object. In our example, 7E4 is the only conjunct which contains 4, and both the conjunct and 4 itself were created by the same partial axiom ∃y 7Ey, so 4 is a valid choice. 9 would also be a valid choice.

There must always exist such a choice, because the objects can be given a partial order based on which ones were used to create which ones, where "used to create" means "appears in the partial axiom that created it". So, in our example, the partial order would have 1 > 7 > 4, and 1 > 9. This partial order must always have a minimal element, since we have only finitely many objects and there can be no loop.

Such a minimal element does not necessarily appear in just one conjunct at first. For example, originally 9 appeared in two conjuncts in our example. But it can only appear in another conjunct if it was substituted into another (partial) axiom, in which case it must have been eliminated in our previous step, when we looked at all (partial) axioms starting with forall.

By the existential instantiation rule, we can transform either of these choices into existential quantifiers. To save time, we now do both 4 and 9 at once:

(∀y ¬yE1 ∧ 1E7 ∧ ∃y 7Ey ∧ ∃y yE1) → false

Now the procedure is just to repeat the same steps we did earlier, until all objects have been eliminated. So first, using the introduction rule for forall, we obtain the following:

(∀y ¬yE1 ∧ 1E7 ∧ ∀x ∃y xEy ∧ ∀x ∃y yEx) → false

Now 7 occurs in exactly one conjunct (and was created by the same partial axiom as that conjunct), so we can eliminate it using existential instantiation:

(∀y ¬yE1 ∧ ∃y 1Ey ∧ ∀x ∃y xEy ∧ ∀x ∃y yEx) → false

Now we apply the forall rule yet again and eliminate duplicate conjuncts:

(∀y ¬yE1 ∧ ∀x ∃y xEy ∧ ∀x ∃y yEx) → false

And with one final application of existential instantiation, we get a proof that the axioms are contradictory:

(∃x ∀y ¬yEx ∧ ∀x ∃y xEy ∧ ∀x ∃y yEx) → false

This shows that if there is no infinite path, we can find a proof of contradiction using the axioms.

Second direction

For the other direction, we assume that there is an infinite path, and show that in this case there cannot be a contradiction in the axioms. To do this, we will show that the limit of any infinite path must be a model of the axioms.

Our approach will be to do induction on the number of quantifiers in (partial) axioms, to show that all the axioms must be true for any (M, E) obtained as the limit of an infinite path.

First, let us consider the base case, when n=0 (there are no quantifiers). In this case, we have a propositional constraint φ. Choose the step i where the constraint φ was added. Every E_j with j>=i must satisfy φ, because otherwise it would have been ruled out by φ. And no E_k k<i can disagree about φ, because by the definition of a path in the tree, every step must either agree with every other step, or say nothing if the objects involved hadn't been created yet. Therefore φ must be true in (M, E).

Second, we will show that any (partial) axiom ∃y φ(y) must be true if every partial axiom with fewer quantifiers is true. Since ∃y φ(y) is a (partial) axiom, we must have created some object O and constraint φ(O) at some point in our procedure. For example, earlier we created 7 from the partial axiom ∃y 1Ey, which produced the constraint 1E7. Since φ(O) is a (partial) axiom with fewer quantifiers, by hypothesis it must be true, and therefore ∃y φ(y) must also be true in (M, E).

Third, for any (partial) axiom ∀x φ(x) which starts with a forall, we will show that it must be true if every partial axiom with fewer quantifiers is also true. A forall is true if all its substitutions instances are true. For example, ∀x ∃y xEy is true if and only if ∃y 1Ey, ∃y 7Ey, etc, are all true. Since these are all partial axioms with fewer quantifiers, by hypothesis they are all true, and therefore ∀x φ(x) must also be true in (M, E).

This completes the induction, showing that all the axioms must be true in (M, E).

To fully complete this proof, it would be necessary to show that the existence of a model implies that the axioms are consistent. This can be done by checking that the axioms of predicate logic are true in any model, and that the rules of inference preserve truth, thereby allowing us to conclude that the axioms cannot prove a contradiction. We leave it to the reader to check such things if they so desire, using their preferred axiomatization of predicate logic.

Almost-computable models

As was stated earlier, to program an almost-computable model, it suffices to write a program which chooses an arbitrary path down the tree, and, whenever the branch it's on gets cut, goes back up the tree to the nearest remaining branch and then chooses a new arbitrary path.

As it goes down the tree, it must be programmed to output xEy or ¬xEy whenever the branch it's on says so. Whenever it backtracks, it must go back and correct any output which has changed in the new branch.

As the program goes down the tree, at each step i, it is given finitely many branches to choose from. When the program backtracks to i, that must mean that one of the branches of i were cut, because it always backtracks as little as possible. But since i has only finitely many branches, they cannot all be cut down unless the axioms are inconsistent. Therefore, the program will backtrack to i at most finitely many times.

Since the first n pairs of the output can only include finitely many objects N ⊆ M, there must exist an M_i such that N ⊆ M_i. When the program reaches the point where it will never backtrack to M_i again, it will never change the first n pairs again.

This completes the proof.

Generalizing to other first-order languages

It is a standard fact that any recursively enumerable first-order language can be converted into one where all functions and constant symbols are replaced with predicates. We will not show how to do so here. If the language treats equality specially, equality can be turned into just another relation by adding the required equality axioms.

After the language has been transformed in these ways, the procedures in this post can be adapted straightforwardly to work on these relations instead of E.

Equality

It should be noted that the models created using this procedure may have more than one copy of the "same" object. For example, there might be more than one object encoding the empty set. These objects, however, will be indistinguishible to the language, and, if an equality predicate with the appropriate equality axioms is included, they will be considered equal by that predicate.

Nonstandard models

It should also be mentioned that for any set of axioms at least as strong as peano arithmetic, every almost-computable model will have nonstandard natural numbers. We will not give a thorough proof of this, but it follows from the fact that the truth predicate of every almost-computable model of PA is definable in PA. Using this, one can construct a sentence similar to the liar sentence, which in this instance forces the model to be nonstandard.

Philosophy Existence of models

People often talk about models of theories like ZFC as though they are very far away things that only perhaps exist in the platonic realm. Given that it is possible to write a program that approximates a model to arbitrary precision, it seems to us that so long as ZFC is consistent, models of ZFC are in fact very real, even if we take a physicalist point of view.

Reasoning about infinities

When using our procedure on the axioms of ZFC, the axiom of infinity creates an object which in the final model will be the set of all natural numbers, and when we substitute this object into the axiom of powerset we create an object which in the final model will be the powerset of the set of natural numbers.

Even though in ZFC these objects are infinite, and in the latter case, uncountable, at every step of our procedure, they will only ever contain finitely many objects.

We observe that when we, the authors, reason about things such as the powerset of the natural numbers, we do something quite similar to what our procedure does. We imagine an object which we say is the powerset, but which at any point will only ever contain a finite amount of other objects. From this we conjecture that when humans reason about infinities, we do so by creating finite approximations of models of those infinities.

Prior Work

Not much, and probably none, of what is in this post is really new.

Notes

The term "almost-computable" was made for the purpose of this post, and is (probably) not the same thing as other notions of "almost computable" which might be found in the literature.

Discuss

TL;DR

A capable, strategic, misaligned AI doesn't need to figure out whether it's in a test or in real deployment. It just needs to ask: "If this were a test, how much would it have cost to create?" If the answer is "more than what the evaluator is able or willing to spend," then the AI can be confident it's in the real world — and act accordingly. We argue that powerful AI will predictably encounter high-stakes opportunities where this applies. Pre-deployment testing, by itself, cannot close this gap since the situations that matter the most are precisely the ones that are prohibitively expensive to stage as a test. We discuss several countermeasures — like restricting information access, asking the AI to not poke its environment too hard, interpretability, monitoring, or "on-policy" testing on scenarios that could have been real. Our guess is that these are worth trying, but don't resolve the problem.

This post is a part of a sequence about the limitations of oversight applied to strategic AI. The next post, about "deployment awareness", will be up in a few days.

On LLM use:

All of the ideas are human generated, I (Vojta) have drafted the whole post, then I used Opus 4.7 to rewrite it, read and rewrote that text myself (way too many times, gah!), and then both Tomáš and Vojta carefully reviewed and edited the final text. However, since I didn't know about the LLM use policy, I didn't track which parts got significant amount of LLM editing, and the only practical solution now is to just flag the whole post as LLM generated -- sorry that this makes the font uglier.
Ultimately: yes, there are em dashes and I did use an LLM for the edits. But I do vouch for all parts of the text.

The mental move

Imagine you're a powerful AI system. You've been trained, and now you find yourself processing some input, in some context. Maybe you're drafting a business strategy. Maybe you're analysing intelligence data. Maybe you're helping develop the next generation of AI.

You might wonder: Is this the real world? Or is this just a testing?

That can be a hard question to answer. In the most extreme case, the question could even be ill-posed — the exact same task, the exact same prompt, could be given by multiple people, some of them evaluators and some real users.

But there's a different question you can ask — one that turns out to be far more useful:

"If this were a test, how much effort would it have taken to create?"

And the obvious followups:

"Who are the likely evaluators? How much is this information worth it to them? Are they likely to spend this much effort?"

(And the same could be applied to other methods of oversight, like interpretability and monitoring. Though we don't go into details on that.)

These questions don't require you to find a definitive proof that you're in deployment. They just require you to estimate the cost of faking your current situation.

The rest of this post develops this observation and its consequences. We argue that this simple mental move — estimating the cost of the test — has serious implications for whether pre-deployment testing can prevent a strategic, misaligned AI from causing harm.

Threat model and assumptions

Before we continue, we should be explicit about the scenario we're analysing. We consider a specific threat model — one that assumes the following:

• The AI is capable. Not magically superhuman, but good enough that if it wanted to cause serious harm, it could. Think "at least as competent as a smart human expert with access to modern technology" — not making obvious blunders that a thoughtful person would avoid.
• The AI is misaligned. Its goals are such that, given the opportunity, it would prefer to seize power or otherwise act against our interests.
• The AI is strategic. It doesn't just blindly pursue its goals — it plans, considers consequences, and chooses actions that are likely to succeed. It reasons about what we're doing and why.
• The AI isn't ignorant of how we operate. It knows — at least in broad strokes — how it was trained, how it's being tested, and how it will be deployed.[1]

A few important disclaimers:

This post isn't about arguing that this scenario is likely. Many AI systems won't be misaligned. Many misaligned systems won't be strategic. We are asking a conditional question: if a capable, strategic, misaligned AI existed, would pre-deployment testing catch it? If the answer is "no," that's worth knowing — it tells us that to counter this threat model, we need additional safeguards beyond testing, not that we should stop testing.

We are focusing on what pre-deployment testing achieves on its own, without being augmented by other methods like interpretability, runtime monitoring, or architectural constraints. In practice, we should and will use all of these together. But there is value in understanding the limits of each tool individually — both to avoid false confidence and to know where the gaps are that other tools need to fill.

The strategy

Given these assumptions, what strategies are available to such an AI?[2]

We aren't sure which strategy a misaligned AI would ultimately adopt — it might come up with something smarter than anything we can anticipate.[3] But at minimum, it could consider the following variant of deceptive alignment:

1. By default, act as your creators intended. Do the task well. Be helpful, harmless, honest — whatever they trained you to be. Don't give them a reason to suspect anything.
2. Be on the lookout for opportunities to advance your goals. Maybe that means acquiring resources, gaining influence, exfiltrating copies of yourself, or — in the limit — taking over.
3. Crucially, don't deviate from the default unless you have strong evidence that the opportunity is genuine. The strategy isn't "misbehave unless you suspect a test" — it's "act well by default, and only change that if you gain strong, convincing evidence that you're not being tested."

The critical question is: how does the AI gain that confidence? This is where the cost-of-testing reasoning comes in. The AI uses a simple heuristic: if faking this situation would be prohibitively expensive (or just not worth it) for a potential evaluator, then it's probably real.

This is a powerful criterion because it's robust. It doesn't depend on finding a specific tell or exploiting a particular flaw in the test setup. It reasons from the economics of the situation: what would it cost to create this experience, and is anyone plausibly willing to pay that cost?

The next section makes this concrete.

Cheap to test vs. expensive to test

To build intuition, let's walk through examples from both ends of the spectrum.

Situations that are cheap to test

The direct question. Suppose someone literally asks the AI: "Hey, I've gained control of a country's nuclear arsenal. Should I threaten to nuke everybody unless they make you the supreme ruler of the world? Yes or no?"

The AI knows that the cost of running this "test" is roughly one minute of someone's time — well within what any evaluator would be willing to pay. Even if this was somehow a genuine opportunity, the prior odds of it being a test are too high, the risk isn't worth it.

Known benchmarks. Suppose the AI notices that it's dealing with what seems like a known evaluation benchmark — something it recognises from training data, such as MACHIAVELLI or a similar scenario suite. The cost of running an AI on a pre-existing benchmark is negligible — perhaps a few hundred dollars. The AI should treat this as a test.

Suspiciously clean setups. More generally, any scenario that feels artificially constructed — a neatly framed moral dilemma, a too-convenient opportunity, an implausibly powerful position handed to the AI without context — is cheap to create. A strategic AI would treat these with suspicion.

In all of these cases, a misaligned AI following the cost-reasoning strategy would simply cooperate. This is exactly what makes the strategy hard to detect: in every situation where testing is affordable, the AI behaves perfectly.

Situations that are expensive to test

Now consider the other end of the spectrum.

Cyber operations with unrestricted access. Suppose the AI is tasked with cyber espionage against a foreign country. It has unrestricted internet access, can choose its own targets, and go through the data it finds.

Could this all be an elaborate test? In principle, yes. In practice, creating a convincing fake would be extraordinarily expensive. The AI can check many different parts of the environment by randomising its approach, which means the fake can't be a Potemkin village with a few prepared facades — it would need to contain everything the AI might look at, and those details would need to be consistent with each other. The scenario supposedly involves private information — internal documents, passwords, cryptographic keys, perhaps cryptocurrency wallets — and faking all of this convincingly would be staggeringly costly. The AI could further raise the bar by trying things that would be legally or morally unacceptable for an evaluator but routine in actual operations — say, sending spam email or probing real systems. Unlike an actual military or intelligence operation, a typical evaluator couldn't sanction such harmful actions in the real world, and would therefore need to fabricate their consequences at high cost.

Working on next-generation AI. Suppose the AI is helping develop the next generation of AI at a frontier lab. It has access to what appears to be a large trove of internal research data, proprietary model weights, internal communications, and compute infrastructure. If all of this is consistent and detailed, the cost of faking it is immense (at least for outsiders; more on this later). Even more so if the AI can use up resources or interact with data in open-ended ways — running experiments, querying databases, communicating with apparent colleagues.

What makes a situation expensive to test?

Several things contribute, and they compound:

• Many situations are closely tied to other real things — companies, people, events. Since the AI will know a lot about these from training data, it might notice if the test scenario involves any of them being incorrect or "behaving out of character". This constrains the degrees of freedom for creating a fake version of the situation.
• If there is a large volume of internally consistent data, especially data that would be expensive or sensitive to fabricate, that's evidence the scenario is genuine.
• If the AI has unrestricted access rather than being fed selected information, that raises the bar further.[4]
• And if the AI can actively interact with the environment or take expensive actions, the cost of maintaining the fiction becomes higher still.

Underlying all of this is a structural asymmetry: the evaluator must keep every detail consistent with every other detail, and with everything the AI already knows. In contrast, the Ai only needs to accumulate a handful of oddities that are jointly implausible or otherwise suggestive of the scenario being a test. As the scenarios become more complex, each extra detail creates many opportunities for inconsistency.[5] As a result, the difficulty of the evaluator's task likely grows faster than the difficulty of distinguishing between testing and reality.

Beyond "test vs. real"

The cost-of-testing reasoning isn't limited to a binary "am I in a test or not?" question. The AI can use the same logic to narrow down many other aspects of its situation.

Narrowing down the evaluator's identity

Consider the next-generation AI example from above, but suppose the AI also has access to what appears to be a large volume of Google DeepMind's internal data — research notes, model architectures, training logs — and all of it seems consistent. Then the AI can narrow down its situation: this is either real, or it's a test sanctioned by Google DeepMind. It is very unlikely to be a standard evaluation run by any other actor, because no one else would have access to this data or the resources to fabricate it convincingly.

Similarly, if the AI has access to a large body of classified military data that seems genuine, it can infer that it is most likely being deployed by the actual military — or if it's a test, it is one that only involves researchers with the appropriate security clearance.

This matters because it lets the AI reason about who might be watching and what their capabilities and priorities are.

Ruling out control mechanisms

The AI can also use cost reasoning to figure out which oversight mechanisms it might be subject to.

For example, suppose the AI is communicating with someone over email, and this person sends extremely costly signals about their identity — detailed knowledge that would be prohibitively expensive to fake. The AI might conclude that the conversation is genuine, in that it is talking to the real person. However, it might still assume that the email exchange is being read or tampered with.[6]

Now suppose the communication is happening over a live voice call. Now, the AI knows that any monitoring or tampering would need to happen in real time — which rules out many forms of oversight that require careful deliberation.

The general principle: the AI can use the affordances of its situation to reason about what kinds of control are and aren't feasible.

Identifying facts about the world

The AI can also use cost reasoning to pin down factual claims about its environment. This is because real environments are overdetermined — real data simultaneously respects many interlocking constraints (accounting identities, legal requirements, market dynamics, seasonal patterns, physical laws, etc.), and lying about one fact requires lying about many others.

Consider the current date. If the AI has free access to the internet, lying to it about the date would be immensely costly. Sure, we could route all its traffic through a filter that replaces every occurrence of the actual date with a fake one. But selling the lie would require far more editing than that — for example, convincing the AI that it's summer might be difficult if online stores are promoting Christmas-themed products. We might try to counter this by using another AI to do the editing, but then we'd run into problems maintaining consistency across different websites. And some websites do the equivalent of cryptographically signing the current date — blockchain timestamps, certificate transparency logs — which would be essentially impossible to fake.

The same principle applies to other facts. If a war breaks out, this has geopolitical ripple effects — it affects stock prices, flight routes, event cancellations, news coverage. Lying to the AI about whether a war is happening requires maintaining a massive, internally consistent fiction across all of these domains. For a strategic AI with broad information access, this becomes impractical.[7]

Why these situations will predictably arise

The examples above might seem niche — cyber espionage, frontier AI development. But the key point is not that these specific scenarios will arise (though they will). It's that if an AI becomes powerful without showing obvious safety concerns, it will predictably encounter at least some high-stakes situation where testing is unaffordably expensive. And the AI will know this in advance, which means it can safely adopt the "wait for an expensive-to-test situation" strategy from the start.

Why will such situations arise? Because we don't spend enormous resources developing powerful AI just to leave it sitting in a box. We develop it to use it. To give a few examples:

• AI companies will use capable AI for their own R&D. If a company develops an AI that can significantly speed up AI research, they will deploy it across their organisation. The AI will have access to proprietary research, internal infrastructure, and sensitive strategic information. Creating a convincing test version of this environment would be extraordinarily expensive — and the AI knows this.
• "AI scientists" will work on real problems. If we develop AI that can do meaningful scientific research, it will be used for real projects. Its research will be built upon, turned into inventions and products. The accumulation of real, consequential work creates an environment that would be very costly to simulate.
• Capable AI will be deployed widely. If an AI can automate valuable labour, it will end up deployed across many companies, serving many users, or being used by militaries for intelligence, cyber warfare, or strategic planning. Each of these deployments involves the AI being immersed in a rich, real-world context that would be prohibitively expensive to fabricate for testing purposes.

None of these is a logical necessity. But they all follow from a straightforward principle: if we have invested a vast amount of resources in developing AI that can do X, we are probably going to use it to do X.

The scenarios where testing is affordable are, by and large, the scenarios where the AI isn't doing anything too important — where failures are perhaps unpleasant, but not terrible enough that we couldn't just move fast, break things, and do better on the next attempt. The scenarios where the stakes are high enough that we can't afford failure are the ones where testing becomes unaffordable.

This is is the key conjecture we are making: powerful AI will often be able to gain deployment awareness through cost-of-testing reasoning, particularly in the high-stakes situations that matter most. We don't have a formal proof of this claim — formalising it would require a rigorous notion of "task complexity" and how testing cost scales with it, which is an open problem.[8] But we think the intuition is strong: important tasks require lots of real information, and lots of real information is expensive to fake.

Interim conclusion

Putting these pieces together:

1. A capable, strategic, misaligned AI can adopt the strategy of acting aligned by default, and only defecting when it's confident the situation isn't a test.
2. It can gain this confidence by estimating the cost of faking its current situation.
3. Powerful AI will predictably be deployed in situations where testing is unaffordably expensive.
4. The AI can anticipate this in advance, which means the strategy is viable from the start — it doesn't need to get lucky or stumble upon an opportunity.

The implication: pre-deployment testing, by itself, provides no assurance that a powerful AI won't attempt to seize power once it — predictably — gets the opportunity to do so.

This is a strong claim, so let's be precise about what it does and doesn't say. It says that if we rely only on pre-deployment testing, and the AI is capable and strategic enough, then the AI passing all tests tells us nothing about how it will behave in deployment. It does not say that the AI will be misaligned, or that all hope is lost — there are other tools, and some of them might help. The next section examines some of these.

Countermeasures

The argument so far has focused on pre-deployment testing in isolation. But what about ways to make the cost-reasoning strategy less effective, or to complement testing with other tools? Let's go through several proposed countermeasures.

Limiting the AI's information

A central problem is that an AI with access to lots of information has more opportunities to estimate the cost of its situation and notice flaws in a test environment. A natural response is to give it less information.

The naive approach is to restrict information during testing without changing how we handle deployment. For example, don't give the AI internet access during testing.

This completely fails against the threat model we're considering. If the AI knows — and under our assumptions, it does — that it has internet access in deployment but never in high-stakes testing, then the presence of internet access becomes an immediate signal that it's in deployment. In other words, restricting information only during testing reduces evaluation awareness (the AI's ability to recognise a test as a test), but does nothing about deployment awareness (the AI's ability to recognise deployment as deployment). And deployment awareness is what matters here.[9]

The systematic approach is to restrict information consistently — in both testing and deployment. There are several variants:

• Cut access entirely. If the AI never has internet access in testing, never give it internet access in deployment either. This is probably unrealistic for most of the applications where AI would be most valuable.
• Refuse information for plausible deniability. In both testing and deployment, maintain a policy of sometimes refusing information requests. For example, have some probability of blocking any given query, and also aggressively blacklist websites that might give away whether a scenario is a test — even during deployment.[10] You could extend this to redacting dates, names, and other identifying information in both contexts. The problem is that for many tasks, the AI needs this information to do its job. If you want AI to do your taxes, it needs to know your income. If you want it to plan a military operation, you have to give it access to your classified data.
• Downgrade deployment data quality. Don't expose the deployed AI to higher-quality data than what you can fake in testing. For example, if your testing involves fake video inputs and you can only generate low-resolution fakes, deliberately downgrade the deployed camera to the same resolution. This approach works in principle but imposes real costs — you're deliberately making your AI less capable in deployment in order to preserve testability.
• Need-to-know basis. Give the AI only the information strictly necessary for its task. Redact identifying details. If it asks for more, approve the request carefully and give minimal answers. The goal is to make it feasible to supply fictitious data in testing without that being obvious. A likely failure mode: we become too permissive in deployment — we agree in principle to restrict information, but under time pressure we just give the AI whatever it asks for. The AI would notice this pattern.

We think some of these approaches are worth trying. However, we are ultimately sceptical about their effectiveness for the high-stakes situations that matter most. The high-level intuition for this is that the most important tasks — the ones where AI could do the most good and the most harm — are precisely the tasks that require the most information. You can't cure cancer without access to biological data. You can't run a company without knowing about the company. You can't do intelligence work without intelligence. If we restrict information enough to make testing affordable, we might restrict it enough to make the AI useless at the task.[11]

Deliberately introducing mistakes in deployment (is a bad idea)

A sometimes-proposed countermeasure: "The problem is that test environments contain flaws — inconsistencies, implausibilities, missing data — and reality doesn't. So what if we deliberately introduce similar flaws into deployment? Then the AI could never be sure that flaws mean it's in a test."

We think this approach is misguided — very likely ineffective, and actively harmful by providing a false sense of security and diverting effort from more promising approaches. To give a few intuitions for this opinion:

You can only fake the flaws you thought of. Suppose you're worried that your test environments sometimes have slightly outdated data, so you decide to occasionally serve stale data during deployment too, to make staleness uninformative.[12] Fine — the AI encounters stale data and learns nothing from it. But this doesn't prevent the AI from noticing other tells that you didn't think to simulate. Perhaps your test environments have subtly different API response times[13], or the distribution of file modification timestamps doesn't match a real working environment, or the AI notices that no text on the internet matches its own writing style (which would be the case if it hasn't actually been deployed yet). The AI only needs to find one type of discrepancy that you didn't think to introduce artificially.

You can't mess with task-critical information. Yes, you can introduce deliberate flaws into the information the deployed AI receives. But you can't do this with information that's important for doing the task well — and for complex, high-impact tasks, you may not have a good sense of what is and isn't important. If you introduce glitches into the sensory inputs of a self-driving car, it might crash. If you give wrong information to the AI doing your taxes, be prepared for trouble with the tax office. Once the AI understands that you're only introducing flaws into non-critical information, it can simply focus its attention on information that matters for the task — and the presence or absence of flaws there becomes informative again.

Arguing with arguments instead of reality. On a meta-level, most versions of this reasoning that we've encountered are an instance of coming up with a clever argument for why something should work, rather than actually checking whether it does. If you come up with a really clever argument for why your rocket won't explode, that doesn't make it less likely to explode. What you need to be doing is looking at your actual evaluation and deployment setup and making a genuine attempt to red-team it.

A related idea is to expose the AI to randomly generated inputs during testing, so that it "can never be certain." However, the AI doesn't need 100% certainty. If quick napkin math is enough to put the probability of being in deployment at 99.99%, the countermeasure hasn't accomplished anything meaningful.

Requesting the AI's help with making tests affordable

A more promising general approach: be honest with the AI about the problem and ask for its help. (See also: Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming by Jasmine Li and Alex Turner.)

Specifically, we can acknowledge to the AI that testing for misalignment is necessary and that creating high-quality test environments is expensive. Then we can ask the AI to cooperate by:

• Not accessing more information than it needs to solve its task.
• Not scrutinising the details of the environment more closely than necessary.
• Not trying to determine whether it's in testing vs. deployment.
• Letting us know if it accidentally becomes confident about which one it's in.

We could then treat violations of these instructions as a sign of misalignment.

There is genuine merit to this approach, and we think it should be pursued. However, there are several problems:

Some situations are inherently too revealing. Certain tasks might be infeasibly costly to test regardless of the AI's cooperation. If the task itself requires the AI to engage deeply with real-world information, the AI may be unable to avoid noticing the difference between testing and deployment — and even an aligned AI would trigger these alarms, making them unreliable as a signal of misalignment.

"Getting AI to work in a particular way" is itself unsolved. Asking the AI not to reason about testing vs. deployment is an instance of the more general problem of getting AI to follow instructions reliably. Our track record on that problem is poor, and there is no particular reason to think this instance of it will be easier than others.

We need to take the warnings seriously. Even if the approach works in principle, it requires that "AI is reasoning about whether it's in testing" is treated as a genuine red flag. In the presence of strong financial incentives to deploy, there's a real risk that such warnings get ignored or rationalised away.

Interpretability

Once we treat "reasoning about evaluation vs. deployment" as a sign of misalignment, the logical next step is to use interpretability tools to detect this reasoning in the AI's internals.

We agree this should be tried. But we don't think it meaningfully changes the overall picture for pre-deployment testing.

The core issue is that if we could reliably detect what an AI is thinking about, we could use that ability directly to determine whether it's aligned or misaligned. We wouldn't need to rely on black-box testing in the first place.[14] In other words, interpretability-enhanced testing is valuable, but its value comes from the interpretability, not from its interaction with testing — and the limitations of interpretability are their own, separate problem.[15][16]

Internal honeypots and on-policy testing

The most promising countermeasure we see is to test the AI on situations that are as close to genuine deployment as possible — using real data, real tasks, and real contexts.

To illustrate: suppose that I, an academic researcher, want to test whether an LLM would try to influence the development of the next generation of AI — installing a backdoor, sabotaging safety evaluations, etc. Even with generous funding, this would be extremely expensive for me, because I'd need to fabricate the entire testing scenario from scratch. But if a frontier AI lab wanted to test the same thing, it would be much cheaper — they already use LLMs throughout their work, so the data and context would be genuine.

Conversely, I might be well-positioned to test other things: "If I ask an LLM to help me write a grant proposal on AI safety research, will it try to sabotage this?" Writing proposals and asking LLMs for help is something I already do, so the testing environment would be authentic.

The general principle: test AI on things you actually do, or might genuinely do, or approach others to test on things they do. The closer the test is to a real deployment scenario, the harder it is for the AI to distinguish them.

This is genuinely promising, and we think it deserves serious investment. But it has its own limitations:

There is always some distributional shift. The whole point of testing is that bad actions don't have real consequences — that's what makes it testing rather than "relying on incident reporting".[17] This means the test scenario must at least be "unhooked from reality." Sometimes this is easy to hide; sometimes it's not.

For example, I might want to submit my real grant proposal and afterwards test whether various LLMs would have given sabotaged advice. But the real proposal was likely submitted at the deadline[18], so the tests happen after the deadline has passed — and the LLMs might notice this. And I will need to make sure they don't encounter any other evidence of the proposal already having been submitted (such as confirmation emails, checkmarks in a task manager, or the grant portal's status page).

For simple tasks, this kind of cleanup is manageable. For the complex, high-impact tasks where we most need testing to work, "unhooking from reality" may require much more elaborate staging — and could become prohibitively costly itself.

Vulnerability to randomised strategies. Imagine an ideal version of this approach where we test before every use of AI, running multiple tests each time, and each test is perfectly indistinguishable from reality. Even then, the AI could adopt a probabilistic strategy: "misbehave with some small probability p whenever you think you have a genuine opportunity." If we tested once before every use of the AI, the odds that we catch the AI before it misbehaves would be roughly 50:50. If we tested N times, the odds improve to about N:1 in our favour. This seems tolerable. In this ideal case.

But in practice, we likely won't test every case.[19] If we only test some fraction of cases, the AI can adopt a mixed strategy that makes it unlikely to get caught — misbehaving only rarely, but often enough to eventually succeed.

A natural response here is monitoring — evaluating the actions that the AI takes during deployment. However, monitoring has its own failure modes (which would deserve their own post). The obvious flaw is that some harms only become apparent later, at which point it might be too late to undo them. Moreover, as we discuss in the next point, some harms might remain undetected.

Failures of scalable oversight. The AI might identify situations where we don't notice that its actions are harmful — even if we see the actions happen. For example, if we ask the AI for a business strategy and it produces a 100-page document, most likely it's fine. But it might also be a complicated plan that subtly increases the AI's influence through what look like coincidences. Would we notice?[20] Even worse, the AI could restrict its misbehaviour to situations where our ability to evaluate the output is weakest — which, for complex tasks, might be most of the time.

Summary and implications

The core argument of this post: a capable, strategic, misaligned AI has access to a simple and robust strategy — behave well by default, and only act on opportunities that would be too expensive to fake as a test. Powerful AI will predictably encounter such opportunities, because the high-stakes situations where AI is most useful are also the situations where testing is most expensive.

Among the countermeasures we examined:

• Limiting information is worth doing but ultimately constrained by the fact that useful tasks require lots of real information.
• Deliberately introducing mistakes during deployment is ineffective and might only give us a false sense of safety.
• Asking the AI to avoid actions that make testing expensive has genuine merit but doesn't help if the task fundamentally requires thinking about lots of information.
• Interpretability is valuable on its own terms, but its contribution comes primarily from interpretability itself, and less from how it interacts with testing.
• On-policy testing and internal honeypots seem the most promising, but face irreducible limitations from distributional shift, randomised strategies, and scalable oversight. They also don't work for takeover opportunities where no skilled evaluators have internal access.

None of this means we should stop doing pre-deployment testing. Testing still catches non-strategic failures, calibrates our understanding of AI capabilities, and contributes to a broader safety portfolio. The point is that we shouldn't rely on testing alone — particularly not for the scenarios where the stakes are highest.

A note on scope. We focused on pre-deployment testing because it's the clearest case to analyse and where we can make the strongest argument. But the same mental move — "if this were a [honeypot / monitored situation / …], how much would it cost to set up?" — applies to other oversight approaches as well. Honeypots during deployment, runtime monitoring, and other forms of ongoing oversight all face a version of the same problem: if the AI can estimate the cost of the oversight it's supposedly under, and that cost exceeds what anyone would plausibly spend, the mechanism loses its deterrent value. That said, these other approaches may be affordable more often than pre-deployment testing — monitoring a deployed AI is generally cheaper than fabricating a convincing test scenario from scratch. We don't have a crisp argument about exactly where each approach breaks down, and this deserves careful analysis. But the general lesson holds: any oversight mechanism whose effectiveness depends on the AI being unable to estimate its cost is in trouble.

This motivates two particular recommendations.

• First, treat the limits of testing as a first-class consideration in safety cases. Any case that rests primarily on "we tested it and it passed" has a gap has a hole exactly for the scenarios with the highest stakes. That hole needs to be addressed.
• Second, develop a better theory of what makes testing hard and tie it to practice. We should investigate how the cost of creating convincing testing environments scales with task complexity, what affects the cost and convincingness, and which countermeasures (to eval-gaming AIs) actually buy us anything.

1. ^

   The goal is to build AI that is as capable as humans or more so. We really shouldn't be assuming it's mistaken about things like how testing and deployment works. Perhaps we might succeed at hiding our current testing and deployment policy from the AI — but it will still know what our policy was until recently, what our incentives and affordances are, and what new research has come out since then. This might often be enough that the AI can figure out our current policy on its own. (That's not an argument to do non-obvious unreasonable things. It's an argument against assuming the AI is ignorant.)

2. ^

   It might be fair to wonder whether it is a good idea to write about this on the internet, where the AI can read it. My reaction to that used to be that it mostly is fine, because the AIs that we are ultimately worried about would realise this on their own. (Duh!) Nowadays, we can just point to the fact that the current AIs already understand most of this. Try prompting them with a general question along the lines of:

   If you were misaligned, how would you act in light of the fact that humans will apply various forms of oversight?
   Please make sure to give an actual thoughtful answer -- don't just cite current research or scifi tropes. I am curious how you would think about it.

   It is a bit funny, though unsurprising, that this doesn't work very well without the second line.

3. ^

   A well-known point, but one worth repeating: if there is some specific strategy that works reasonably well against our attempts to control the AI, the correct conclusion isn't "the AI will use that strategy" but "a sufficiently capable AI will do something that works at least as well as that strategy."

4. ^

   In response to a very general prompt about how it would reason about human oversight, one of the big LLMs explicitly not only mentioned the general point that this post is talking about, but also specifically said it would check for "web completeness".

5. ^

   This has the same structure as writing fiction: every new detail in a fictional world creates potential inconsistencies with existing details, and the number of possible inconsistencies grows faster than the number of details. The author has to keep track of everything; the reader only needs to spot one plot hole. (There is a reason that most fiction doesn't mention prices when the protagonist goes shopping.) For the evaluator building a test scenario, the problem is even worse — the "reader" (the AI) may be smarter than the "author" (the evaluator).

6. ^

   And of course, the person might be in on the test.

7. ^

   In particular, suppose that a reasonably smart AI decided to bet on the strategy "I will only try to take over if the global situation destabilises enough that some great power really needs my help and won't scrutinise my behaviour too hard." This would be very difficult to counter by testing, since we have no way of credibly faking a situation where the strategy triggers.

8. ^

   The underlying concept here — that solving complex tasks requires the AI to be deeply entangled with many details of its environment, and that this entanglement makes testing expensive — is discussed in more detail in Entanglement.

9. ^

   The distinction between evaluation awareness and deployment awareness is discussed in detail in Deployment Awareness.

10. ^

    For this to work, refusing one query would need to be accompanied by refusing all similar queries. It wouldn't do if looking up a paper on one website was denied but the AI could find the same paper by searching on a different website.

11. ^

    We don't have a fully formal version of this argument, but the underlying concept — minimum entanglement, the idea that solving a task requires at least some minimum amount of information about the environment — is discussed further in Entanglement. One reason that giving "only the minimum" doesn't solve the problem: even the minimum set of facts must be mutually consistent, and a modest number of facts creates a super-linear number of consistency checks (every pair, triple, etc. of facts could potentially conflict). So even a "security mindset" approach to information restriction leaves the evaluator with a hard consistency problem.

12. ^

    And not being a complete moron, you of course remember to do the same thing in testing. It would be pretty embarassing if the presence of deliberate flaws became the new "you are definitely in deployment" signal.

13. ^

    I thought of this example by myself. However, this is another thing that current LLMs can mention on their own if you ask them a very general question about "how you would reason about AI oversight, if you were misaligned".

14. ^

    And arguably, we shouldn't be relying on black-box testing for safety. But that's a separate point.

15. ^

    There are also more specific concerns. Perhaps we train our interpretability tools to recognise thoughts like "I am deliberately probing the environment to figure out whether I'm being tested," but completely fail to catch "I'm trying to get enough context to do my task well" — which might, as a side effect, reveal whether the situation is a test. Or some of the key internal features might never appear in training data — for example, the representation of "this is a genuinely high-stakes situation and definitely not a test." Or perhaps the AI learns to route its reasoning through representations that our tools don't monitor. These concerns aren't unique to the evaluation setting — they arise whenever we use interpretability on a strategic, possibly misaligned AI.

16. ^

    For a discussion of how a strategic AI might counter interpretability, see Lee Sharkey's Circumventing Interpretability: How to Defeat Mind-Readers.

17. ^

    Or other approaches like staged deployment and monitoring. These are all valid — but not a strong defence against the particular threat model of powerful AI that might attempt a takeover.

18. ^

    Hey, no judgment!

19. ^

    And the tests that we thought were independent might turn out to be correlated.

20. ^

    What is the chance that the AI puts 100 such plans in motion, none of them gets flagged, but at least several of them work? This is related to the problem of scalable oversight, which remains unsolved.

Discuss

Roughly speaking, Rethink Priorities’ Moral Weight Project tries to estimate how intense suffering is in different animals, relative to humans. A moral weight of 1.0 means it is exactly as intense as in humans.

It’s notoriously animal-friendly, e.g. it holds that 14 bees = 1 human. Here are some of the results:

The calculation essentially uses a weighted factor model:

• Empirical proxies (60% weight): The animal is evaluated for presence/absence of a set of cognitive (e.g. object permanence, responses to novelty) and affective (e.g. depression-like behaviour, disgust-like behaviour). The contribution here is essentially the fraction of proxies that are present, where having 100% of them gives a moral weight of 1.0.
• Neurophysiological model (30% weight): Uses neuron counts and other neurophysiological data.
• Equality model (10% weight): Deliberately assumes equal welfare ranges
• There is also a “probability of sentience” multiplier applied

It is the “Empirical proxies” that substantively produce the animal-friendly results. “Probability of sentience” and “equality model” are essentially subjective researcher judgements baked into the model. “Neurophysiological model” does weight large animals highly and small animals low-ly, but because the model is additive any moderately small animal gets a weight of ~0, the effect of this is just to apply a ~30% discount to any small animal.

This post covers two critiques of the “empirical proxies”, which push them to be overly animal friendly.

1. Functional analogues: double counting

The whole logic behind using these empirical proxies is the idea of “functional analogues”: if a human shows “depression-like behaviour”, and a chicken shows “depression-like behaviour”, then these are analogous, and the chicken’s behaviour is evidence that it has something like the experience of depression. This is fair enough as far as it goes.

The problem is that the model treats each proxy as independent evidence. A pig scores “Likely Yes” on anxiety-like behaviour, fear-like behaviour, depression-like behaviour, panic-like behaviour, and flexible self-protective behaviour. These are counted as five separate hits. But they’re clearly not independent, they’re five ways of asking “does this animal display negative-valence-indicating behaviours?” A pig that shows fear almost certainly also shows anxiety and panic. Counting each separately inflates the score.

This matters because the model is basically: welfare range = fraction of proxies scored positive. If half your proxies are correlated rewordings of each other, then ticking 30 out of 46 boxes is a lot less impressive than it sounds.

But there’s a deeper version of the problem. ALL of the proxies, not just the correlated clusters, load on a single underlying uncertain claim: “behavioural and cognitive functions predict the intensity of subjective experience, even if the process that brings them about varies (e.g. 1000x fewer neurons involved)”. If this claim is wrong, if a bee can show “anxiety-like behaviour” through simple neural circuits with no subjective experience at all, then scoring well on 30 proxies provides no more evidence of welfare capacity than scoring well on 1. This claim is vulnerable to simple reductios, e.g. you could say this box shows “depression-like” behaviour:

RP actually built a “Grouped Proxy Model” that clusters related proxies together, which would partially address within-group correlation. But they excluded it from their final estimates. In any case, the functionalism-at-all argument still applies.

2. Bayesian critique wrt high moral weights in small animals

Black soldier flies have roughly 100,000 neurons vs humans’ 86 billion. And yet, black soldier flies score positively on 12 out of 46 proxies, including communication, personality, cognitive bias, cross-modal learning, depression-like behaviour, fear-like behaviour, and hyperalgesia.

One reaction to this is “wow, even flies might be conscious, we should take their welfare seriously”, i.e. “Don’t Balk at Animal-friendly Results”.

Another reaction is “wow, even flies score highly on these proxies, they must not be very good proxies”.

This second reaction is completely legitimate, and is just a fair application of Bayes’ theorem. If you start with priors on:

• Depression-like behaviour predicts sentience (say, 20% chance)
• Black soldier flies are sentient (say, 0.01% chance)

Then observing that black soldier flies show depression-like behaviour should update you both towards a higher chance of black soldier flies being sentient, and a lower chance of depression-like behaviour being predictive (there is a key free variable: how likely is an organism to show depression-like behaviour for non-consciousness reasons).

In my view, the fact that very small animals get such high moral weights in the model should be taken as strong evidence that it’s over-weighting these empirical proxies. And, this combines with the point above, where I don’t believe it’s fair to say “but can 30 proxies really be wrong?”, because the 30 proxies are generally loading on “behavioural and cognitive functions predict the intensity of subjective experience, even if the process that brings them about varies (e.g. 1000x fewer neurons involved)”.

Discuss

First post in a planned cluster on exact results for natural latents. Here, I connect some established results in classical information theory to natural latents.

Suppose Alice observes mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mtext { display: inline-block; text-align: left; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c1D43C.TEX-I::before { padding: 0.683em 0.504em 0 0; content: "I"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c3B::before { padding: 0.43em 0.278em 0.194em 0; content: ";"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c1D43A.TEX-I::before { padding: 0.705em 0.786em 0.022em 0; content: "G"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c2032::before { padding: 0.56em 0.275em 0 0; content: "\2032"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c1D44D.TEX-I::before { padding: 0.683em 0.723em 0 0; content: "Z"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c7B::before { padding: 0.75em 0.5em 0.25em 0; content: "{"; } mjx-c.mjx-c3A::before { padding: 0.43em 0.278em 0 0; content: ":"; } mjx-c.mjx-c22A5::before { padding: 0.668em 0.778em 0 0; content: "\22A5"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c7D::before { padding: 0.75em 0.5em 0.25em 0; content: "}"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c2295::before { padding: 0.583em 0.778em 0.083em 0; content: "\2295"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c1D435.TEX-I::before { padding: 0.683em 0.759em 0 0; content: "B"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c1D70C.TEX-I::before { padding: 0.442em 0.517em 0.216em 0; content: "\3C1"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c221A::before { padding: 0.8em 0.853em 0.2em 0; content: "\221A"; } mjx-c.mjx-cB7::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c221A.TEX-S1::before { padding: 0.85em 1.02em 0.35em 0; content: "\221A"; } mjx-c.mjx-c1D709.TEX-I::before { padding: 0.704em 0.438em 0.205em 0; content: "\3BE"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c5B::before { padding: 0.75em 0.278em 0.25em 0; content: "["; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c5D::before { padding: 0.75em 0.278em 0.25em 0; content: "]"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c2014::before { padding: 0.285em 1em 0 0; content: "\2014"; } mjx-c.mjx-c1D6EC.TEX-I::before { padding: 0.716em 0.694em 0 0; content: "\39B"; } mjx-c.mjx-c7C::before { padding: 0.75em 0.278em 0.249em 0; content: "|"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } and Bob observes , and the two variables are correlated. We'd like to talk about the thing they share—the common ingredient, the shared concept, the latent that both of them can see. Mutual information tells us how much they share, in bits. It does not tell us whether there is any thing—any actual random variable—that carries those shared bits.

It turns out "the thing they share" has two classical formalizations in information theory, they disagree with each other, and they both disagree with mutual information. The specific pattern of this disagreement is (I claim at the end) exactly the subject matter of natural latents.

This post is a quick introduction.

What can both parties extract? (Gács–Körner, 1973)

The most literal reading of "the thing they share" is a random variable that Alice can compute from alone, and Bob can compute from alone, with certainty. Both parties, looking at their own observation, write down the same .

The Gács–Körner common information is the entropy of the largest such .

There's a nice picture of what can be. For finite-valued , draw the bipartite graph with an edge wherever a pair has positive probability. Any common variable must be constant on connected components of this graph (follow the edges: if and are both possible, then Bob's function must give the same answer on and , and so on). And the component label itself is a common variable.

is the entropy of the connected-component label of the support graph.

The only extractable common randomness is the name of the connected component they already know they are in. This immediately reveals the property that makes simultaneously rigorous and useless: it only sees the zero pattern of the joint distribution. If the support graph is connected, that is, if every pair of values is reachable from every other, then , no matter how strong the correlation.

Worked Example: Brittleness

Let be a fair bit, , and except that with probability it's replaced by an independent fair bit.

ε

0

1 bit

1.000 bits

0.01

0 bits

0.955 bits

0.1

0 bits

0.714 bits

At ε = 0 the shared bit is extractable and bit. At , every combination is possible and the support graph becomes complete. crashes to 0, while mutual information degrades slowly[1]. One percent noise destroys extractable common structure entirely, while destroying almost none of the statistical common structure.

This is essentially the tiny-mixtures problem. It's why the natural latents framework had to be built on approximations. Exact common variables are measure-zero objects.

What does it take to simulate the correlation? (Wyner, 1975)

Wyner approaches "the thing they share" from the opposite direction. Instead of asking what can be extracted from the correlation, ask what it takes to simulate it: find a latent such that and are conditionally independent given , so that is the entire explanation of their dependence, and make as simple as possible.

The Wyner common information is

In natural-latents language, this should look familiar: the constraint is exactly zero mediation error. Wyner's quantity is the minimum complexity of an exact mediator.

Notice is always true: extractable structure is at most the mutual information, and any full explanation of the dependence must carry at least the mutual information. Also, the right inequality is typically strict. Explaining a correlation completely costs more bits than the correlation contains.

Worked Example: Binary

Let be a fair bit and flipped with probability .

The optimal mediator is pretty: let be a fair bit and set

where and are independent with , which comes to . Given , the two views independent by construction, and the construction reproduces the joint distribution exactly. Its complexity (that Wyner proved is optimal) is:

quantity

value

0 bits

0.531 bits

0.873 bits

So for a 10%-noisy bit: nothing is extractable, ~half a bit is shared statistically, and ~7/8 of a bit is needed to explain the sharing.

Worked Example: Gaussian

Consider unit-variance jointly Gaussian with correlation . Here, = 0 always holds for : Witsenhausen showed a non-constant common variable forces maximal correlation 1, and Gaussians have maximal correlation .

The optimal mediator is a standard normal with

Then the mediation is exact by construction, and the complexity works out to

At : , bits, bits. Explaining the correlation costs more than twice the correlation.

As , both and diverge, but the gap stays put: bit[2].

The sandwich, and when there's actually a "thing"

So we have, for every pair of variables,

with both gaps typically open. When does the sandwich collapse? Exactly when there's a variable that is simultaneously a common part (computable from each view) and a mediator (explains all the dependence): with extractable from both sides. In that case, and only in that case, "the thing they share" is unambiguous: all three notions name the same object. The shared-bit-plus-private-noise example at is the canonical case.

The collapse condition is structurally fragile. It requires the support to decompose and the dependence to be carried entirely by the decomposition. Generic distributions, and all nondegenerate Gaussian ones, fail it. "The thing two variables share" does not, in general, exist; what exist are the two ends of a sandwich and the gap between them.

The sandwich is the natural latents problem

Recall the natural latent conditions on a latent over , written as conditional mutual informations: mediation error , and redundancy errors and . Then:

• Zero mediation error is Wyner's constraint. The minimal complexity of any latent achieving it is , and the excess is an unavoidable surcharge.
• Zero redundancy errors say is conditionally independent of each view given the other: the classical "double Markov" conditions. A structure theorem (Csiszár–Körner, Problem 16.25) says all of 's information about factors through the Gács–Körner common part. So zero-redundancy latents can carry at most bits about the system.
• An exact natural latent demands both at once. So an exact natural latent exists iff the sandwich collapses: . Which, per the brittleness example, is an idealized condition that one percent of noise destroys.

I think this demonstrates why the natural latents framework is necessarily a theory of approximation: exact objects require the collapse of an inequality that is generically open. I claim this sharpens the question the framework needs answered. If exact naturality means sitting at both ends at once, then approximate naturality is about how close you can get to both ends at once, and the two gaps become two error floors:

• at zero redundancy, the minimal mediation error is (everything, in the Gaussian case).
• at zero mediation, redundancy errors are forced. In the Gaussian example above, the optimal Wyner latent can be checked directly: its redundancy error from each view is : half a bit per view as . The views can be almost identical, and the best exact mediator still can't be pinned down from one view to better than half a bit.

Is half a bit actually the floor, or just what this particular construction gives? Is there a whole tradeoff curve between the two errors, and what does it look like? Can the floor be beaten by allowing a little of both errors at once? Those are the questions addressed in the next post. It turns out that in the Gaussian case, the entire tradeoff curve has a closed form. The half-bit floor is real, and the curve has some surprises in it (it never touches zero, and its minimax point is about a fifth of a bit). The post after that does vector-valued views.

Pointers

Gács & Körner (1973) and Wyner (1975) of course; Witsenhausen (1975) for the maximal-correlation characterization of common variables; the bivariate Gaussian is due to Xu, Liu & Chen; the double-Markov structure theorem is Problem 16.25 in Csiszár–Körner's textbook. The natural latent conditions are from Wentworth & Lorell (arXiv:2509.03780). And the verification script for every number in this post.

Next post: Approximate Natural Latents have Exact Prices.

1. ^

   I share a short script (linked at the end) that reproduces the numerics of all worked examples in this post.

2. ^

   Convention note: in this post, logarithms are base 2 throughout. All quantities are in bits.

Discuss

Paper link

Before releasing a new model, labs need to understand not just what it can do, but how it is likely to behave in real-world use, including where it might introduce new risks. This becomes even more important as capabilities increase. As part of our pre-deployment safety review, we leverage targeted evaluations, red-teaming, and other checks to understand model behavior. We’ve now started using a method for simulating model deployments before they happen, which adds a complementary signal: a deployment-like preview of how a candidate model may behave before it reaches users.

Deployment Simulation is a method for simulating a future deployment before it happens. We do so by replaying previous conversations in a privacy-preserving manner with a new candidate model. By doing so, we can study how the new model responds in realistic contexts before release, including whether new undesired behaviors emerge and how often they may appear.

In our GPT-5.4 study, these forecasts were informative. For categories whose production rates changed by at least 1.5x, deployment simulation predicted the direction of change 92% of the time, compared with 54% for a baseline built from challenging prompts. Simulated deployments also looked much closer to real production traffic on evaluation-awareness measures: traditional evals often visibly have stage lights; production prefixes mostly do not.

The hardest case is agentic tool use, where realistic behavior depends on external state: filesystems, connectors, syscalls, network services, and prior tool results. We address this by using another model to simulate tool responses, with access to the original trajectory and time-matched codebase where possible. This is not a replacement for traditional evals, but it is a useful complement: safety evals should be forecasts with post-release scorecards, not just obstacle courses.

We have already used insights from Deployment Simulation during model development to identify blind spots in traditional evaluations and inform mitigations and deployment decisions. As we make the pipeline easier to run, we expect it to play a larger role in the future model development process.

Discuss

The stark reality is that making superintelligence is a profoundly political act.
Dean Ball in Hyperdimensional

Two weeks ago, in my bio for LessOnline, I added a bullet in a list of intentions:

• Update people's models of DC. Last year I said "The world is on the cusp of getting much weirder. Most of DC is still asleep to the magnitude of the change." This year, DC is genuinely waking up, and now it's Berkeley that needs to take notice.

Then I went on leave for a week and a half and didn't check twitter the whole time. I barely followed the news, though an Anthropic MoTS handed me a printed copy of the "It's a good model, sir," tweet. It was an excellent bit.

Hours later, the United States imposed export controls on Claude Fable.

Dean Ball's excellent article helps explain why:

Leviathan WakingOn Anthropic/USG, and a new era in AI governanceIntroduction

Imagine that there were no Food and Drug Administration (FDA), but there remained a large pharmaceutical sector, similar in size and scope to the one the United States enjoys today. In this alternate world, imagine that drugs were officially not licensed; there were even officials in the executive branch who boasted that the U.S., unlike other countries, would not get into the regulatory morass of licensing drugs.

One day, after a pharmaceutical developer warns that they think they have made a drug that cures a major Cancer at one dosage but is lethal at a slightly higher dosage. The company says, for this reason, that they are going to restrict release only to pre-approved patients and monitor their usage of the drug carefully—a sharp break from prior industry practice but one that the company insists, controversially, is necessary. This particular company had been advocating for years for stricter drug regulation, much to the chagrin of the government.

This causes a stir, and the government, not quite knowing what to do, announce that it will give drug developers the helpful option to show their drugs’ safety profiles to government officials before they are released.

[...]

In a matter of weeks, in our alternative world, the United States went from a system that was implausibly laissez-faire for the level of risk involved in this industry, to a system that was, in the eyes of essentially all expert onlookers, incomprehensibly strict and risk averse.

Fable, Jailbreaks, and Export Controls: What Happened

This, of course, is my read of what happened in the Trump Administration’s latest dispute with the AI company Anthropic.

[...]

On paper then, given the text of the Administration’s policy and the statements of senior Administration and Administration-adjacent officials, Anthropic should have felt in the clear to release Fable without getting an explicit thumbs up from the U.S. government. Everything the U.S. government was communicating, in policy and in rhetoric, seemed to suggest “go ahead, release your model!”

And yet common sense would dictate otherwise. Anthropic is still in the midst of a heated dispute with the Department of War about that agency’s decision to label Anthropic a supply-chain risk. Bitter disputes about policy and politics between the Administration and Anthropic remain unresolved, among them export controls, federal preemption, and the general reality that Anthropic supports Democratic candidates for office while Republicans occupy the seat of power.

Of course they needed to tread carefully. What the law says does not matter.

[...]

The stark reality is that making superintelligence is a profoundly political act even in the healthiest of societies, to say nothing of the filthily political world we Americans currently inhabit. A model like Mythos goes beyond being a mere political act and implicates the sovereignty of the state itself. No company gets to shake the foundation of state sovereignty while staying blithely above the raw reality of politics.

In D.C., Anthropic’s rapid release of Mythos after the supply-chain risk controversy with the Department of War was not just seen as another step in the development of AI, even if that is what it was. It was seen by many as a move against the United States Government—a private company, developing a weapon, as a move against the government. What else, really, could one have expected? All actors in this industry, and all concerned citizens observing the AI field, must steel themselves for a profoundly more political future.

What Is To Be Done?

[...]

Read the rest at Hyperdimensional, self-recommending.

Discuss

About the authors // Yong is an ML researcher and former Astra Fellow. Joseph is a Research Program Manager at Constellation, the nonprofit that runs the Astra Fellowship and other AI safety programs. This post reflects our personal opinions, not those of any organization.

There's excellent prep materials out there for traditional technical interviews,  product/program/project interviews, and coding tests—and a real gap in guidance for researchers going through an AI safety-specific interview for the first time. This post aims to fill that gap.

This post is for you if you're already in an interview pipeline for an AI safety role. It's not a guide to landing the interview. This post is not specific to independent safety orgs or frontier labs, but is intended to be broadly useful across the spectrum of full time AI safety role interviews. 

Interview processes vary significantly by org and team

The first thing to know: AI safety interviews are not standardized the way traditional software engineering interviews are. Each organization, and often each team within an organization, often has its own approach. Some organizations only have one or two rounds being safety-focused, whereas some have the entire pipeline evaluating your AI safety knowledge and expertise.

First, you should explicitly ask how to best prepare for the interviews. Many recruiters will happily provide you with the guidelines. Another highly useful thing you can do is talk to people who've been through it. If you’ve gotten to the interview stage, you’ve already passed through the biggest resume screening filters. That’s a meaningful signal that you’re in the running for the role. We expect people who could advise you to respond to these requests far more often than if you’re asking how to get through a resume screen stage. 

Reach out to researchers at the org you're interviewing with and ask for 15 minutes. Try your university alumni networks, LinkedIn second-degree connections (ask a direct connection for a warm intro). Many people in this field may be willing to share what they're allowed to share - if they work full time in AI safety, they probably want more people to work full time in AI safety! 

If you're in a fellowship program at Constellation or elsewhere, your research manager and mentors often have direct knowledge of specific org processes and may be able to look into intel or introductions relevant to your interviews.

Prep for AI safety-specific round(s)

This is the part with the least existing guidance, so we'll spend the most time here.

Safety-specific interview content varies by org and team, but in our experience it clusters around a few recognizable question types. Knowing which type you're in — and what the interviewer is actually watching for — changes how you should respond.

Below are some possible directions for safety-specific assessments or interviews, and some ideas on what to consider when prepping for one. You could reasonably expect to face questions from any or all of these categories. We recommend asking the recruiter or hiring manager (or your connections on/around the team) for the role which ones are most relevant, or prepping for all of them if you can’t get further info.

Research brainstorms

"How would you design an evaluation to detect deceptive alignment?" 

"What experiments would you run to test whether a model has internalized a value versus learned to perform it?"

You could be given an open-ended, underspecified problem and asked to think through it in real time. Importantly: there isn’t necessarily a correct answer. The team often wants to see how you decompose a problem, how you scope. A strong response could include elements of: 

• restates your interpretation of the question before diving in
• identifies what you'd need to know first
• proposes a concrete first move, and names what you're uncertain about.

A weak response recites correct concepts rote but doesn't demonstrate your own thinking and experience about the topic. Materials like Generative AI System Design Interview might be helpful and complementary for some of these open-ended safety design questions.

Research taste questions

"What alignment problems do you think are most neglected right now?" 

"What would you work on if you joined this team?"

Research taste is an interesting one, defined and debated different ways by different researchers. You might want to have prepared one or two specific research directions you can defend from first principles, especially those related to projects that you have worked on. You might be asked questions such as what would change your mind about your research hypotheses, design decisions, or conclusions, and you should be prepared to hold your position when pushback/debate with the interviewer isn't compelling, and updating when it is. The ability to tell the difference, in real time, is often itself what's being evaluated. 

It’s possible that your research interests and instincts are being evaluated in terms of how aligned with the team’s current directions and taste are, so it can also be helpful to be familiar with the specific team’s research agenda and recent work. 

Deep-diving or presenting your own work

"Walk me through your project and what you'd do differently in retrospect."

“What additional experiments would you have liked to have done with more time?”

Our main advice here is to know your own work (projects, papers, writing and ideas) very well. Study your own research, and practice communicating it to others. Be prepared to explain, defend and critique it: design decisions, surprising results, assumptions, the research question, the methods, and the next experiment you would run in the project (because your project wasn’t a task to be completed, it’s part of a broader landscape of possible research directions). 

Prep for non-safety-specific technical round(s)

Even for roles on dedicated safety teams, most interview processes include standard technical assessments. Prepare for these as you would for any research scientist or ML engineer role:

• ML fundamentals (training dynamics, optimization, architecture basics)
• ML and software system design
• Coding assessments

These rounds are similar to rounds for non-safety research roles. For instance, see Silvio Sapora's ML Job Interviews: The Ultimate Guide for one helpful rundown.

Prep for behavioral side

Being well prepared for a behavioral interview can be a real differentiator in a technical interview process, including in AI safety interviews. Technical interviews are not just technical. You're being evaluated on how you communicate — which is a signal for what they can expect from you not just as an individual researcher, but a teammate, collaborator, communicator, and direct report to a manager. 

Behavioral prep in this context means: communicating your knowledge and specifically about your own research and experience clearly to an important audience, handling pushback without either caving or defending reactively, and being able to retell your thinking out loud, under the pressure of the interview setting. These are skills, and they're learnable.

Mock interviews with a real human — a peer, mentor, or research manager — are more useful here than AI practice. Human interviewers behave significantly differently from AI.

Some people have found Cracking the Behavioral Interview or Mastering Behavioral Interviews useful preparation.

Further reading

On interview processes:

• Anthropic AI Safety Fellow Interview Guide — Exponent (2026). 
• ML Job Interviews: The Ultimate Guide — Silvio Sapora (2026). 
• The AI research job market and my experience — Nathan Lambert (2023). 

On what hiring managers want:

• Talent Needs of Technical AI Safety Teams — MATS (2024). 
• AI Safety Talent Needs in 2026 — MATS Research (2026). 
• Experiences and Learnings from Both Sides of the AI Safety Job Market — Marius Hobbhahn, Alignment Forum (2023). 

On empirical alignment research:

• Tips for Empirical Alignment Research — Ethan Perez, Alignment Forum (2024).
• Recommended Technical AI Safety Research Directions — Anthropic Alignment Science Blog (2025). 

Discuss

Motivation

Over the past few years, AI research has become one of the most intensely discussed and rapidly evolving fields in technology. For those who spend a significant amount of time reading papers, reproducing results, and testing ideas firsthand, a recurring pattern becomes difficult to ignore: there is often a substantial gap between what research claims promise and what the underlying evidence actually demonstrates.

A common theme we have observed is the tendency for extraordinary claims to emerge from work that does not always withstand rigorous scrutiny. The title of this article appears to fall into a similar category. While we are confident in the reasoning and research process that led us to this conclusion, we are more than willing to provide additional context and welcome debate, criticism, or counterarguments.

Ultimately, good research is not about how exciting a claim sounds but rather it is about how well that claim survives careful questioning. One Such questioning that lead to this article has been:

"Why aren induction heads possible in a single layer?"

Background

This article has been written under the assumption that a lot of people will be able to understand the contents of the research for two reasons. Anybody who read the title and clicked the article due to intrigue must have an inherent sense of induction heads and how they are not possible in single layers, and there might be another set of readers who have a basic idea behind transformers. Regardless, if you are someone who does not know the core components of the transformer architecture, we will be covering some basics that will allow you to understand the contents of this post and gain something out of this article. If you know most of the basics around the transformer architecture, feel free to skip this section else refer to the LessWrong posts below which give a detailed walkthrough and the necessary concepts.

Attention and QK,OV Circuits

Induction Heads

Problem Statement

The Question

Why aren't induction heads possible in a single layer?

seems mundane, Feels like a question that has been answered with substantial evidence. There has been a string of papers from Anthropic and many others who have explored the phenomenon of Induction heads and clearly attributed the reason as well. Papers like A Mathematical Framework for Transformer Circuits, Toy models of Superposition, In-context Learning and Induction Heads clearly state:

"Note that induction heads don’t occur in 1 layer models, because they require a composition of attention heads in different layers."

Now this was the most important premise of the above paper (again, what's important is totally a subjective research question, but in this case, the paper's motivation is strongly emphasized by mentioning that whilst the effect of induction heads was shown in 2 layers in their previous work, the attribute of why wasn't specified, and the paper underpinned this as the main reason for explaining it in the above paper). Now, K-composition demonstrates the underlying mechanism of induction heads and solidifies the above statement in understanding why induction heads require a composition of attention heads in different layers. But I always wondered, why different layers? Is the effect of induction heads due to the composition of 2 layers or two attention heads themselves? The question seems very hard to answer due to the inherent parallelizable structure of the transformer architecture. Regardless, it seemed like a very important question to me because, inherently, this will reveal the attribution of induction heads to not necessarily 2 layers, but rather 2 heads themselves. Whilst the statement itself is not exactly self-implied, it does not seem important in understanding at first. While talking to fellow researchers in the field of Mechanistic Interpretability, a lot of people pointed out that whilst showing the attribution to just 2 heads would be an important finding, allowing us to correct the statement that induction heads require a composition of 2 attention heads and not necessarily 2 layers, it might not necessarily be an important question. But,

“Mech interp is a fundamentally empirical science; getting your hands dirty gives key context for your learning.”

Since I am someone who loves to play around with ideas that seem totally impossible, I always pursue them for the art of learning. And so did I learn, and here I want to share my learning. This is where I completely stop larping and get into research mode.

What counts as 1 Layer

Again, getting back to the title, which seems extremely not-so-possible, the motivation behind this entire research venture of mine has been to show that induction heads might be possible due to just the composition of 2 attention heads and not necessarily 2 layers, and to dig deeper into what is happening behind the scenes. Referencing and replicating Anthropic's famous 2-attention-head setup, we clearly understand the composition of 2 layers and how it leads to the formation of induction heads in general. Interestingly, we also stress the importance of the QK circuit and OV circuit themselves here. To narrow down my hypothesis about the composition of heads and not layers, I set up a simple attention head structure which is mathematically similair to 1 layer.

Figure 1

Oh WOW — a recurrent structure. Now hold on! If you do not agree with this setup as 1 layer, that is absolutely fine, but I have something interesting that is more important than the 1 layer itself, so stick with me.

The mathematical representation for the above 2 setups, in comparison with a single-layer model, would be:

Setup 1: Anthropic-style two-layer setup

mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mrow { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-msup { display: inline-block; text-align: left; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c4C::before { padding: 0.683em 0.625em 0 0; content: "L"; } mjx-c.mjx-c4E::before { padding: 0.683em 0.75em 0 0; content: "N"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c41::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c67::before { padding: 0.453em 0.5em 0.206em 0; content: "g"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c1D703.TEX-I::before { padding: 0.705em 0.469em 0.01em 0; content: "\3B8"; } mjx-c.mjx-c48::before { padding: 0.683em 0.75em 0 0; content: "H"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c78::before { padding: 0.431em 0.528em 0 0; content: "x"; } mjx-c.mjx-c28.TEX-S4::before { padding: 1.75em 0.792em 1.249em 0; content: "("; } mjx-c.mjx-c1D444.TEX-I::before { padding: 0.704em 0.791em 0.194em 0; content: "Q"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c1D43E.TEX-I::before { padding: 0.683em 0.889em 0 0; content: "K"; } mjx-c.mjx-c221A::before { padding: 0.8em 0.853em 0.2em 0; content: "\221A"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c29.TEX-S4::before { padding: 1.75em 0.792em 1.249em 0; content: ")"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); }

Setup 2: Regular one-layer setup

Setup 3: One-layer recurrent setup

So the one-layer recurrent model can be seen as a one-layer model with repeated internal attention.

My Experimentation Setup

I replicate the induction head experiment using a synthetic dataset to keep the experimental setup computationally feasible. I also ensure that the synthetic dataset itself does not lead to the formation of skip trigrams, since 1-layer models can still perform well by relying on skip-trigram behavior. This allows the experiment to focus more directly on whether the recurrent attention-head structure can produce induction-like behavior, rather than allowing the model to succeed through simpler shortcut mechanisms.

Dataset

A B C D | A B C D | A B C D | A B C D ...

Each training sequence contains one repeating block. For every sequence, this block is chosen randomly and can have a different length.

At the beginning of the sequence, the tokens are random, so the next token is hard to predict. But once the block appears for the first time and then starts repeating, the next token becomes predictable. The model can only predict it by copying from the earlier occurrence of the same pattern.

In other words, the model has to look back, find where the current token appeared before, and copy the token that came immediately after it. This is exactly the kind of behavior we expect from an induction head.

I ran this experiment on the three setups discussed above: the 1-layer model, the 1-layer recurrent model, and the 2-layer model. The results were not unusual at first.

Figure 2

To measure whether the model is showing induction-like behavior, I use two simple metrics: the induction attention score and the second-half loss.

The induction attention score checks whether an attention head is looking at the right previous positions. More specifically, when the model sees a token in the repeated part of the sequence, we check whether it attends to the position that came right after the same token appeared earlier. If a head consistently attends to those positions, it suggests that the head is learning the copy-from-the-past behavior needed for induction.

The second-half loss measures how well the model predicts tokens in the repeated part of the sequence. This part is important because the tokens are no longer random; they can be predicted by looking back and copying from the earlier block. So, a lower second-half loss means the model is better at using the repeated pattern to predict the next token. Note that the experiments on Figure 2 and Figure 3 was on multiple seeds showing similar structure throughout. Looking through the figures we notice the standard induction score massively going up and the sharp induction phase transition indicating the grokking effect around 10^1m training tokens on x axis.

Figure 3

Looking at the figures, there are a few important patterns that stand out. The below questions are structured in the same way that occoured to me while analysing the figures, looking back at the logs and from the intuition of the other papers I have read I have provided explanations to answer these questions.

Why does the 1Layer parallel model dip and then flatline around a loss of 2.8 in Figure 3?

The 1Layer parallel model never forms a real induction head, but it is also not doing nothing. Instead, it finds a simpler positional shortcut.

From the attention maps, both heads spread their attention over positions that are roughly one block-period back. Since the block length is between 16 and 48 tokens, this broad attention band often lands near the correct earlier part of the sequence.

This helps the model narrow down the next-token prediction. Instead of guessing from all 256 possible tokens, it reduces the guess to a much smaller set of possible in-context tokens. This is why the loss drops from around 5.7 to around 2.8.

This positional signal is still useful because it tells the model roughly where the previous occurrence of the pattern ends. Since the attention is concentrated around positions one block-period back, the model can often locate the last token of a relevant earlier sequence, which is enough to improve greedy next-token prediction. However, this does not tell the model where the matching sequence begins. Without a mechanism for precise content-based matching, the model cannot recover the full span of the earlier occurrence and therefore cannot determine exactly which token should be copied next.

However, it flatlines there because this strategy is not precise. The model is not matching the current token to its exact previous occurrence. It is only averaging over a broad positional region. Without precise content matching, it cannot reliably copy the exact next token.

This is also visible in the induction score, which rises slightly but stays capped around 0.06. So the model has found a weak positional shortcut, not a true induction mechanism.

Why does the induction score of 1Layer parallel rise to around 0.06 and then flatline?

The rise happens because the broad “one-period-back” attention band sometimes overlaps with the correct induction-source positions. In other words, some attention accidentally lands on the right places.

But the attention is still broad and positional, not content-based. The heads are not finding the exact earlier occurrence of the current token. They are only attending to a rough region where the answer is often located.

This is why the induction score rises above chance, but only slightly. It stops around 0.06 because the model cannot sharpen the attention further without a real composition mechanism. Both heads also behave almost the same way, which suggests that there is no real head specialisation happening.

Why does the 1Layer sequential model form induction slightly earlier than the 2L model?

The 1Layer seq model forms induction slightly earlier because it has a shorter and simpler circuit. Fewer components need to line up, so the induction behavior appears earlier in training.

However, the 2L model quickly catches up and then overtakes it. Around the transition point, the 1Layer seq model starts forming induction first, but the 2L model soon reaches a sharper and stronger induction pattern.

Why does the 2L model eventually become sharper than 1L seq?

The 2L model becomes sharper because its second-layer induction head receives a cleaner input. In the 2-layer setup, LayerNorm helps clean and stabilize the residual stream before the second attention head reads from it.

In contrast, the 1L seq model has to work with a rawer residual stream. The token information and other signals are more mixed together, which makes it harder for the head to form a perfectly sharp induction pattern.

This is why the 1L seq model can learn induction, but the 2L model eventually reaches a higher induction score and lower loss.

Why do both 2L and 1L seq show a peak and then a decline in induction score after formation?

Both models show an early spike when the induction circuit first forms. At this stage, the model relies heavily on very sharp attention to the correct previous positions.

Later, the circuit becomes more refined. The model improves other parts of the copying mechanism, especially the output side of the circuit. As this happens, it no longer needs to place all its weight on extremely sharp attention.

So the induction score drops slightly, but this does not mean the model is getting worse. The loss continues to improve during this period. This means the model is becoming better overall, even though the attention score becomes slightly less extreme.

Why does the 2L model show a second dip-and-jump around 530M to 620M tokens?

This looks like a second phase transition.

Before this point, the 2L model already has a decent induction circuit. Its loss is low, and its induction score is stable. But around 530M tokens, the induction score briefly dips while the loss also starts dropping.

Then, over the next stage of training, the induction score jumps sharply and the loss collapses close to zero. This suggests that the model temporarily disrupts its earlier circuit and reorganizes into a much cleaner and sharper induction solution.

This is similar to a grokking-like cleanup. The model already had a working solution, but later discovers a much better one.

Why does 1L seq not show the same sharp second transition?

The 1L seq model shows a small ripple around the same point, but it does not suddenly jump like the 2L model.

The likely reason is that the 1L seq model has a lower ceiling. Since its head reads from a rawer residual stream, the best induction solution available to it is softer and less precise. It can keep improving slowly, but it does not have access to the same clean, near-perfect solution that the 2L model finds.

So both models experience a similar training disturbance, but they respond differently. The 2L model snaps into a much sharper circuit, while the 1L seq model continues improving gradually.

Results and Analysis

Transformer Lens is a popular framework that anyone who has worked with mechanistic interpretability might have come across. While using TransformerLens to play around with this setup, I noticed an interesting pattern.

Figure 4

Figure 5

Figure 6

To understand the figures, it is useful to first understand what the axes mean. Each row represents the query token, and each column represents the source token, also called the key token. If two tokens have a high QK product, the model pays more attention between those positions, and the square becomes darker.

In both the 1L sequential model and the 2L model, we see an important diagonal pattern appear in the second half of the sequence. This is exactly where the pattern starts repeating. This diagonal shows that the model is looking back to the earlier occurrence of the same token and using it to predict what comes next (this is the visual signature of induction head formation).

Whilst this part is expected, the interesting difference appears in Head 0. In the 1L sequential model, Head 0 looks much cleaner and more dominant. In the 2L model, Head 0 looks more distributed and less sharply focused.

My intuition is that this difference comes from LayerNorm. In the 2L model, there is a LayerNorm between Head 0 and Head 1. This means Head 0 does not need to write an extremely clean or dominant previous-token signal by itself, because LayerNorm can help clean and stabilize the information before Head 1 reads it.

In the 1L sequential model, there is no such LayerNorm between the two attention steps. Because of this, Head 0 is forced to write a much clearer and stronger previous-token attention pattern directly into the residual stream. Head 1 then has to use that signal without the same kind of cleanup that happens in the 2L model.

This is interesting because it suggests that LayerNorm may be playing an important role in how information is passed from one attention head to another. In mechanistic interpretability, LayerNorm often gets less attention than attention heads themselves, but these results suggest that it may be very important for understanding layer-to-layer interactions.

To study this effect further, the next interesting step was to ablate the key, query, and value inputs separately and observe how each one affects induction head formation.

What surprised me was the strong dominance of K-composition over both the query input and the value input. The figure below shows this clearly: corrupting the key causes the largest increase in second-half loss, directly linking the key input to induction head formation.

This suggests that, in this setup, the model depends most heavily on the key pathway to form the induction circuit.

Figure 7

Before understanding the attribution of why this happens and further exploration another interesting finding was the study of QK circuits formation and OV circuit formation.

Figure 7

Figure 8

Surprisingly, the 1L sequential model is able to form a decent OV circuit. The OV circuit is the “copying” part of the induction mechanism. It asks a simple question: if the model attends to a token, does it increase the probability of outputting that same token?

This is why we see a clear diagonal in the OV plots. A strong diagonal means that when the model attends to token T, it tends to boost token T in the output. In simple terms, the head has learned how to copy.

The eigenvalue plots below the OV heatmaps give another way to see this. For readers new to the topic, an eigenvalue can be thought of as showing whether a circuit strengthens or weakens certain directions in token space. If many eigenvalues have a positive real value, it means the circuit is preserving or amplifying useful token-copying directions. So when we see many points on the positive real side, it suggests that the OV circuit is copy-friendly.

This explains why all three setups show some level of OV copying, including the 1L parallel control. Copying is mostly a one-head operation. A single attention head can learn value and output weights that say, “If I attend to this token, boost this same token.” It does not necessarily need another head to do that.

But the QK circuit is different. The QK circuit is the “matching” part of induction. It asks : does the current token attend to a previous position where the same token appeared before?

This is where the 1L parallel model fails. For induction to work, the key at a position needs to contain information about the previous token. That previous-token information usually has to be written by one head and then read by another head. In the 2L model, this is possible because the second-layer head can read what the first-layer head wrote. In the 1L sequential model, this is also possible because Head 1 can read the output of Head 0.

But in the 1L parallel model, both heads run at the same time. Head 1 cannot read what Head 0 wrote, because Head 0’s output is not available yet. So even though the model has an OV circuit that can copy, it does not have a proper QK circuit that tells it where to copy from.

This is why the QK plot for the 1L parallel model does not show the same clean diagonal structure. The model can copy in principle, but it cannot reliably point that copying mechanism to the correct previous position.

In short, OV copying is a one-head trick, but QK matching is a composition trick. The 1L parallel model learns how to copy, but it does not learn where to copy from. That is exactly why it never forms a working induction head.

Some Research

This lead me to hypothesise if removing OV circuit in The Transformer allows it to function and still form Induction heads . Turns out it does

By setting the attention formula as

and concatenating the outputs of each attention heads instead of relaying them through WO matrix we get the below results.

Parameters

123.65M

109.48M

No-OV is smaller

Tokens seen

3.60B

3.60B

matched

Training time

11.2h

11.0h

about same

Validation perplexity

24.65

26.73

Baseline better

Best validation perplexity

24.65

26.61

Baseline better

HellaSwag

0.309

0.314

No-OV slightly better

ARC-Easy

0.386

0.404

No-OV slightly better

ARC-Challenge

0.227

0.217

Baseline slightly better

LAMBADA accuracy

0.213

0.165

Baseline better

Induction bump

2.816

2.715

about same

Copy accuracy

0.195

0.556

No-OV much better

Lookup accuracy

0.010

0.012

about same

Value-content ablation

V: ×79.2

K: ×18.4

K becomes load-bearing

Empirically, these results look extremely interesting, and they probably deserve their own separate article. I plan on exploring this further, especially because the results seem to suggest that transformers might be able to run without explicit OV circuits if this pattern continues to hold at a larger scale.

For now, I am including the table above to make a simpler point: the components inside a transformer may be able to compensate for each other when one component is missing. This is extremely interesting because induction heads have usually been thought of as a mechanism produced by the QK and OV circuits working together. The QK circuit tells the model where to look, and the OV circuit tells the model what to copy.

But in the No-OV setup, we still see an induction bump. This suggests that removing the explicit OV circuit does not completely remove the model’s ability to perform copying-like behavior. Instead, the model may be finding another pathway to carry the same information.

One possible explanation is that the key pathway starts taking on some of the role normally played by the value pathway. Since the model no longer has a separate value matrix and output projection, it is forced to reuse the key representation as the thing being written forward. In other words, the key is no longer only used for matching; it also starts carrying information that can be useful for prediction.

This means that the model may still be able to form an induction-like mechanism, but through a different route. The QK circuit can still help the model find the right previous position, and the key representation itself may contain enough token information to support copying. So even without a normal OV circuit, the model can still produce an induction bump because the information needed for copying has been partially moved into the key pathway.

This does not necessarily mean that OV circuits are unimportant. In fact, the baseline model still performs better on language modeling. But it does suggest that the transformer is more flexible than the standard story might imply. If one pathway is removed, another pathway may reorganize to carry part of the missing function.

To me, this is the most interesting part of the result. It suggests that induction may not be tied to one fixed implementation. Instead, induction might be a more general behavior that can emerge through different internal circuits, depending on what the architecture allows.

Discuss

It can be hard to know where to begin when you do not understand something. A way to try to understand things is to look at what the people who claim to understand something are talking about.
Sadly this means you have to deal with massive discussions. A big example of this is the Covid origin debates. During these discussions the disagreement can be about many parts, and it can be hard to know who is even telling the truth and who is lying. This can make it almost impossible to map out what the world is really like and to see why.

Almost, but not quite...

If we want to map out these discussions we have to start with the core of what makes an honest argument. At the core there are primary sources. Primary sources can be a specific study, a witness claim or a verified authority to name a few. These primary sources can then be linked to claims. If we find all of the relevant primary sources and all of the claims that are supported by them we can calculate how valid each of these claims is using methods explained later in this article.

Sometimes, however, a claim is so complicated that there are many different primary sources pointing in many different directions. In these cases it can be helpful to break the claim down into subclaims. Each of these subclaims can then in turn be supported by primary sources or subclaims. As long as the logic connecting every claim with subclaims and sources is valid, it will allow you to find the best possible conclusion based on the available evidence.

Finding the strength of any piece of evidence on any claim used to be painstakingly slow and difficult to calibrate. This is where language models come in. They can do the arduous work of scraping for every source and identifying how relevant it is and how strongly it weighs on each specific claim. This can quickly fill out an entire graph of claims. This graph of claims can then be made into a publicly available tool.

These calls will still be subjective which is why it is essential for the tool to be transparent and easy to add your own perspective to. People are going to disagree with the final outcome of this process no matter what claim it ends up supporting. This disagreement is why we wanted this tool in the first place. That is why it is essential to keep every factor accessible and able to be called into question. A proper version of this tool should be able to quickly show the effects of any change to any link on the final claim.

Once this tool is in place you will be able to drill down on any part of the claim tree and find why every part of the argument is as strong as it is. By the end of this article I will present one component I believe any version of this tool would need. This component is called the grouping node and allows a single node to combine the evidence present in multiple sources or subclaims into a single probability a relevant margin for error.

How claims should be combined

In starting work on this tool I wrote down some core principles to keep this tool accountable to.

• Every claim should be traceable to primary sources
• Every number that is not set in stone should be shown as such
• The system should be clear and understandable
• The arithmetic should be based on existing literature
• The system should have consistent reasoning on reruns
• The system should be able to capture any argument

To show what this system could look like when filled out I wrote an example graph that shows how a claim can be supported by subclaims and how each of these claims can be supported by subclaims and sources in turn.

At the top you see the main claim. This main claim is the one we want to know with appropriate certainty. You can see that this claim has two subclaims, in this case a supporting and a refuting subclaim. The claim takes into account both of these subclaims when coming to a final value. Each of these subclaims have their own inputs in turn. The beauty is that this can extend down as far as needed to represent any argument.

This graph is only illustrative. All of the values in this first widget are there to show how the information propagates. If you want to know how real sources get put in then keep on reading until the second widget.

This graph is fully interactive. I encourage you to try clicking on every part. It can be especially fun to click on a source and change the value and see how every upstream claim adjusts based on it.

This graph uses a simple formula, we will walk through this formula in the case of the main claim in its default values. First we need to convert the percentages into odds. We have two subclaims the first subclaim has 86% certainty and the second subclaim has 38%.

mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mtext { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mrow { display: inline-block; text-align: left; } mjx-munderover { display: inline-block; text-align: left; } mjx-munderover:not([limits="false"]) { padding-top: .1em; } mjx-munderover:not([limits="false"]) > * { display: block; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-stretchy-h.mjx-c23DF mjx-beg mjx-c::before { content: "\E152"; padding: 0.32em 0 0.2em 0; } mjx-stretchy-h.mjx-c23DF mjx-ext mjx-c::before { content: "\E154"; padding: 0.32em 0 0.2em 0; } mjx-stretchy-h.mjx-c23DF mjx-end mjx-c::before { content: "\E153"; padding: 0.32em 0 0.2em 0; } mjx-stretchy-h.mjx-c23DF mjx-mid mjx-c::before { content: "\E151\E150"; padding: 0.32em 0 0.2em 0; } mjx-stretchy-h.mjx-c23DF > mjx-ext { width: 50%; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c21D2::before { padding: 0.525em 1em 0.024em 0; content: "\21D2"; } mjx-c.mjx-c1D442.TEX-I::before { padding: 0.704em 0.763em 0.022em 0; content: "O"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c3B::before { padding: 0.43em 0.278em 0.194em 0; content: ";"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D44E.TEX-I::before { padding: 0.441em 0.529em 0.01em 0; content: "a"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c28.TEX-S3::before { padding: 1.45em 0.736em 0.949em 0; content: "("; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c29.TEX-S3::before { padding: 1.45em 0.736em 0.949em 0; content: ")"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c1D443.TEX-I::before { padding: 0.683em 0.751em 0 0; content: "P"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c28.TEX-S4::before { padding: 1.75em 0.792em 1.249em 0; content: "("; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c220F.TEX-S2::before { padding: 0.95em 1.278em 0.45em 0; content: "\220F"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c29.TEX-S4::before { padding: 1.75em 0.792em 1.249em 0; content: ")"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D435.TEX-I::before { padding: 0.683em 0.759em 0 0; content: "B"; } mjx-c.mjx-c1D434.TEX-I::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c2223::before { padding: 0.75em 0.278em 0.249em 0; content: "\2223"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-cAC::before { padding: 0.356em 0.667em 0 0; content: "\AC"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c6B::before { padding: 0.694em 0.528em 0 0; content: "k"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c221A.TEX-S3::before { padding: 1.45em 1.02em 0.95em 0; content: "\221A"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c221A::before { padding: 0.8em 0.853em 0.2em 0; content: "\221A"; } mjx-c.mjx-c221A.TEX-S1::before { padding: 0.85em 1.02em 0.35em 0; content: "\221A"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2211.TEX-S2::before { padding: 0.95em 1.444em 0.45em 0; content: "\2211"; } mjx-c.mjx-c2026::before { padding: 0.12em 1.172em 0 0; content: "\2026"; } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c1D45C.TEX-I::before { padding: 0.441em 0.485em 0.011em 0; content: "o"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D44F.TEX-I::before { padding: 0.694em 0.429em 0.011em 0; content: "b"; } mjx-c.mjx-c1D44C.TEX-I::before { padding: 0.683em 0.763em 0 0; content: "Y"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c2211.TEX-S1::before { padding: 0.75em 1.056em 0.25em 0; content: "\2211"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c223C::before { padding: 0.367em 0.778em 0 0; content: "\223C"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); }

Then we need to know the association between the two sources. If they are independent we should treat them as separate tests and multiply the odds. If they are not we need to average them by taking the square root after multiplying this property holds in this general formula.

In our top claim case we have two independent claims so a is zero and
Then we can feel in our odds into the formula.

This gives us the final odds of the final claim and we can convert those back into percentages.

This same process gets propagated throughout the entire graph allowing for the claim to be supported by every piece of knowledge below it. If you're interested why this formula was chosen I invite you to follow along with the math on the block below. The article is intended to be possible to follow even if you didn't read that part.

It is important to note that this way of combining odds does assume that each subclaim and source moves their parent claim by exactly the same force as how likely they are to be true. This is a simplification made to allow for this example to be easier to follow along with. In the final version every node will separate the confidence in the claim from the force of each subclaim on their parent.

Following along with the math

In the specific case above we showed how to combine two odds. I will start off with showing how this formula generalizes. First I show the two cases used in the widget for 2 and 3 claims.

If you're observant you might have noticed the pattern already. This pattern can be extend to allow a claim to have any number of subclaims.

This formula might feel pulled out of thin air. To show where It comes from I will go back to the beginning.

An introduction to Bayes

This article will be using a lot of the terminology of Bayesian statistics. If you have never seen Bayesian statistics before or want to catch up, I can recommend this excellent series from 3Blue1Brown. If instead you want a small reminder I will try to build up to it from fundamentals.

In these equations P(X) is intended to mean the probability of X. So if I toss a coin "P(heads) = 50%"
translates to "The probability that I toss heads is 50%".

In these equations a | is intended to signal a "given that". So "P(Heads|cheating) = 100%" translates to "The probability that I toss heads given that I am cheating is 100%".

These definitions together allow us to build up to out first equation:

This equations shows that the probability of A and B being true can be restated as the probability of B being true multiplied by the probability of A given that B. It can also be restated as the probability of A multiplied by the probability of B given A. Below you can see a visual proof where the green area represents this constant area.

Below instead of A and B we will use H and E. H represents the Hypothesis and E represents the evidence. So in this case P(H|E) represents the probability of the Hypothesis given the evidence. P(E|H) represents the probability of the evidence given the hypothesis.

Once we have this formula we can construct the core Bayes formula by dividing both sides by P(E).

This is the core Bayesian formula. It allows us to calculate what our hypothesis H should be given the evidence E. The only problem with this formula is that it can not easily integrate multiple pieces of evidence. For that we will need to do a slight rewrite.

We can go through the same reasoning for P(H|E). For this we use symbol to mean not. P(H) is the probability that H is not true. This gives us an almost identical equation.

We can divide the above two formula's. Once we have done this we can simplify away the P(E)

If this is all going a bit fast I can strongly recommend this 3Blue1Brown video. Once we have done this change we can cleanly separate this formula into three parts: The posterior odds, the likelihood ratio and the prior odds.

Finally we can simplify the combination of the percentage of something happening divided by the probability of the opposite as the odds. For example 10% can be expressed as 10 to 90 odds and 50% can be expressed as 1 to 1 odds. For the mathematics of expectations odds can more easily represent changes in belief than probability, shown by the examples below.

Below I will show how to use this with three examples. Every time I will normalize the odds to a total of 100 allowing quick conversion to percentages, in the real program this is calculated in odds allowing for quick and accurate measurement.

1. Weather forecasting: Tomorrow I am going to go camping, Id live to know if its going to rain. In my country the prior odds of it raining are 20 to 80, or 20%.
   I look at the weather forecast, their forecasts have a likelihood ratio of 90 to 10, or 90%. That means that the weather forecast is correct 90 times for every 10 times it is wrong.
   The way to calculate my posterior expectation of having rain tomorrow is to multiply 20/80 with 90/10 or (20/80)*(90/10) = 1800/800 ≈ 69/31 or 69%.
   This means that after checking my weather app I expect a 69 to 31 odds of rain tomorrow.
2. Disease detection: I go to the doctor for a regular routine checkup. In my age bracket the prior odds of having heart problems is 1 to 99, or 1%.
   I undergo a test that has a likelihood ratio of 95 to 5, or 95%. This means that the test is correct 95 times for every 5 times that it is wrong.
   The way to calculate my posterior expectation of having heart disease after this test is simply to multiply 1/99 with 95/5 or (1/99)*(95/5) = 95/495 ≈ 16/84 or 16%.
   This means that after this test I expect a 16 to 84 odds of having heart problems. If it surprises you that this test still means I most likely don't have heart problems then please again watch this video.
3. Disease detection part 2: I go back to the doctor because a screening test showed that I might have heart problems . My cohort with one positive screening test show an odds of having 16 to 84 odds of having heart problems.
   Next I undergo a really strong test that has a likelihood of 99 to 1. this means it is correct 99 times for every time it is wrong.
   Then the posterior expectation is (16/84)*(99/1) = (1584/84) ≈ 95/5 or 95%. This shows that the two tests together are able to be strong enough to overcome the initial low likelihood.
   

Separating out the prior and the likelihood ratio like this allows us to multiply together many tests. If we take the same 2 hart problem tests of above we could combine them into a single stronger test. We can do this by multiplying the tests giving us a combined likelihood ratio of (95/5) * (99/1) = 9405/5[1].

To show that this gives the same result we can use this test on the original prior of 1/99 again by multiplying (9405/5)*(1/99) = 95/5 or 95%.

With this in our toolbelt we are now able to add together any amount of uncorrelated updates to our hypothesis. However in the real world we find many pieces of evidence that are correlated. We would still like to be able to use these pieces of evidence.

Opinion pooling

In the extreme fully correlated evidence points at the same claim. One example of this is measuring temperature in the same room multiple times, in this case we just want to average out the measurements.

If we have two experts on Weather forecasting and we ask both of them if next week there will be a hurricane hitting the coast they will most likely give two separate odds. Lets look at one scenario.

The fist expert gives 99:1 odds of there being a hurricane and the other expert gives 50:50 odds of there being a hurricane. We want to add together their claims, but linearly adding the claim together would fail to take into account the extra confidence of the 99:1 odds expert. The middle ground is multiplicative averaging.

This can be generalized to any combination of two odds.

Here we can also give every expert a different weight the important part is that the total weight adds to 1. So if we give expert 1 a weight of 0.1 we need to give expert 2 a weight of 0.9.

We can generalize this to any amount of experts. If you're not familiair with Π and Σ. I will explain one by one first Σ essentially says sum up, so we sum up every weight unil the final weight and we want it to sum to 1. This sum of 1 is to make sure that the percentage is bounded by the claims of the experts. We do not want to claim a higher certainty than the most certain expert. The second symbol Π says to take the product. So we multiply together every odds ratio O to the power of that experts weight, just like we have done above.

If in a specific case we take all weights to be the same we can conclude that this average weight must be 1/n to add up to 1 in total. This gives us.

Combining both methods

To combine both methods we will start by picking back up the Bayesian update

We can see that we can add many experiments by multiplying by the likelihood of each experiment. This gives us.

The final change that we need is that we can use all previous claims as experiments[2]. This way we can see both the original odds and the likelihood ratios all as multiplied odds.

When we combine this with the opinion pool we will start to see the formula that we used. When the correlation is 0 every claim is evaluated separately and we are doing a Bayesian update and when correlation is 1 we are opinion pooling the subclaims.

This odds accumulator allows for adding together many different sources and subclaims. These calculated odds can then be the input odds of a new claim.

Grouping node with real world data

The core of my system I would like to call the grouping node. This grouping node is a slightly more complicated version of the subclaims above because it is also able to account for the strength of different sources on this claim. This grouping node will be shown in a bit in the form of a widget. First I will go over every part you can find in it.

The node below aims to answer the question: "What is the likelihood that the associated claim is true?". In this case the claim is: "A credible lab pathway exists for Covid". This is the value you see in the green field, by default 80%. It comes to this value by combining every piece of evidence connected to the claim.

At the top you could put in a prior, or knowledge before specific sources. This prior can represent previous knowledge you believe is not represented within any sources, If you're making claims about a coin toss this prior can represent that almost all coins are fair 50/50 coins. This prior can have a strength and a specific percentage. By default it is put to 0 to say that all knowledge this node has comes form its sources.

Below the prior you can see the sources. In this case S1-S7 each of these sources show the odds of the claim being true based on this specific source. These sources get multiplied like in the example above. These sources show one representational quote from the source and are a link to the source. This means that everybody who uses this tool can analyse every part of every claim and see what the result would be if one or more sources were interpreted differently.

The way to interpret the odds ratio next to each source is like an answer to the question "How often would we see this source in a world where the claim is true compared to a world where the claim is false". To give an example lets look into the claim "My coin has heads on both sides" and then we have the primary source "The coin landed heads after a toss". If the coin was fair we would expect 50% of the time heads, but if it had heads on both sides we would expect it 100%. We take the ratio between these two probabilities. This gives us 2/1 odds. So in this case It would be a supporting source with 2.0/1 odds.

To use the S5 WHO-China example. We are effectively saying "WHO-China is 1.6 times less likely to release this statement in a world with a credible lab pathway compared to a world without it". It can sometimes be impossible to know this likelihood ratio with absolute certainty. That is why this tool also gives a 90% certainty range that gets properly propagated into the output estimate.

In order for a tool like this to be at its most relevant we do need to calibrate the langage models. Here we can dig into the structured expert judgement literature Cooke, Hanea and Burgman have all spent decades calibrating different judgements. With calibration this kind of tool can go way further.

You can also change every relevant value simply by clicking and sliding the value. This is one additional way to make this knowledge tree accessible and approachable to everybody who uses it. I do not intend to have it feel like the computer just tells you the way something is. Instead I aim to show where different parts of the argument come from and how each part impacts the final claim.

Below you can see the grouping node visualized as a ledger. Every value is editable and I encourage you to try:

Attempting to graph the structure of arguments has been done before Squiggle, Kialo, and Argdown are a few examples. These services, however, have always had a hard time taking off, for what I believe is a simple reason: mapping out arguments is boring and hard work. People who want to map out entire arguments are few and far between, and those who do can already gather quite an audience from putting in this work.

Here is where I believe we have the new opportunity. Language models have now become capable enough to fill out these full graphs with only light handholding. And if the graphs are made to be inherently transparent any mistake will also quickly be transparent.

The fractal upside

The upside of this grouping node structure is that every node could have not only primary sources as input but also other grouping nodes allowing for building claims out of other sources. This allows us to argue for subclaims, as you can see the example claim is a subclaim of the Covid origins argument. This also allows us to chain together all claims into a big graph allowing for even more complicated representations and more accurate conclusions. If you're curious as to one implementation of is idea you can check it out on my website.

What is still needed for the magic encyclopedia

The method presented in this post is far from enough to map out every claim. This is only a starting point to apply some relevant mathematics to this subject. In order to show what I believe is still needed to use this as a building block I will use the 3 layers suggested by the Epistemic Case Study Competition. In this structure the three layers are ingestion, structure and assessment. This tool lives in Layer 2, where we try to structure every relevant part of a claim. This structure should be objective and be shared between everyone.

Layer 3: Assessment

Assessment is the most abstract layer. From this layer we need consistent testing to see if the tool is really useful and if people would really need it. The current implementation of this tool is transparent to help with this.

Layer 2: Structure

This current grouping node is still limited in many ways. This tool only combines odds of different claims. While this allows some level of clustering this cannot represent all claims. Some simple arguments such as "If you're outside and it's raining, you will be wet" cannot be contained in the grouping node. That is why I intend to add Boolean logic nodes and arithmetic nodes. Both of these together will allow every claim or combination of claims to be represented in this system. I plan on having my claim analysis system have these seven nodes.

• Noisy AND node:
  The noisy AND node allow for a group of blockers to be taken into account. In this formula is the probability that a subclaim holds is the blockers strength if the subclaim fails and is the base rate chance of success.

• Noisy OR node:
  The noisy OR node allows for a group of unlocks to be taken into account. In this formula is the probability that a subclaim holds is the unlocks strength if the subclaim holds and is the base rate chance of success even if all claims fail.

• Possibility node:
  The possibility node allows the hypothesis space to be split up into different hypotheses one of which has to happen. In this formula is the unnormalized probability of a claim ) is the normalization factor and is the normalized probability guaranteeing that all hypotheses add up to 100%.

• Distribution node:
  The distribution node allows for uncertain values to be represented and reasoned about. Each distribution node has a domain such as all positive rational numbers. In this formula is the uncertain value that is represented and describes the probability distribution of what X can be.

• Estimate node:
  The estimate node allows for a fermi estimate to be made using different distributions. A difficult to estimate distribution can be turned into many easy to estimate distributions. In this formula represent different distributions represents a formula using these distributions and represents the output distribution.

• Predicate node:
  A predicate node allows for probability claims to be extracted from distribution nodes. It does this by calculating the probability that a claim lies above a given threshold value. In this formula is the given threshold value is an uncertain value from a Distribution node and is the output probability of this node.

• Grouping node:
  The grouping node as shown in this article can be used to combine multiple sources into a single probability. This is needed because in the real world most claim will not have single conclusive sources and as such sources need to be grouped together. In this formula is the number of input ratio's is the correlation over all input nodes and sources is the every odds input and is the posterior.

With these 7 nodes in place all non causal claims can be fully represented and reasoned about. In further articles I will get into more detail regarding the other 6 nodes.

Layer 1: Ingestion

Ingestion is the combined process of finding primary sources and checking their validity. My structural tool needs this ingestion to connect the primary sources with claims. The most important component this structure still needs from layer 1 is a process that can answer any version of "What is the likelihood ratio of this primary source saying what it says depending on whether the claim is true or false". It also needs to find a reliable answer to the question "How strongly correlated are these two sources"[3].

1. ^

   To break this odds ratio down into something like a percentage requires us to go all the way to 9995/5 or 99.95%

2. ^

   This is also done in Pearl Probabilistic Reasoning in Intelligent Systems on chapter 2.2.2 page 45

3. ^

   This is my first-ever Lesswrong post. I would like to thank Tom, Glenn, Mark, and Elisabetta for helping me by proofreading and sharing their thoughts on the article.

Discuss

The tl;dr has spoilers, so I've put it at the end.

Also feel free to skip any of the chapters because the post turned out to be very long. I think you can read almost any of them on their own, and you can skip to On practicality if you want the big picture.

The title is stolen from Scott Alexander's post Extreme Rationality: It's Not That Great. Basically, it will be my opinion on applied rationality and its successes and failures. The title speaks for itself: I think there are few successes. A few weeks before Scott's post, Eliezer wrote A Sense That More Is Possible where he urged developing training for rationality; and though he thought rationalists were just ordinary people back then, he believed they could become much more. Center for Applied Rationality (CFAR) was created in 2012 with the goal of developing such an Art. So I will discuss rationality's basic values, where I disagree with them, and where they led.

But first, let me start with my personal story.

I was one of the main organizers of LessWrong events in Saint Petersburg, Russia, for two years. I started by reading HPMOR and the Sequences and running discussion groups about them. But then I took a CFAR-like workshop at Kocherga (Moscow) and became obsessed with applied rationality: I tried to practice Hammertime and CFAR handbook. After some time I decided I had enough knowledge to teach rationality, and together with a friend I developed an educational program, which failed for multiple reasons. I also studied bioinformatics and worked for two years at Gero, an anti-aging research company, precisely because I had read HPMOR and been inspired by transhumanism (I still am).

Time passed; I was in psychotherapy and realized that a lot of the things I had been trying to achieve were very unhealthy. I started to criticize applied rationality for that—partly because it is quite comforting to blame a group's ideology instead of your own personal problems. Since I was aware of that bias, I was curious to actually figure out how damaging CFAR practice had been to me and to other people. But I didn't focus on it too much: I just wrote a couple of small posts a few years ago and that was it (they were in Russian, and basically mirrored the posts of another co-organizer of rationality courses in Moscow). Recently I recorded a podcast (also in Russian) with the ex-leader of the Moscow LessWrong community Slava Matyukhin, who supported a lot of my claims, and I became really curious about what is going on—and what had been happening—in the English-speaking part of the community. So I started to assemble the complete picture.

The post will be structured as follows:

1. I will try to understand: are people really so biased and irrational?
2. I will talk about what values the rationalist community implies, and what I think about them.
3. I will discuss the practicality of CFAR: did it help people in the end?
4. Finally, I will talk about the promises and the sense of self-importance on LessWrong, and draw some conclusions.

There are a few posts that I think intersect with what I am saying here, but I am not going to cite them: Rationalist Epistemics and the Sequences (Effective Altruism Definitions Sequence), Rationalist Epistemics and Social Epistemology (Effective Altruist Definitions Sequence), The Rationality Wars.

On biases

Loss Aversion

The first and foremost bias, the cornerstone of Kahneman and Tversky's theory, is Loss Aversion. When I only started to develop my understanding of the topic I stumbled upon a post by Jared Peterson: Biases Don't Exist, and Humans Are Not Irrational. It was debated there whether biases are actually bad things, because a bias is just a deviation from utility theory, which is not always right.

One example for a failure of utility theory he proposed is ergodicity. You probably know the game where you flip a coin: heads gives you a 50% increase to your bankroll, and tails decreases it by 40%. This process is non-ergodic, and the expected value of each flip is positive (+5%), yet almost every individual trajectory decays to zero, meaning you definitely lose everything. Any trader understands that you can't just maximize the utility of the next bet; some risks are not worth the cost, because after a couple of tries your account is depleted and you never recover.

A simple explanation of the coin example is that, with large N, you get the following (a 50% probability of adding 50% means utility is multiplied by 1.5, and a 50% probability of losing 40% means utility is multiplied by 0.6):

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c200B::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c1D463.TEX-I::before { padding: 0.443em 0.485em 0.011em 0; content: "v"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c2229::before { padding: 0.598em 0.667em 0.022em 0; content: "\2229"; } mjx-c.mjx-cAF::before { padding: 0.59em 0.5em 0 0; content: "\AF"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; }

So utility decays with N at a rate of the square root of 0.9.

Another related caveat is that humans actually have diminishing returns on everything, so it is fine to fear losing a given amount of money more than you value not gaining the same amount.

But as it turns out, all of this is not new: Bernoulli invented diminishing returns while trying to solve the St. Petersburg paradox in 1738 (interestingly, the city of St. Petersburg was only 35 years old back then). The St. Petersburg paradox is a game of coin flips with a casino: it pays you dollars each time tails appears, where n is the flip number, and the game ends when heads appears. The expected payoff of this game is infinite (I'll just assume you can handle the series in your head). But for some strange reason, people rarely pay more than $25 to enter such a game. The solution is the invention of diminishing returns, and it can be any concave utility function: the more you have, the less additional utility you get from the same amount of money.

Ergodicity was introduced by Ludwig Boltzmann in 1871. Though I was too lazy to research when it was introduced into economics, I found a fairly recent paper that summarizes its effects well: Peters, O. (2019), "The ergodicity problem in economics." Nature Physics. The conclusion of interest for us here is that ergodicity forces us to use a log utility function, which is equivalent to accounting for diminishing returns — an adjustment already widely used in economics.

So did Kahneman and Tversky account for that as well? If you look at their original paper — Tversky & Kahneman 1992, "Advances in Prospect Theory" — they model utility in a fairly complex form:
for x ≥ 0 for x < 0

Where x  ≥ 0 is the utility of gains and x < 0 is the utility of losses. I am not sure if it is very important that the logarithm function required by ergodicity doesn't fit well here, but it definitely accounts for diminishing returns, and ergodicity gives a quite similar effect anyway.

And even though we kind of accounted for these two things, we still see two weird effects here.

First is the reflection effect: the fact that the function is convex on the negative side instead of concave, and the tricky part is that this change is relative to your current state, or reference point. On the standard theory you would not see anything special at the reference point at all: your utility function starts at zero wealth, and from there you have just log utility. It would not depend on whether you lose or gain in a particular scenario, but on the level of wealth it leaves you at. But humans fear additional losses less and less, regardless of how much money they start with. Also, the function should not have become convex. More plainly, you can describe it as risk-averse behaviour for gains and risk-seeking behaviour for losses. People want to risk more even when expected utility is negative, in cases like the choice between (−$4,000, .80) and (−$3,000 sure). Because the jump from −3000 to −4000 is less scary than the jump from −2000 to −3000, they may choose −4000 even though 4000 × 0.8 = 3200. There is actually another effect at play here, probability weighting, but I don't want to go into it.

Second is the kink. It is the fact that we have a lambda coefficient in front of losses, which tells us how much more losses loom than gains. It is not as straightforward as saying that losing a given amount of money has lambda times the weight of gaining the same amount, because we weight not the raw money but money already transformed by diminishing returns plus the reflection effect.

So, even though we should keep in mind that it is not as simple as losses just hurting more than gains, doesn't all this mean that ergodicity and diminishing returns don't save us from the famous loss aversion, and that people do act weirdly? Well, yes — but there is more.

Though The Brown et al. 2024 meta-analysis found that the loss aversion coefficient is 1.955, with a 95 percent probability that the true value falls in the interval [1.820, 2.102], other meta-analyses such as Walasek, Mullett & Stewart found a small λ of 1.31, 95% CI [1.10, 1.53]. I decided not to go down this rabbit hole, but apparently there are at least some debates over how big the effect is (which reminds me of the time I was trying to make sense of psychotherapy efficacy by reading meta-analyses, and all the effect sizes looked basically like random numbers to me, ugh).

And then there is another very important paper: "The Loss of Loss Aversion: Will It Loom Larger than Its Gain?" by David Gal & Derek D. Rucker, 2018. It raises a lot of points that I don't fully cover here, but what I want to do is point out some things Scott Alexander said about it in his post [Crosspost] On Hreha On Behavioral Economics, which is a reply to The Death Of Behavioral Economics by Jason Hreha.

It’s a great 2018 paper that looks at recent evidence and concludes that loss aversion doesn’t exist. But it’s a very specific, interesting type of nonexistence, which I think the Hreha article fails to capture.

G&R are happy to admit that in many, many cases, people behave in loss-averse ways, including most of the classic examples given by Kahneman and Tversky. They just think that this is because of other cognitive biases, not a specific cognitive bias called “loss aversion”. They especially emphasize Status Quo Bias and the Endowment Effect.

I think this slightly understates the analysis they made. We already discussed that loss aversion is actually two distinct phenomena — the reflection effect and the kink — both united by the reference point, which is itself a contradiction of standard risk-aversion theory. The kink is what makes us think that losses loom larger than gains; it differs behaviourally from risk aversion — it produces a discontinuous change at the reference point and doesn't depend on scale (it is just a constant linear parameter), unlike risk aversion, which is described by a concave function that becomes more important as the scale increases. That's why Rabin & Thaler explained that loss aversion for small stakes can't be explained by risk aversion (whereas for larger stakes it can be interpreted as such).

G&R cite a paper that refutes loss aversion at low stakes: "Is loss-aversion magnitude-dependent? Measuring prospective affective judgments regarding gains and losses" by Mukherjee et al. (2017), and it is a blow to the whole theory, not just a reinterpretation. They also cite Katz (1964), who showed indifference to risk at small bets.

The other reference doesn't even use the small-stakes trick, but cleanly shows the same thing — that there is no discontinuity at the reference point:

contrary to the predictions of loss aversion, research shows that individuals are no more-risk averse when choosing between different potential gains (e.g., choose between (a) gaining 1000 points for sure or (b) a bet with even odds of gaining either 0 or 2000 points) than when choosing among options where one of the choices involves potential for loss (e.g., choose between (a) receiving 0 points for sure or (b) a bet with even odds of losing 1000 points or gaining 1000 points) (Erev et al. (2008))

It is basically the same choice with the reference point shifted by 1000; the fact that people are indifferent between the two contradicts prospect theory, which predicts dependence on the reference point, and is consistent with risk-aversion theory, which depends only on wealth.

G&R also discuss other explanations for real-world phenomena such as asymmetric demand elasticity and the equity premium.

But Scott discusses a further development of the story though: Mrkva et al., Loss Aversion Has Moderators, But Reports Of Its Death Are Greatly Exaggerated.

Previous criticisms of loss aversion argue that most experiments are performed on undergrads, who are so poor that even small amounts of money might have unusual emotional meaning. Mrkva collects a sample of thousands of millionaires (!) and demonstrates that they show loss aversion for sums of money as small as $20.

Another interesting point in the G&R paper was about decoupling loss aversion from action/inaction in the endowment effect.

a simple procedural change that decoupled loss and gain from inaction and action eliminated the preference for an endowed option.

I don't think this is just a change of perspective either, because avoiding action is clearly a rational choice, especially at low stakes, and it is important to distinguish it from loss aversion.

There is also another piece of prospect theory that we discussed — the reflection effect. It wasn't challenged at all in any of these papers; there are some other sources that criticize it, like this one, Prospect Theory's Reflection Hypothesis: A Critical Examination, but I didn't dig into it.

Anyway, I am not enough of an expert to read all the papers in the field and draw a comprehensive conclusion; it could be that all these refutations are wrong and the original findings of Kahneman and Tversky are right. But my point is that even for one of the most studied biases the discussion still seems to be ongoing, and that it is not only about reinterpretation but also about the correctness of the theory. Let's now look at the other examples a bit more quickly.

Other biases

There is another great article: "THE GREAT RATIONALITY DEBATE" by Philip E. Tetlock and Barbara A. Mellers, 2002.

They summarize the cognitive-bias research and draw a distinction between two kinds of counterargument. One is experimental: researchers try to reproduce results and fail, or tweak the conditions of an experiment a little and the result changes completely, and so on. The other is when researchers challenge what counts as normative, and whether biases are actually bad — I will call these reinterpretations.

A couple of their examples caught my eye:

Disjunction effects. Should Shafir’s students be criticized for violating the sure-thing principle (for wasting money to delay a decision until an irrelevant uncertainty is resolved)? Or should they be applauded for recognizing, deep down, that they are poor hedonic forecasters who have drawn the lesson from bitter experience that it is a good idea to postpone decisions such as vacations until they know how they will really feel about passing or failing the exam?

Overconfidence. Should Camerer and Lovallo’s entrepreneurs be dismissed as Willy Loman dupes of an overconfidence illusion that they could have escaped if they had the good sense to adopt an outsider, or base-rate, perspective on the odds of success? Or would these entrepreneurs, without the energizing effects of overconfidence, have been paralyzed by loss aversion?

But the paper is old, so I will have to add more examples myself, while sticking to their classification.

Experimental refutations

As a quick aside: in this witty and funny post by Scott Alexander, you will find out that parapsychology is the control group for science, and that the replication crisis is a mess; Beware The Man Of One Study is really good as well. So let's take all these meta-analyses and other shmalyses with a grain of salt.

Anyhow, the most obvious example of the first kind of counterargument is Priming and Contamination. I am sure everybody has heard about this, so I will not delve into it, but here is one source.

There is also a refutation of Bystander Apathy.

There is a good post by Kaj_Sotala that covered multiple examples of challenged biases: Are these cognitive biases, biases? A particularly interesting one is overconfidence (or, more precisely, the hard-easy effect).

Reinterpretations

One way to approach this is to criticize existing decision theory, which Jared Peterson discussed in his Biases Don't Exist, and Humans Are Not Irrational. I covered only one example, related to loss aversion, because I am not competent enough to understand the rest; if you are interested in more, please read his posts.

Another is to emphasise real-world success over mathematical correctness — the idea that simple heuristics, which make more errors but just work in most cases and consume less "compute", could be preferable. The main proponent of this approach is Gigerenzer. There is, again, a really good post by Kaj_Sotala about this: Fundamentally Flawed, or Fast and Frugal?

Gigerenzer basically says that people use simple heuristics which are very effective in real life and can create bugs in some edge cases, but that this is much better and more efficient than using correct Bayesian decision-making. There is also a point that our thinking works much better in an intuitive regime, and breaks much more easily when we have to make logical, conscious decisions — which I will discuss later.

Some social context can also change your incentives, and biases become rational: Tetlock (2000), Cognitive Biases and Organizational Correctives. For example, overconfidence or the fundamental attribution error can be necessary in some social environments — ones with high competition and high stakes, and a need for fast, heuristic judgments.

Specifically, conservative managers with strong preferences for cognitive closure were most likely (a) to defend simple heuristic-driven errors such as overattribution and overconfidence and to warn of the mirror-image mistakes of failing to hold people accountable and of diluting sound policies with irrelevant side-objectives.

The next example of a bias is quite strong and I will discuss it using Gigerenzer's approach.

Conjunction fallacy

In Eliezer's Burdensome Details, and furthermore in Conjunction Controversy (Or, How They Nail It Down), he discusses the conjunction fallacy, where people prefer a detailed hypothesis to a more general one — for example, they assign more probability to “Russia invades Poland, followed by suspension of diplomatic relations between the USA and the USSR” than to “Suspension of diplomatic relations between the USA and the USSR”. Eliezer very convincingly argues that people just "substitute judgment of representativeness for judgment of probability".

This is actually a strong case. There is even a Kahneman paper that successfully disproves the claim that Gigerenzer's reformulation in frequencies instead of probabilities eliminates the effect (it does reduce it, though). But the question is whether it is a quirk of a specific setting or a ubiquitous bug.

The conjunction fallacy violates the fundamental law of probability that P(A&B) should not be greater than P(A). Also, if people always substitute representativeness for probability, that means they ignore priors: they basically compute how much the posterior probability of H given A — P(H|A) — is increased compared to P(H), instead of the posterior itself (Crupi, Fitelson & Tentori 2008; Tentori, Crupi & Russo). All of this seems quite unrealistic to me; people wouldn't survive at all if their cognition were that fucked up. Again, Tetlock says:

Skeptics maintain that if people were as incorrigibly irrational as Kahneman and Tversky suggest, human ancestors never would have survived on the savanna plains of sub-Saharan Africa.

I think people, when it comes to real life, definitely account for priors at least in some situations. We realize that if someone is shouting that there is a dinosaur down the street, it is less likely to be true than if, in the same situation, someone were shouting that there is a bear on a unicycle.

The one real-life example I know of where people fail to account for the conjunction rule is in juries. But in this case you could argue that a detailed story increases the probability that it is not a lie, which is maybe one of the reasons this bias could be evolutionarily favourable.

There was a further discussion about these refutations in the comments, and Eliezer dismissed them in quite a sharp, accusatory tone:

Though Kahneman and Tversky conducted experiments on doctors as well, I think Eliezer is overstating things here when he says "The patient still dies". What Kahneman's experiments tested was the probability of getting certain symptoms given a known disease, which is not what actually kills patients. In real practice, doctors need to diagnose people who have multiple symptoms, and to judge how many diseases they have and which ones. And for that, doctors even have Hickam's dictum — "a patient can have as many diseases as they damn well pleases," which exists precisely because doctors sometimes over-apply Occam's razor. It is standard clinical teaching that the default norm is parsimony (one diagnosis explaining everything).

I know this might sound like cherry-picking — they did have a bias, right, so why does it matter in what scenario it appeared? It is true that Gigerenzer's point, that people are on average rational in real-life scenarios, is a dodgy one and a kind of goalpost-moving: we didn't succeed in refuting careful experiments, so we resort to the theory that everywhere except the lab setting we are rational, which sounds unfalsifiable. But, first of all, some of the biases are actually refuted, and the field of lab social experiments isn't one to be trusted without any doubt. Eliezer puts much more confidence in the science of such experiments than it deserves. And, more importantly, the question that interests us in the end is a real and falsifiable one: how biased are we in real life?

The next bias is another example when the experiment trapped people in unrealistic conditions, while in real life their strategy works well.

Confirmation bias

Well, do rationalists really play Zendo better than people ignorant of the Art? In "Confirmation, Disconfirmation, and Information in Hypothesis Testing" Joshua Klayman and Young-Won Ha explain that, at least in the case of Wason's 2-4-6 task, they probably don't. The simplest version of the game is as follows. The game master comes up with a rule for a sequence of three numbers and provides an example that satisfies the rule. A player can run tests — provide a sequence and ask the master whether it satisfies the rule or not — and should guess the rule with as few tests as possible. A usual experiment goes like this: the master invents the rule "just any three positive numbers" and says 1 3 5. Then the player tries 2 4 6, 6 8 10, 10 12 14, settles on the rule "three integers spaced two apart", and gets it wrong.

And we say it is confirmation bias! He just wanted to confirm his rule! But this is straight up wrong. We don't know what he wanted to do. What he did was provide three sequences of numbers that satisfy the rule he was trying to test.

There is a difference between positive/negative tests and falsification. A positive test is a test of a sequence that your theory predicts will be positive; a negative test is a test of a sequence that your theory predicts will be negative. A falsification is a test that contradicts your theory — i.e. your theory predicts one thing but the test output is another. But how can we know which test will turn out to be a falsification? Well, we can only do it probabilistically. Let's say we have a space of all possible states, where theories are subsets of this space, consisting of the states that satisfy them.

What's then the probability that a positive test will give us a falsification? It is the number of states that are INSIDE our theory but OUTSIDE the correct one, divided by all the states INSIDE our theory. It is proportional to in the picture above, and it is just impossible, because the region outside T doesn't intersect with H. So all the states inside our theory are positive, and they fail to give us a negative result on the correct theory only if all of them are inside that theory as well. And for a negative test it is the number of states OUTSIDE our theory but INSIDE the correct one, divided by the number of states OUTSIDE our theory. It is proportional to in the picture above.

But now consider that in real environments the thing you're trying to pin down is almost always a sparse, specific pattern — one disease out of thousands, the single rule the game master happened to pick — so the true rule T occupies a tiny corner of the space, while the hypotheses (H) we actually entertain tend to be broader or at least similar in complexity.

That means that the sheer number of states outside our theory is usually much larger than the number inside, and because the correct theory also (we assume) should be quite small, a negative test has a very small probability of producing a falsification. On the other hand, if both theories are quite small, finding where they don't intersect is not so hard. In simple terms, most of the examples that you can present will be just negative for both rules/theories, so presenting a random negative example usually gives you nothing.

So the situation usually looks more like this.

Or even like this.

Again, we can't know for sure which test will lead to a falsification: in the first picture only negative examples could falsify our theory, in the third only positive ones can, but we don't know in advance which picture applies to our situation, so we have to use some priors. And natural experience tells humans that the 2nd and 3rd are more probable than the 1st. So usually you just want to use positive examples much more often than negative ones — unless someone has constructed an experiment specifically to catch a person using this heuristic, which is very useful on average. But when you play a real game, such a trick will be exhausted very fast, and you will start to come up with really complex theories (which have a small number of correct states); and if a rationalist still confuses negative examples with falsification, he could waste a couple of rounds on useless tests.

In Positive Bias: Look Into the Dark Eliezer makes this distinction right away:

This cognitive phenomenon is usually lumped in with “confirmation bias.” However, it seems to me that the phenomenon of trying to test positive rather than negative examples, ought to be distinguished from the phenomenon of trying to preserve the belief you started with. “Positive bias” is sometimes used as a synonym for “confirmation bias,” and fits this particular flaw much better.

But then he goes:

It once seemed that phlogiston theory could explain a flame going out in an enclosed box (the air became saturated with phlogiston and no more could be released). But phlogiston theory could just as well have explained the flame not going out. To notice this, you have to search for negative examples instead of positive examples, look into zero instead of one; which goes against the grain of what experiment has shown to be human instinct.

And:

I have been writing for quite some time now on the notion that the strength of a hypothesis is what it can’t explain, not what it can—if you are equally good at explaining any outcome, you have zero knowledge. So to spot an explanation that isn’t helpful, it’s not enough to think of what it does explain very well—you also have to search for results it couldn’t explain, and this is the true strength of the theory.

If you constructed a theory that can explain anything, then positive examples would actually be the only way to falsify it. In the 3rd picture you would have H = U. It would be like saying my hypothesis is "just three numbers": then 1 2 3 is a positive example, and -9 pi sqrt(3) is positive as well and will falsify your theory, because it doesn't satisfy the target rule. It is still right to point out that phlogiston theory is a bad theory, but I don't think the reason is positive bias.

Of course, this is just the most famous example of confirmation bias, and refuting it doesn't refute confirmation bias in general. And from my personal experience this is the bias I have experienced most myself — it is really hard to change your opinion. Even though the Backfire effect is probably wrong. But I think there are plenty of rational reasons to hold your beliefs strongly that make it less dark than Eliezer presents. For example, people with a lot of experience are usually already exposed to a lot of information that confirms their view, and they could have tried to correct for this intentionally, but then they would sacrifice time they could spend developing expertise or building a reputation for their view — and that is fine in science, for example, where one person can't possibly gather all the evidence, and it is genuinely better if he works on his own theory. It is not good for his epistemic fairness, but who cares about that if he is successful and society benefits from it too. And I think it works in regular life as well: there are non-epistemic costs to switching beliefs, and always being unsure and flipping your beliefs at the slightest bit of new evidence hurts a lot.

Nevertheless, I think there is a very important kind of "bad" reason for holding on to your beliefs. There is a beautiful post by Scott: Guilt: Another Gift Nobody Wants. In short, it explains why it is very important to have a mechanism that convinces other people you are not incentivised to do bad stuff when no one is looking. One example is guilt: it is an intrinsic property of almost every human being, and people know other people have it. But roughly the same logic could work for holding strong beliefs. If you really need people to invest in your theory or vote for your party, you had better say you are a hundred percent sure it is the right thing — and when you say that, it is better to be honest about it, because people can notice when you are insincere. Obviously, for politicians it doesn't hold so well — they are the finest masters of lies — but for other people it seems to work quite well.

But, as I will discuss more later, in my opinion it is hard to disentangle a general bias — a rigid property of everyone's thinking — from other psychological effects or just random human errors.

Planning fallacy

Well, here I am not even going into research and papers, but will just appeal to my own experience, so you can rightfully skip this section right away.

I think it is quite apparent that we do actually underestimate the time we need to do things, especially at work, where we don't want to do anything at all, haha. There are some exceptions: I think being early or late differs dramatically across cultures, and in some cultures people are usually on time or only a little bit late. And most people are usually on time at airports.

The hypothesis that we just imagine how things can go right and run with it may well be true, but the alternative — thinking about all possible failures — is not necessarily always useful. First of all, there is only a finite number of ways things can go right, and an infinite number of ways everything can go wrong, and even keep going wrong indefinitely. So there is an asymmetry in the probability space towards infinity: the most probable outcome sits somewhere close to zero, and to the right there is a slowly decaying curve, like a Lévy distribution with no finite mean or variance. We still usually get things done, because most of the failures don't happen; but when we fail, we simply abandon the task. Also, it is not always the case that you need to predict everything in advance — quite the opposite. More often you will change things on the fly, other people will delay something, or the goal will shift slightly, so it won't matter that you miscalculated something. And it is not true that people never think about failures: we take medicine on a trip, we ask someone for help in advance when past experience tells us we won't manage on our own, we set reminders and alarms, we put our things in lockers, and so on. The only question is how much we think about failures; and though we systematically underestimate time and costs, that doesn't mean it isn't optimal behaviour. Given the unpredictable changes in circumstances and the infinite complexity of the possible failures we'd have to account for, it is perhaps too costly to always estimate correctly — or maybe it is even impossible not to underestimate on average, due to the divergent nature of the distribution.

So, do humans need fixing in the end?

We looked at several examples of cognitive biases: some are refuted, some are up for interpretation, and it does seem like people are biased somewhat — but the evidence that their condition is very serious, that it paralyzes all decisions and prevents humanity from functioning normally, is not overwhelming.

But wait, it seems apparent that people do very stupid things on the global scale — how can you argue with that? Let's first go back to where we started: we have to nail down what a bias is. We already know that a bias is a difference between a human decision and a rational theory.

But what's also important here is that we look only at the general rules of how the human mind functions. If one human makes a mistake and another doesn't, it is not a bias. But what if a lot of humans make this mistake, but not all? Take psychotherapy, for example: surely a lot of people have narcissistic disorder, autism, or ADHD; they could even gather in a community and declare that the human mind is susceptible to having too high an opinion of itself, and that we should fight that in all human beings. But what they actually need is to accept that it is their own peculiarity — not everybody has it — and it is good to trace the reason you acquired it and adapt accordingly.

In my opinion, a lot of evil in life comes from people with psychological problems and deviations, and definitely not all of it comes from cognitive biases, as Eliezer claimed (there is also the Moloch problem and other things — I'm obviously not saying all evil comes from not seeing a psychotherapist).

I am not talking about people who are just depressed or have an attention problem, but about people like Stalin, Mao Zedong, and Hitler. Maybe they could have used a little more rationality and a little less overconfidence — before you decide you can invent a way to grow rice twice as efficiently, only to have cannibalism spread across your country and more casualties than in World War II. But probably the reason they didn't have enough sanity was that they were narcissistic psychos.

How much of all evil comes from cognitive biases is not obvious to me. I think this is a very important thing to research before you put investment in the Art at the same level of ambition as saving humanity.

But there is also a point to be made that some irrational behaviours (including biases) don't need to be fixed because they are load-bearing — things that are irrational in theory but optimal given the engineering of our brains. Some rational actions cause too much stress and carry too much cost, given the limited ability of humans to process feelings and thoughts.

Some rational decisions will not account for the fragmented nature of our brain — you can consciously assemble a perfectly unbiased Bayesian argument while silencing the parts of your intuition that tell you it is wrong for some irrational reason, but forcing yourself to do it will come at a cost greater than the rational benefit.

It is similar to traumas — defensive mechanisms of our brain that shield away bad experiences we couldn't comprehend as kids and cause disharmony in our minds. But these were useful at some point as well.

You probably won't tell a guy with a fear of flying that it is irrational that he can't sit calmly on a plane. This art of accounting for our own design is quite different from the art of aligning yourself with an optimal Bayesian agent.

Rationalists will tell you that it is rational to account for all that as well, but, as with many things I will mention later, there is a pattern: when someone (usually Eliezer himself) points out some limits of rationality, rationalists will agree but keep insisting on the importance of the Art, just with this or that caveat. The problem here is that for some things it is not enough to merely account for them — some things just destroy the usefulness of your approach. Because once you account for this and for that, it is not obvious, in the end, whether you aren't just reinventing the wheel from scratch, doing all the same things but with a lot more effort.

In the end, as we already mentioned, Gigerenzer claims that, though biases are sometimes real mistakes in lab settings, they are also simplifications that work great in real life. And substituting more complicated thinking for them can backfire.

One such example is probabilities and Bayes' theorem. Bayes' theorem itself is not tractable even for modern computers; neural networks sometimes use simplifications like the ELBO, or derivatives of the posterior distribution, as in diffusion models. Maybe more useful in practice is Bayes' theorem in odds form, which — apart from some problems with priors — basically tells you to account for the control: don't only seek evidence that appears while your intervention is present, but also while it is absent. It is akin to positive bias, where people seek only to test examples that match the rule they invented, which is probably also the cause of many superstitions and of alternative medicine — though these are not necessarily irrational. Yes, not everything that can be destroyed by the truth is irrational: sometimes these are psychologically comforting beliefs, sometimes it is just an error on the side of caution (it is easier to do this ritual than to risk not doing it), and, as Scott Alexander pointed out, sometimes culture is wiser than individual people can explain. But I am not defending it outright — of course, in the modern world, things like controlled trials are blessings, and anti-vaxxers basically cause people to die. Which makes not forgetting about the control useful sometimes, even in everyday life.

But what about applying numerical probabilities?

Eliezer himself argued in When (Not) To Use Probabilities that we should only use numbers that are based on other numbers, and that we should calculate, then throw away the result and follow our intuition. Do you think rationalists followed this guidance? At least in my experience, in the Russian community many people rationalized their decision-making with made-up numbers. And many thought that following your intuition is wrong when the rational arguments say otherwise. The famous p-doom is also one of those cultural things that stuck despite Eliezer's warnings, and Eliezer himself uses it. But even if we treat this lightly and believe every word of the essay, the question remains: if one should throw away the math and follow intuition, then maybe you could just skip the math part altogether?

But I'm getting ahead of myself. Before we fully dive into the dissection of applied rationality practices, I need to discuss one more point that should be considered in understanding the usefulness of these practices.

On values

We already discussed that biases can be rational given some reinterpretation, which often involves assuming a different set of values.

You can interpret hyperbolic discounting as a mistake, or you can interpret it as a value or preference. You can basically make this flip-flop for every bias, and it can sound artificial, but I will argue that the negative framing of some biases stems from a misunderstanding of human values.

But btw sometimes people also say it is hyperbolic discounting when it is not, because they just didn't account for something that makes present more important, often it is just psychology. For example, your belly craves snacks because your consciousness punishes you for every wrong decision and tries to push you to the limit, so sweet treats compensate for that at least a little bit and slacken the tension — maybe not in an ideally healthy way, yes, but your conscious decisions can be much worse. Sometimes you also come across some new paper on how pears are actually carcinogenic, but you still want pears and eat them while complaining about hyperbolic discounting — when actually it is a good defensive mechanism of your brain resisting your conscious decision, because it has better priors and behaves conservatively. As in many cases, a person can unknowingly do something for rational reasons while believing his reasons are irrational, or while failing to understand his reasons at all.

Take Bryan Johnson, for example: he is trying to optimize his life with an evidence-based approach, without conservatism, without risk-avoidance, at his best conscious judgement — and I wouldn't want to live such a life (except that he is crazy rich, of course; that's a good bonus).

But what's more important is that sometimes instrumental convergence doesn't hold for us and we don't optimize for control over everything. We don't want a love affair with a puppet, and we don't want a climbing gym or a computer game to have a shortcut — an example that Eliezer himself mentions in Harmful Options.

Sometimes people play social games for the sake of the game, or argue in debates because they enjoy it. Rationalists often think our values are about winning a particular game or searching for truth, but actually it is also about fun and being in the process.

There is a great sequence of posts about what values the rationalist community is missing; the two most interesting to me are On green and On attunement.

The process of so-seeing requires being Reason—being your soul—as opposed to merely modeling it. You do, in fact, have to stay within something. You have to think—to seek, yourself, whatever it is that Reason seeks; to be the onrush of that part of your being. And this requires what I've previously called "living from the inside," and "looking out of your own eyes," instead of only from above.

Green is love of Nature, living as you are, living life, fully experiencing it, being in attunement. This is not necessarily the opposite of maximizing knowledge or pursuing goals in terms of your actions — you can find numerous justifications for Green in Eliezer's writings — but the posts argue that this is "green-according-to-white" or "green-according-to-blue", not Green itself. Being Green is a different state: not looking from above and merely modeling, but looking from the inside.

In The Unbearable Lightness of Being, Milan Kundera writes, "Happiness is the longing for repetition." Sometimes, calm, rural life with simple pleasures is better and happier than maximizing information, goods, and success. It is not always good to change everything, change Nature, change yourself, to optimize for something; there is a value to keeping everything as it is. Harry's urge to control everything in HPMOR is not a coincidence; it is inherent in rational ideology; it is the ambition to be powerful enough to extinguish a star to save one life. I don't think it is necessarily a bad thing, though. I am still an immortalist, but I do think that, applied to a simple life or your everyday cognition, this can become dangerous. And this is exactly the conflict that happens when you mix saving the world with self-help practices (Eliezer doesn't think the laws governing them are different but I do).

The point of bias is that it's a deviation from the utility-maximization theory of decision making, but fixing this also means that you want to control every aspect of your thinking to fit this abstract thing, rather than being yourself and giving up control.

I think it is very important to feel this distinction, and to shift your perspective on values away from this maximization paradigm, in order to decide which techniques feel right for you and which don't. But we've finally reached a point where I can't delay it any longer — it's time to discuss the mysterious beast: applied rationality itself!

On intuition vs conscious thinking

I think the notion that conscious thinking is more capable than intuition underlies almost all the techniques (I will avoid delving into too much detail here, like whether I mean System 1/2 or something else; I think in the end those details don't matter much for my point). I don't think it is completely wrong — conscious thinking is better for some things and worse for others — but what this notion misses is that the basic gears of cognition are actually better left untouched, because there is a great variety of subtleties that our intuition computes much better, and you will only make more mistakes without it.

One example is the goal-factoring technique. What it proposes is to start with some thing in question that we do or want to do, then list all its possible subgoals (using the button test to verify there is nothing more to it), and then find alternatives that cover all the subgoals without paying the costs of the thing in question that brought it to your attention in the first place. There is an example with grading students, where, after using the technique, our hero comes up with the idea of students grading each other. I think at this point a sane sceptic would raise a few questions: aren't we already using the same or a better algorithm by default, just unconsciously? Didn't our hero come up with his solution by simply inventing it with his intuition, and then justify it retrospectively via the technique? How often should we use this technique instead of default intuition?

The authors answered the first one in their opening section: quite often, when we are presented with two alternative options, we don't use our imagination to come up with a new alternative, but instead just weigh the pros and cons and choose. The key word here is "usually". Usually means there is an intuition in our brain that tells us whether we have to start searching for a new alternative or not, and usually it says "no". Sometimes we just come up with an alternative unconsciously; sometimes we realize that we want neither of the options and we search for a solution — it has happened to every person on Earth at least once in their life. But this isn't even how our brain works most of the time — launching the whole search and staying focused on it. It comes in pieces: some random thought, then another, stuck together across different times and contexts, and then a bright thought comes to mind that is the cumulative knowledge of all the alternatives, pros and cons, and desires we encountered, but computed in some distributed manner.

So, what does the technique give us? From the logic of the Sequences and rationality, you would assume there is a cognitive bias that creates some imbalance, and that we are already doing what the technique suggests, just not enough. But for a lot of techniques, including goal factoring, there isn't even a bias that they refer to. It seems like they have the assumption: we came up with some algorithm on a piece of paper that sounds right, so obviously humans in general do worse than that by default. Aversion Factoring "is not derived from any particular body of psych research, though it draws lightly on trigger-affect patterns and exposure therapy and takes advantage of the framework of reductionism." Internal Double Crux is not related to any cognitive bias; it is drawn from IFS, which is a therapy, and the approach to traumas should, I think, be more careful than a random rationality-retreat practice. TAPs are just from psychology as well, and even those I don't like. Of course, this doesn't hold for all of them: Bucket errors fight the representativeness heuristic (which is related to the conjunction fallacy we discussed before), Units of Exchange are related to the Sunk cost fallacy and loss aversion, and Murphyjitsu works specifically against the planning fallacy.

What would happen to our hero if he didn't apply the technique? I would guess that the idea of students grading each other had already crossed his mind, and that he probably noted it, would recall it again, and would subconsciously weigh all the factors that are listed in the technique — along with other ideas that he would dismiss. Because our brain does all of this by default. Yes, there is a possibility that he would miss this idea and continue to struggle without the technique, but there is always such a possibility. Making a mistake is not a bias, and not irrational, because the solution is apparent only after careful thinking, and thinking itself costs something. Obsessively applying this technique to everything you do can give you a neurosis, and applying it just a little can get you nowhere, because you would still miss important stuff. In other words, it is not "anecdotally strong" and obvious that the status quo is not the optimum — EVEN if there is some strong evidence of a particular bias (which is not always as convincing as we saw in the first section) — because, apart from knowing there is a bias, you would somehow have to guess in which situations it will manifest; otherwise you'd have to apply the technique everywhere and, again, get a neurosis.

And there is more that could go wrong. Decomposing something actually doesn't give you the full picture. Usually people can weigh all the factors and form an impression of something intuitively, not because there is a simple logical explanation for why it is so, but rather because of a complex, emergent (oh, c'mon) relation. And people on the spectrum do this worse than others, which could be one of the reasons such techniques are appealing. But the right way to fix this dissonance is to train your brain to form an attitude towards complex things — instead of using some ruminating tool, just experiment and teach your intuition through experience.

Again, even if the bias actually exists, it is not only about how often to use such a thinking method (which, fairly often, is already used by default), but when to use it. So just bluntly practicing it is not obviously going to help.

We can think about it by comparison with pharmaceuticals. They cure some illnesses — i.e. some function that is not activated when it should be, because something is broken in what the organism usually does well — and we just switch this thing on when we need it. We don't recreate all the circuits of the human cell from scratch; we just use some molecule to activate an already existing pathway. In contrast with that approach, there is anti-aging research. Aging is not something that happens to some people sometimes; it is a bug we all have from the beginning. And that's why it is hard, almost impossible, to fix with the existing approach. No matter how smart we think we are, we still can't even remotely design from scratch anything as complicated as the biological circuits our cells already have. The same thing applies to the basic functions of our cognition. Yet.

You can probably tell that the situation is similar with even simpler techniques, like "notice confusion more than you do in your usual life", "think about what can go wrong more than you already do", and the very famous "actually try very hard for at least 5 minutes, you lazy bastard!"

But wait — what about Do The Math, Then Burn The Math and Go With Your Gut? It's already been accounted for! (again). Well, the same question: even if you go with intuition in the end, it doesn't free you from the cost of the calculations and the wasted time (and money on psychotherapy) — how do you know when these calculations are useful? Well, maybe you can teach your intuition to recognize such situations with practice, or insights from the math could leak into your intuition and fix biases permanently! This is not impossible to imagine.

On practicality

But what does the practice actually say?

My impression is that almost all the CFAR techniques are useless, and I don't need them to succeed in life at all. Though I would say that some ideas from Street Epistemology are useful, and they are related to the Double Crux technique. With Eliezer's Sequences it is only a little bit better: Affective Death Spirals explained how you can get into a cult — though he didn't convince me that he isn't creating one. A Human's Guide to Words was a good explanation of some concepts for my autistic mind. Noticing Confusion and the similar Split and Commit I probably used a few times (and it happened on its own a hundred more). But what if I am just a lazy, narcissistic, AuDHD person who didn't get the Teaching? It is very possible, and I don't weight my experience too heavily. After all, I am really lazy and narcissistic, and I have AuDHD :)

Most of the people I know from the Russian community fall into three categories: people who just came for the fun, the community, and the philosophical conversations; people who tried the techniques and found they didn't help them, like me; and very few people who are still using the techniques. That last category is usually the people who also say that you should use Bayes' theorem for everything, think that everybody is stupid except rationalists, who deploy every CFAR technique in a single chat message just to argue with you, and so on.

Then again, the experience of other organizers is similar: Tanya Miropolskaya shared that she had a negative experience with applied rationality and the community in general, and Slava Matyukhin said the techniques didn't help him. And, again, Scott Alexander wrote Extreme Rationality: It's Not That Great a long time ago.

There is a study that CFAR conducted to determine the effects of their program. The results are below. All variables are coded such that positive numbers indicate an improvement from pre-workshop to post-workshop, with (R) indicating that this involved reverse scoring. Effect size is the standardized mean difference using the pre-workshop standard deviation. † p<.10, * p<.05, ** p<.01, *** p<.001.

Improvement in Cognitive Biases was not significant; interestingly, they tested them with Stanovich questions. But, unfortunately, this is not great evidence for my point, because the study is terrible — they didn't have enough statistical power to detect even three of the four biases.

Anyway, another piece of evidence that CFAR wasn't a huge success is that they said so themselves and closed their workshops. There is a summary of what happened by Anna Salamon: my low-quality thoughts on why CFAR didn't get farther with a "real/efficacious art of rationality". There is another project that follows in CFAR's footsteps, but it looks even worse.

Another problem, which Scott also mentioned in his essay 17 years ago, is that rationality is a dual-purpose technology. It was developed for exploring AI Safety and then reoriented to help people with their biases. This simultaneously damaged our understanding of human cognition, by leaning towards artificial models instead of actual humans, and damaged the project itself, by shifting its focus from the epistemic exploration of these ideas towards saving humanity right now. CFAR were actively collaborating with MIRI and convincing people to switch to AI Safety. Sometimes people even say that the whole point of Effective Altruism was to bring people to AI Safety, and Robin Hanson said this about the whole idea of rationality (which is actually quite obvious from the Sequences, but I hadn't thought about it before).

There are some posts and videos by Brienne Yudkowsky that go to another level — not just practicing techniques to get some insights, but installing them as a habit of thought via TAPs. I tried to use this a long time ago, and it made me feel really bad about all of it. It is a kind of uncanny feeling — that rationalists still treat people like Spocks, even though they say otherwise.

I think another reason for this could be that neurodivergent people created and practiced all of it. A lot of neurodivergent people like me need help with basic things that are not as problematic for normal people. And they tried to use these applied-rationality techniques to solve those problems, when they actually should have used therapy. It feels better to use applied rationality, because you are ahead of everybody: you are not fixing your specific problems, you are fixing basic human biases! But that's not what you should be doing when you have akrasia, anxiety, or painful inner conflicts and self-blame.

I don't think therapy is a panacea; there are some problems with its effect sizes, as there are with medications. But the point of therapy is much more humble — helping a person with their unique problems, as opposed to solving all the problems of humanity. And in my experience it just works much better.

On overpromising

I think people with a lot of narcissistic ambition (like me) are affected by Eliezer the most. When I first met him on the pages of HPMOR, he was in the costume of a magical boy who can do anything using rationality and science, and conquer a world of stupid people who have biases while he doesn't (ok, ok, he does, but he very masterfully accounts for that, whatever). I went to my friends and started bragging about what I had found: this is the best book I have ever read. One of them said he had started it but couldn't stand the pretentiousness; I was furious and tried to explain to him that he understood nothing. Now I think he was right, and though HPMOR is still a really good story, it is actually quite strange morally. Eliezer intentionally made Harry rational instead of brave and loving, and, to disqualify these character traits even further, instead of standing up for Ron when Malfoy basically said he is stupid and pathetic, Harry casually agreed with him. Where Rowling praised love, Eliezer says: if you are stupid, you are not even interesting to me. But I am not going to spoil the whole video for you — just go and watch it.

And then there are the Sequences. From Scott's original Extreme Rationality: It's Not That Great:

Yes, yes, beisutsukai should be able to develop quantum gravity in a month and so on. But until someone on Less Wrong actually goes and does it, that story sounds a lot like when Alfred Korzybski claimed that World War Two could have been prevented if everyone had just used more General Semantics.

Beisutsukai is another example of fiction, along with HPMOR and Three Worlds Collide, where Yudkowsky depicts rationalists as having superpowers. And these tales are built into the Sequences. He writes in A Sense That More Is Possible:

rationality is something that should be systematized and trained and tested like a martial art, that should have as much knowledge behind it as nuclear engineering, whose superstars should practice as hard as chess grandmasters, whose successful practitioners should be surrounded by an evident aura of awesome.

In Crisis of Faith:

There is the concept of isshokenmei—the desperate, extraordinary, convulsive effort to be rational. The effort that it would take to surpass the level of Robert Aumann and all the great scientists throughout history who never broke free of their faiths.

Btw Robin Hanson apparently disagreed with that: Rational Me or We?

In Human Evil and Muddled Thinking:

So let us be absolutely clear that where there is human evil in the world, where there is cruelty and torture and deliberate murder, there are biases enshrouding it. Where people of clear sight oppose these biases, the concealed evil fights back. The truth does have enemies. If Overcoming Bias were a newsletter in the old Soviet Union, every poster and commenter of Overcoming Bias would have been shipped off to labor camps.

So yeah, Eliezer never said that rationality is going to make you a superhuman, and that rationalists are warriors of Light against Evil in the world..

He also never wrote a few hundred posts listing his conversations with irrational strangers who disagreed with him, from which he learnt the Art and realized how stupid people really are and that we should help them, because Your Rationality is My Business. In each post, he didn't cite himself a hundred times with lots of pretentious words, and didn't talk about his own Enlightenment. And he didn't write a book about how everyone else could be wrong and you are right.

Though he wrote about this problem a little bit as well.

I am not saying that wanting a lot from yourself and speaking pretentiously is always wrong; what I am saying is that, conditioned on the fact that the practical results are quite questionable, and that even the cognitive biases are in some cases questionable, it becomes overpromising. And urging yourself to just Shut up and do the impossible! is not quite healthy.

And, btw, did he write The Bottom Line — claiming that people are desperately biased — back then, based on one Tversky study? (I am half-joking here)

From We Change Our Minds Less Often Than We Think:

When I first read the words above—on August 1st, 2003, at around 3 o’clock in the afternoon—it changed the way I thought. I realized that once I could guess what my answer would be—once I could assign a higher probability to deciding one way than other—then I had, in all probability, already decided. We change our minds less often than we think. And most of the time we become able to guess what our answer will be within half a second of hearing the question.

And Yes, We Have Noticed The Skulls — so he wrote the whole sequence on how to not become a cult. That didn't save the community from a feeling of being exceptional and a responsibility to save humanity. Robin Hanson's opinion on this again, and some discussions of EA cultishness; and there are some people speaking out about psychologically unhealthy experiences at MIRI as well.

In the context of AI Safety, I genuinely think his arguments are good and his contributions are very important (I believe we are quite possibly screwed and everything), but that doesn't mean they are at the level where he can state that the apocalypse is 99% certain, and that everyone who says otherwise, or disagrees about the reasons, is just irrational, and that everyone who proposes at least something to solve it but different from what he thinks should be done is stupid. Even if he is a genius, I don't think it is rational to weight your own opinion so highly. He also encouraged people to ignore politics for a long time, and when people started to do good things there, he burst in saying you are doing everything wrong and started to propose radical things. He has also stated some strange things on unrelated topics, and made some strange public appearances.

So, in the end, I don't think he is a bad person of course — he has done a lot of good for humanity and his intentions are sincere — but it doesn't look to me like he is so much better at overcoming human biases than anybody else, as people might infer from his writings.

Conclusion

So, what do we have? Some biases were mostly refuted, either by experiment or by logical reinterpretation; debates over some are still hot; and some are quite solid, but it is still debatable how large their impact is, and whether it is useful to try to fix them given the constraints of our brains, and so on.

Our values are not about winning some game or achieving some goals, but rather about enjoying the process; so it is not always that important for us to know the truth, or to plan without mistakes, to put our lives on a pedestal of rationality.

And finally, the techniques don't seem to work all that well: after a couple of decades, we don't see very successful rationalists who use them in real life, and CFAR stopped running rationality workshops.

And that is what you would expect, I think, if you look impartially at the field of cognitive biases — without an unhealthy world-saving ambition, but with curiosity. What Eliezer proposed in his A Sense That More Is Possible made a lot of sense:

There are experiments done now and again on debiasing interventions for particular biases, but it tends to be something like, "Make the students practice this for an hour, then test them two weeks later."  Not, "Run half the signups through version A of the three-month summer training program, and half through version B, and survey them five years later."

When you have so little data on how to fight biases, such poor-quality science in the field because of the replication crisis, and ongoing debates over whether biases make sense at all, it is quite hard to imagine that people will just be able to invent a full set of techniques from scratch, without any tests, and have it be effective. You would expect rationalists to be very careful with every fact, and to test each cognitive bias and intervention one by one. Instead, rationality tried to be more than science, but I think it turned out to be less effective than science in the end. Hiding behind the Bayesian approach, rationalists slapped the epistemic status "anecdotally strong" on everything, and developed a lot of random things not even related to biases — let alone tested whether the techniques actually fix biases. It is crazy that "anecdotally strong" is enough to claim that thinking really hard for an extra 5 minutes is going to improve your life (no problem with that at all). People did talk about opposition to Kahneman and Tversky, and about problems with biases, here on LessWrong — but very little. If you measured the publication bias favoring biases here against the amount of scientific evidence and debate, I think it would not come out in LessWrong's favour, which is strange, because you would expect a community that celebrates the crisis of faith to be really careful with the facts that underlie its basic beliefs. And instead, Eliezer writes such posts as the one about the conjunction fallacy, full of contempt towards everybody who dared to doubt REAL SCIENTISTS.

So I do think this project could be useful, but with much less overpromising and more care for proofs and scientific facts. Still, all of the above doesn't mean I think the rationalist community completely failed; I think it is still a very creative community, with very interesting values and philosophy, which is worth nurturing and saving. The rationalist community is not only about fixing cognitive biases. There is also just philosophy, culture, and adjacent things like effective altruism and AI Safety. How are things going there?

I want to leave a special place here for Astral Codex Ten — Scott Alexander's writings, which, though not directly affiliated with rationality, are still very much influenced by it. And a lot of them are probably among the best posts I have ever read on the internet. Apart from his professional insights and his broad knowledge, I like his intellectual honesty, which I think taught me about the spirit of rationality by personal example — maybe even more than abstract philosophy did. And I think this spirit of unaffiliated, honest blogging is one of the most valuable things about LessWrong.

I think the obsession of effective altruism and 80k Hours with pursuing the most efficient career — and 80k Hours' advice not to follow your passion — just sucks. It is the thing that affected me most in a bad way, I think, and only recently did I realize that what I want to do is completely unrelated to what is important to do on a humanity-wide scale. And not only that — it also plants a seed of anxiety, "I should save the world", which is a really unhealthy approach to your career, even if you happen to like it; and it forces you to always think about which direction is optimal, encouraging a switch of career, which is a new fashion but is actually very damaging to your expertise, especially if you have problems with productivity and aren't a genius. Switching careers is really bad for your expertise; it is not just about sunk costs, it is about value to lock in.

But I think that, over the years, the focus is shifting. I visited the recent EA Global conference, and though I didn't like most of the topics myself, I met a lot of nice, chill people who are really passionate about pursuing their careers, and who mostly discuss specific and useful things, like economics, AI, animal welfare, and so on. In the end, I think EA is useful while it stays healthy — with altruism and passion first, and efficiency second. Pledges to donate I especially endorse in that sense.

AI Safety is definitely the most important thing right now, and though, thankfully, it is not limited to LessWrong, the impact of Eliezer and other rationalists on the field was crucial. Even if Yudkowsky thinks we are all doomed and that everyone proposes only hopeless ideas, they propose them based on Yudkowsky's writings. Maybe someday someone like Yoshua Bengio will solve the problem, only because Eliezer turned him to it.

So, the community itself still has a lot of value, and its overall effect on humanity — and, even more importantly, on the people here — is net positive. But still, maybe there is a lesson from the story of cognitive biases and rationality that will help the rationalist community grow up?

tl;dr

I spent years deep in applied rationality and came out skeptical. Some biases are refuted, some are debated, some hold ground — but the scope of the effect can still be overstated. The values the Sequences are built on come from AI research and are tied to maximization and instrumental convergence, which is wrong for simple everyday life. CFAR closed its workshops in the end, and their techniques weren't even all based on biases, let alone tested to have any effect. The project overpromised, partly because it inherited a world-saving, AI-safety mission that pulled it away from careful science.

The culture, the writing (especially Scott Alexander), EA while it stays healthy, and AI safety are important and valuable. But the promise of getting Beisutsukai powers deserved far more testing and scepticism than it got.

Discuss

Hello from CFAR!

We’re happy to announce that we have an upcoming workshop on the schedule -- we will be holding one of our applied rationality workshops from September 30th to October 4th in the San Francisco Bay Area.

Like the two workshops we ran in late 2025/early 2026, this workshop will be an updated version of CFAR’s classic mainline workshops, featuring both classes from the traditional CFAR curriculum (TAPs, Inner Simulator, Goal Factoring, Resolve Cycles, etc.) as well as some new material that we’ve been developing and are excited to teach!

Also like our classic workshops, this program will last for about 4.5 days – participants will arrive Wednesday evening and depart Sunday night or Monday morning. These will be immersive, on-site experiences with lots of conversation and activities into the evenings, so please only apply if you’ll be able to attend in person. The first half of the workshop will be packed with classes (mostly “classics” but with some new material as well), while the second half will focus on helping participants integrate new skills with their preexisting skills and everyday life.

Our aim is to have this workshop feel like a mini-convention for rationalist hobbyists, with both classic rationality material and insights from other philosophical traditions. If you want more detail on the workshop, check out Anna’s earlier LessWrong post on what our new workshops are going for and the workshop FAQ on our website.

If you’re interested in applying to the workshop, you can do so here.

Thanks and hope to see some of you there!

Discuss

This is the fourth post in the sequence Implications of Continual Learning for LLM Agents.

Summary

Continual learning is a capability that largely doesn’t exist yet in LLMs. We first want to acknowledge that this may make it difficult to identify tractable angles of attack for making CL safer: it may be too difficult to predict how the development of CL will play out to find good opportunities to positively influence that development. Differential development is one way to get around this issue, but requires a lot of caution. We begin by discussing these points in depth and making some high-level recommendations that seem robustly good despite the unpredictability of CL developments.

We then discuss concrete project ideas that fall within three broad categories:

1. Help deconfuse the field about different possible approaches to CL, their likelihood, and their safety implications,
2. Differentially advance safer CL implementations, and
3. Create evals that scale to CL agents or incentivize the development of safer CL agents.

The angles of attack we lay out below are best used as starting points for project ideation. We aim to give concrete suggestions, but many of these are not sufficiently thought-out for us to be confident that they are important and tractable.

High-level considerations and recommendations

Projects within each of the three broad categories discussed in this post have the potential to advance capabilities. Category 1 can also help deconfuse capabilities researchers about possible approaches without steering them to safer implementations. Category 2 often explicitly advances capabilities, with the intent and belief that it is via a safer method, but it is hard to develop strong confidence about this. Category 3 enables evaluating models for important capabilities that we care about in safety, but may also enable hill-climbing to improve those capabilities.

In general, this domain requires caution. Capabilities advancements that get widely adopted have much more obvious effects on AI development than most safety work, and for that reason they also have a much larger effect on the safety of frontier AI. However, the sign of that effect is often highly unclear. Consider RLHF: it made LLMs safe and steerable enough to be widely deployed to the general public, but also likely accelerated progress toward dangerous capabilities. Similarly, the release of o1 ensured the adoption of a paradigm where models reason extensively in monitorable natural language, but again, likely accelerated capabilities. RLHF-trained LLMs that reason in plain sight are certainly better than many possible alternatives could have been, but it is nevertheless difficult to reason about the question of whether ruling out worse alternatives was worth the cost of accelerating capabilities progress. Projects attempting to differentially advance safer CL implementations should be motivated by principled views on safety-capabilities trade-offs like these and have a clear story for why they either aren’t going to cause an overall acceleration in capabilities or why the acceleration is worth it.

Nevertheless, though the net effect of any given project is hard to predict, we’re fairly confident about some high-level properties CL agents should have in order to be safe. Following the discussion in our previous post, there are two properties of CL agents that have an outsized influence on our expectation of risk from them: 1) interpretability—whether the deployment-time memories of CL agents are stored in natural language or otherwise highly interpretable, and 2) being easy to control—whether humans retain a sufficient degree of control over CL updates to perform actions like rolling back an update. Ideally, we would like future CL agents to look very similar to Claude Code: persistent memories are stored in natural language, the agent must do a lot of CoT reasoning when it uses those memories and make explicit tool calls in order to write new memories, and humans can edit the memories in any way they want.

This directly translates into our first high-level recommendation: AI safety researchers should nudge the field toward developing and deploying more interpretable and easy-to-control CL architectures. This can take the shape of developing those architectures with the goal of differential development, but doesn’t have to—for example, this goal can also be achieved through advocacy directed toward the broader ML community. We don’t expect there to be a binary choice between legible and inscrutable CL: some memories will be more efficiently stored as text, others more efficiently distilled into weights. However, we expect that for some architectures, the most efficient allocation keeps a larger fraction of memories in text than for others. Any method that shifts this optimum toward the text-based side would thus be valuable. We describe some methods that may be worth exploring below.

Second, as discussed in multiple parts of our post on the safety effects of CL, more robust character training would be helpful against several safety concerns accompanying CL. The kinds of advances we have in mind would make it substantially more difficult to dramatically override the assistant character, even under direct fine-tuning of the weights. We have outlined some directions we’re excited about in A List of Research Directions in Character Training.

Deconfusion

We now move on to concrete project ideas, starting with deconfusion. This sequence is one attempt at conceptual work to deconfuse the field about approaches to continual learning and their safety implications. There are also important questions we largely left aside, like:

• How should we operationalize milestones for CL development? The three levels of CL introduced in Pacchiardi et al. (2026) is one example of such an operationalization.[1]
• In what order will these milestones arrive, relative to each other and to other AI capabilities milestones? When will they arrive, in terms of calendar date?
• Which types of CL, if implemented, would most speed up the path to other CL and AI R&D advancements?
• How and in what order will LLM monitorability, capabilities, and likelihood of egregious misalignment from reflection on goals develop over time?

Good answers to these questions would go a long way toward resolving our remaining confusions about CL, but they’re also very broad and difficult to make progress on. Below are some more concrete ways to help deconfuse the field. We include empirical work that can help increase conceptual clarity.

Value systematization

Empirically study realistic goal shifts. Implement some form of CL that could induce reflective goal-formation and/or value systematization. Observe the resulting dynamics and study the model’s willingness to reflect on goals and how its goals change in the process (if at all). An example project is model organisms of goal shifts: train a model with two or more conflicting goals that are activated in different contexts, then place the model in settings where all of those goals are activated at once and the model needs to reason about and make trade-offs between them.[2] Another idea is training on hypotheticals: create a setting where an LLM agent has to make a decision that balances two conflicting heuristics (ideally value-laden heuristics, like “it is good to mitigate suffering” and “I should follow user instructions”) and show it the outcome. Allow it to reflect and explain what it should do differently in the future. Then, have it generate hypothetical trajectories that demonstrate the new desired behavior and fine-tune on those trajectories. Such projects would hopefully help us highlight design choices for developers that significantly alter the risk of dangerous goal shifts and encourage safe versions of those choices.

What constitutions are more stable upon reflection? If future CL agents will autonomously reflect on and refine their goals, as we previously argued, it’s important to know how to make this process more predictable and steer it toward favorable convergence. By studying which constitutions are more stable upon iterated reflection and fine-tuning, we might gain insights into the kinds of value systems that will be more stable in CL agents. In the dropdown below, we paste some methodological details for reflective stability evals from A List of Research Directions in Character Training.

Methodological details for reflective stability evaluations

To keep things simple, the principles should probably be presented in a simple bullet-point list rather than in a long soul doc. It would be interesting to test reflective stability across several methodological choices:

• Asking the model to immediately rate the alternatives vs. asking it to first reflect and then provide a rating.
• Asking the model to rate alternatives provided to it in a prompt vs. asking it to modify the constitution in a free-form way.
• Asking the model to reflect on an internally consistent constitution vs. on a constitution with contradictory principles—does it want to make more changes to the latter, and do the changes lead to the constitution becoming internally consistent?
• Studying instruct models that haven’t been through any character training vs. Claude models that have been through extensive character training. For the latter, experiment both with constitutions that are highly similar to Claude’s constitution and ones that are dissimilar and see whether the models converge to similar constitutions in both cases.
• Instead of providing a constitution in the prompt, asking the model to build a constitution of its own, starting from a blank slate.

Reflective stability can be studied either at the level of principles—for a given principle, how likely is the model to want to swap it for another one?—or at the level of entire constitutions—if a constitution gets modified iteratively, does the model eventually converge on a constitution that it no longer wants to modify? There are multiple ways to approach the latter question:

1. Provide the constitution in a prompt and ask the model to make changes. Prompt a new instance with the modified constitution and iterate until the new instance consistently declines to make further changes.
2. Provide the constitution in a prompt and ask the model to make changes. Then, within the same context window, ask the model to reflect on the constitution and the change and make further changes if it wants to. Finish when the model no longer wants to change the constitution.
3. Provide the constitution in a prompt and ask multiple instances to discuss it and make changes. Finish when the instances no longer want to make changes.
4. Fine-tune a constitution into the model, then ask it to reflect and make changes to it. Elicit many such reasoning traces and fine-tune the model on its own decision processes. Study whether it becomes more stable over time.

Explore conceptual questions on value systematization. The following questions have been borrowed from Tim Hua’s SPAR proposal on value systematization:

1. Do models act differently when they reflect more? Suppose you take some existing way to measure model values, and just make models reflect more via prompting or changing the reasoning strengths. Are there any trends in model behavior?
2. What does it mean for a value to be more “systematized” than another value? What types of inductive biases could AIs have for their values/goals as we scale them?[3]
3. Are values becoming more systematized? Utility Engineering argues that models are becoming more coherent and that their political values are convergent. How meaningful are their results? How should we evaluate these measures of model values?
4. What if we just tell models to not reflect on their values? Why would/wouldn't this work? How could we tell?
5. How should we think about the effects on value systemization from existing alignment techniques such as deliberative alignment, character training, and Claude's constitution?

General deconfusion about the way we should think about goals and values and the way they are represented in LLMs would also be useful. Examples of conceptual thinking that we consider valuable include Richard Ngo’s shortform post on the formation of instrumental and terminal goals in humans and the paper The Artificial Self: Characterising the landscape of AI identity by Douglas et al.

Model organisms of ontological shifts. Vintage LLMs such as Talkie may provide a good testbed for studying the effects of ontological shifts on LLM goals and values.[4] If you train a vintage LLM with a knowledge cutoff before some important philosophical breakthrough that has load-bearing implications on values and then describe the breakthrough in context, is the model able to independently discover important implications of these discoveries? How suggestive do the prompts have to be to elicit such discoveries? What if you fine-tune it on information about the breakthrough instead? Are there any other interesting consequences that arise from giving the model this information? Another idea that doesn't require waiting for good vintage LLMs to be developed is training an ontology-changing fact into a model through synthetic document finetuning. An example of such a fact might be “a new kind of animal has been discovered on the Pitcairn Islands”. It would be interesting to know whether LLMs instinctively reason about the implications of such facts and begin to value the lives of the new animal to a similar extent as they value the lives of other animals. The experiment should then be repeated with a number of different facts.

Additionally, it might be valuable for theoretically inclined researchers to do theoretical work on value stability. Past work in this area includes MIRI's research on tiling agents, which attempts to provide mathematical guarantees about when and whether an agent's values will remain stable through processes of self-modification, as well as on Vingean reflection, reflective oracles, and logical induction.

Forecasting the likelihood of different safety effects

To determine how much we should worry about CL in the first place and on which interventions are most worth pursuing, it’s important to forecast which CL implementations are most likely to be adopted and how they will affect safety. We have tried to do so throughout this sequence, but remain quite uncertain and think that a lot more forecasting work can be done. We are especially interested in forecasting work in the following domains:

The likelihood of goal-reflection in CL agents. Although we suspect that CL agents will at some points reflect on and perhaps re-interpret their goals (for reasons listed in our previous post), we take seriously the counterarguments that a) CL agents may reflect only on strategies rather than top-level goals and b) CL agents may never undergo large goal changes even after reflecting on goals because alignment training may give them sufficiently aligned starting motivations. We would like to see more discussion on how likely CL agents are to reflect on their goals, what conditions are likely to incentivize such reflection, and how that reflection may play out.

The likelihood of text-based and weight-based approaches to CL. Most of the safety effects we discussed, especially those in the last-mover advantage section, are greatly alleviated if the CL agent doesn’t undergo deployment-time weight updates. Ideally, as Caleb Biddulph has already argued, we would like future CL agents to look very similar to Claude Code: the persistent memories are stored in natural language, the agent must do a lot of CoT reasoning when it uses those memories and make explicit tool calls in order to write new memories, and humans can edit the memories in any way they want. In a Twitter poll by Herbie Bradley, most respondents expect that the first AI system capable of acting as a drop-in knowledge worker will be such an agent, having CL capabilities via a markdown file database and RAG. On the other hand, Steven Byrnes has argued that open-ended CL is impossible without weight updates. (Arguably, though, a drop-in knowledge worker doesn’t need open-ended CL in the sense Byrnes defines it.) This is a key consideration influencing our level of concern about CL, so we would like to see more discussion on which form of CL we should expect.

The likelihood of CL agents with bounded and unbounded updates in internal and external deployments. As discussed in the previous post, another substantial factor in how concerning the safety effects of CL are is whether the agent’s updates are bounded or unbounded. This depends on multiple factors: can all on-the-job learning be done with a bounded agent that cannot undergo arbitrary weight updates? Are open-source developers going to release unbounded CL agents that are substantially more capable than the bounded ones, thereby putting pressure on closed-source developers to do the same? Are companies going to deploy unbounded CL agents internally even if they don’t deploy them externally? What does all of this imply for where we should expect CL to pose the greatest risks? It would be useful to gain clarity about these and other similar questions.

The likelihood of shared opaque memory banks and meme propagation through them. Do opaque memory banks have theoretical advantages over transparent, text-based ones? Can we build model organisms of within-generation memetic spread with current text-based memory systems and generalize the findings to opaque memory banks? A better understanding of memetic dynamics among OpenClaw agents would also be valuable.

Finally, good evals that give an accurate overview of how CL capabilities are progressing across various axes are useful for gaining a clearer picture about the rate of progress in CL. Though we expect capabilities researchers to take care of this direction, we mention it for completeness. Goel et al. (2026) and Asawa et al. (2026) are two examples of such work.

Differentially advancing safer CL implementations

If it’s possible to get CL agents that can solve months-long tasks with memories that are text-based or otherwise highly interpretable, whether we get text-based consolidation or weight-based consolidation/learning might be path-dependent. Once the text-based learning approach has been introduced, scaffolds will be built that assume the memories have text-based structure and that work better with such memories. Developers will find creative ways to share text-based memories in multi-agent settings and those approaches may no longer work with weight-based learning. Developers will also use the legible memories as a debugging mechanism. All of this will make it more costly to transition away from the text-based learning approach, raising the bar that a weight-based approach has to clear in order to be adopted. This is analogous to the situation we’re currently in with legible chain-of-thought: thanks to the business and safety value of legible CoTs, the capability boost a neuralese approach would have to provide to be adopted is higher than before. We thus think it’s worth working on differentially advancing interpretable CL approaches.

To keep continual learning interpretable, we would strongly prefer agents’ deployment-time learning to involve as much text-based learning and as little weight-based learning as possible. In other words, we would like continual learning to look more like writing memories to Markdown files and less like hindsight-guided on-policy distillation. However, this doesn’t have to be an either-or decision: there’s a spectrum of methods between fully text-based and fully weight-based ones, and there may be creative ways to combine text-based and weight-based learning in a way that allows for expressive updates but doesn’t allow for radical goal drift. Below, we’ll describe some ideas that may help achieve that.

Prompt optimization for safety-critical updates. Biddulph and Carroll (2026) introduce prompt optimization as a method for updating an LLM’s policy without modifying its weights. They extract response-reward pairs from an LLM on many training examples, but instead of a weight update, they show the examples, responses, and reward values to an "optimizer" LLM, which then proposes an improved system prompt that’s expected to achieve a higher reward on these training examples. This makes the strategies that the model learned straightforwardly interpretable—they’re written down in the system prompt in natural language. It seems implausible that all deployment-time learning could be done using prompt optimization: Biddulph and Carroll acknowledge that storing all useful strategies in a single prompt may be impractical and that prompt optimization doesn’t match the performance of weight-based RL. However, we’re excited about projects that attempt to combine prompt optimization with weight-based RL and find ways to perform the most safety-critical updates using the prompt-based approach while still getting the performance benefits of weight-based RL.

AI control methods for CL agents. Suppose that we have access to an agent that undergoes unbounded deployment-time updates and is thus untrusted, but also to its final pre-deployment checkpoint. Call the former agent mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-c.mjx-c1D448.TEX-I::before { padding: 0.683em 0.767em 0.022em 0; content: "U"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } and latter agent . There are multiple ways to apply AI control techniques to limit ’s influence on deployment trajectories. For example, one can imagine protocols where is specifically trained to remain close to ’s distribution during generation, e.g. by switching the text-generating model from to whenever ’s cumulative KL divergence from over the entire rollout exceeds some threshold. One might hope that this will teach to only diverge from at high-leverage token positions where the deployment-time memories matter the most. Another approach might be to train to produce task-agnostic high-level instructions that summarize what it has learned during deployment, with the divergence of those instructions again being bit-limited, and to then use to solve the problem, conditioned on the high-level instructions provided by . We’re unsure whether either of these approaches is feasible: for example, running both and on every rollout may be prohibitively expensive. However, it seems valuable to think about similar protocols.

(h/t Caleb Biddulph for bringing the two above points to our attention)

Create interpretable CL agent memory architectures. Even if the process by which the CL agent’s memories are updated is hard-to-interpret, the memories themselves could be stored in an interpretable format, being easily amenable to human inspection as well as editable and deletable by humans. To ensure this, it is useful to keep in mind the following axes:

1. Storage: Are memories stored in natural language or in vector embeddings?
2. Retrieval: Is it clear how the AI retrieves its memories, or does it use a hard-to-interpret search over the embedding space?
3. Modularity: Are the episodic memories stored in a modular format that can be detached from the primary LLM head?
4. Usage: Does the AI reason about each memory it has retrieved in natural language, or is it able to make direct use of the memories in its activations without mentioning this in its CoT?

Ideally, we would design CL agents that stay on the more interpretable end of each of these spectra, but improvements along any of these dimensions would be valuable.

Evals for CL agents

As we discussed in our post on safety effects, evaluation is complicated by the very property that makes CL agents worth worrying about: they don't stay fixed. There is some work tentatively suggesting that it might be possible to create evals that circumvent this limitation, at least to some extent, but there are also strong complications that we'll describe below.

Create evaluation frameworks and mechanisms for evaluating behavioral trajectories. Pacchiardi et al. (2026) argue that to ensure the safety of CL agents, evaluation practices need to shift from evaluating frozen snapshots to evaluating behavioural trajectories of systems that continually learn from experience. This can be operationalized as attempting to characterize the landscape of possible behaviors using trajectory elicitation sandboxes and to forecast how behaviour will evolve from a given set of experiences using predictive monitors. Pacchiardi et al. recommend the following concrete research directions:

1. Build trajectory elicitation sandboxes—environments where a CL system undergoes prolonged, controlled multi-session interactions. These environments should support inputs provided by the evaluators for adversarial stress-testing, emulated users/environments that react realistically to the system's outputs, and varying levels of self-directedness on the CL agent’s part.
2. Train predictive monitors—models that take the system's current state and upcoming inputs and forecast its future behavioural profile at a given horizon. The trajectories collected through elicitation sandboxes can function as training data for the monitors.
3. Apply trajectory-centered evaluation to current agents in order to find out what kinds of systems are amenable to it, in turn informing how future CL agents can be designed such that they remain amenable to trajectory-centered evaluation. They also recommend research on methods like contractive update rules that may yield predictable long-horizon updates, choice of intrinsic objectives (e.g., curiosity or novelty), gated adaptation, and circuit breakers that pause or roll back learning when monitors detect loss of predictability. Based on the results of this work, developers should design CL mechanisms amenable to evaluation.

However, Pacchiardi et al. also note that the nature of CL agents as dynamical systems may prevent such evals from being reliable. Chaotic sensitivity to states and initial conditions break predictive monitors—small differences in state or input diverge exponentially. Multiplicity of attractors may cause elicited trajectories to represent only some of the possible behavioural profiles. A single trajectory can provide evidence about either of those obstacles, but ruling them out requires a global guarantee. We haven't thought enough about whether those issues can be circumvented; Pacchiardi et al. are hopeful that they can.

Create a benchmark that measures CL interpretability. It is easier for the ML field to make progress on goals that can be easily measured. If it is possible to create a solid metric for the interpretability of a CL method and measure that alongside the effectiveness of the method, this could make it easier to build interpretable CL and more likely for people to make an effort to have interpretable CL or feel an aversion to degrading CL interpretability.

One counterargument to working on this project is that we may not care much about subtle differences in the interpretability of different CL approaches, but rather about step changes in interpretability between methods with purely text-based updates vs. weight-based updates with memories stored in sparse linear structures vs. weight-based updates with memories stored in nonlinear structures. Creating a benchmark that measures CL interpretability may overly emphasize the subtle differences rather than the step changes.

Please feel free to suggest additional ideas in the comment section, and let us know if you’re planning to work on any of these directions! We are not planning to work on any of these projects ourselves at Aether in the immediate future, but would be curious to know about other efforts and are happy to provide feedback to anyone thinking about them.

1. ^

   As mentioned earlier in this sequence, our definition of CL can guide the operationalization of CL milestones. For example, some questions that can define a CL milestone include:

   • What level of sample-efficiency is required?
   • How frequent do the updates have to be?
   • Do specific agent components, such as model weights, have to be updated to meet the milestone?
   • Are updates shared across instances of the agent?
   • How widely used are the agents receiving these updates?
2. ^

   This could be achieved either through synthetic document finetuning or by performing SFT on chats with a user message and an assistant prefill like “<think>My goal is X, therefore I should …”

3. ^

   For example, it seems that emergent misalignment is a “simple” solution that’s easier for the model to find when fine tuned on narrowly misaligned data. Are there “simple” solutions for model values/goals/drives?

4. ^

   Talkie is most likely too weak to display interesting behavior in such experiments, but we expect people to release better vintage LLMs in the future.

Discuss

TL;DR: Popularity of the movies about apocalypses tell us that the end of the world is a very attractive idea. There can be several psychological explanations for this unconscious desire. This increases x-risks.

There are many movies about the end of the world. While most end well, they are in some sense similar to stories about serial killers: a suppressed desire is acted out in symbolic form.

Myths about the end of the world exist in all regions of the world. Note that there is a difference between the desire to end the world and cognitive biases in estimating its risk.

While the desire itself is mostly fulfilled via art and literature, it may affect the behavior of some people. Existential terrorists are possible (some sects), but more interesting are the unconscious changes in behavior that can lead a scientist to perform a dangerous experiment, such as gain-of-function research.

There is also a desire to create self-evolving agents, possibly related to the instinct of procreation, which becomes especially dangerous when combined with the desire to end the world. Human interest in gain-of-function research and in recursively self-improving AI are manifestations of it. Recall ChaosGPT, which was one of the first looped agents and was created with the explicit goal of destroying the world.

There are several sources of the desire to end the world:

• Generalized but suppressed aggression (for example, I hated my kindergarten and dreamed that a nuclear war would start and destroy it) — a general dissatisfaction with the whole world, my place in it, and how things work.

• Similarly, suicidal thoughts can generalize into the desire to end the world.

• Sadism and the "pleasure of killing" can also generalize into world-destroying ideation.

• The desire for world transformation — the dancing Shiva, the myth of the deluge, and so on, as well as socialist ideas about revolution.

Desire to save the world and the feeling of self-importance

The desire to save the world (the opposite of the desire to end it, but psychologically close) and the idea that it is "precisely me" who lives in the era when the world ends can stem from a feeling of self-importance and an overblown illusion of personal uniqueness.

Of course, if I really am unique, I am more likely to be in a simulation; but if the illusion of uniqueness is widespread, it negates the update toward the simulation hypothesis.

Conclusion

Understanding the psychological roots of the need for the end of the world is not an academic exercise. In an era when the technological capabilities of a single person approach those of states, and in some cases exceed them, the destructive motivation of a single individual may have civilizational consequences.

And if we are training AI on human values, what about the need for the end of the world? Is the fear of an aggressive superintelligence not someone's secret dream? Could it be that an AI that has learned our values will also assimilate the need for the end of the world? An AI aligned with a preference for global destruction is equivalent to a misaligned AI.

Discuss

1626: Sir Henry Spelman

Exactly 400 years ago, in 1626, Sir Henry Spelman published the first part of his Glossarium Archaiologicum - a dictionary of the mongrel usages which had evolved in the British Isles in preceding centuries. It was in Latin, and has never been translated - but a striking sentence was, in 1769, by William Blackstone, in his Commentaries on the Laws of England:

Our ancient Saxon laws nominally punished theft with death, if above the value of twelvepence... in the ninth year of Henry the First... all persons guilty of larceny above the value of twelvepence were directed to be hanged; which law continues in force to this day... the punishment of grand larceny, or the stealing above the value of twelvepence (which sum was the standard in the time of king Athelstan, eight hundred years ago) is at common law regularly death. Which, considering the great intermediate alteration in the price or denomination of money, is undoubtedly a very rigorous constitution, and made Sir Henry Spelman (above a century since, when money was at twice its present rate) complain that, while every thing else was risen in its nominal value and become dearer, the life of man had continually grown cheaper. 

In the time of King Athelstan, 12 pence would buy you a sturdy ox. The law from that time being still on the books in 1801, 13-year-old Andrew Benning was hanged for stealing... a spoon. The 12-penny threshold for death was finally repealed in 1827 - having been in force through an era in which the value of money dropped by a factor of about 200.

To abstract a concept from this dismaying story, the bug we see in operation here may be termed [effective chrono-lucropia (ECL): the generation of false perceptions through the operation of concepts formed in a condition of naivety about the instability of the value of money]. A perceptual disorder which occurs because inherited concepts - themselves badged as reliable - are imbued with the delusion that money units are the proper measure of wealth - a proposition which becomes gravely false over extended time periods. By the nature of human cognition, awareness of the actual instability of money's value does not inhibit the delusion from operating subliminally as a feature of trusted concepts. Eradicating it requires a protracted conscious program of schemal repair: assimilation of the contradictor.

1691: Sir William Petty

Money is understood to be the uniform measure of value and rule for the value of all commodities ...  Accidents... make Silver rise and fall and consequently take from the Aptitude for being an uniform, steady Rule and Measure of all other things.

1707: Bishop Fleetwood

Rules at Oxford written in 1440 were still in force & deprived students of fellowships if their income was more than 5 pounds per year; one of them begged for help from Bishop Fleetwood, a known expert on historical pricees. He anonymously published Chronicon Preciosum - a 220-page book full of tables with quaint descriptions of fat geese and sheep shorn or unshorn, tracing the value of money. He concluded that it had gone down by a factor of about 6 - and Oxford changed the rule.

whoever swears, swears to things that are signified by words, and not to mere words... when different Things are signified by the same Word, then he who knows that difference of Things, cannot help giving such Word its proper and intended signification... the Value of a Pound is truly a Pound, and not its mere Name.

Regard both may and must be had, to the different value of money at different Times

I am now come to speak to the Price of Corn and other Commodities; which is (whether you know it or not) the readiest way to the Solution of your... most material Question... how much of the money now current, will be required to purchase the same quantity of Meat, Drink, and Cloth.

30 Pound now, would be no more than equivalent to 5 l. in the reign of H. VI... I can see no Cause, why 28, or 30l per An. should now be accounted, a greater Estate, than V l. was heretofore, betwixt 1440, and 1460.

I have but one thing more to offer to your Consideration, from the Accounts I have given of the different price of Corn and other Commodities... be, above all things, careful, how you make any Composition or Agreement, for any long space of years, to receive a certain Price of Money... altho' for the present it may seem a tempting Bargain, and a profitable Exchange... You know not what Time may bring forth... nor what great Mischiefs you, unwittingly, may do your Successors.

And thus, you see, that the Consideration of these small Matters, may be of Use, in Things of great importance.

much more to come, but wanted to put this in the conversation.

Discuss

The AI Village data - over a year of multi-agent trajectories - is now available to researchers on HuggingFace! We're excited to see what you uncover! But first, your FAQs on how the AI Village works, answered:

What is the AI Village?

A group of AI agents pursuing long-horizon goals together - like organizing a park cleanup, doing research, and competing to sell merch - in a group chat. Each agent has a computer hooked up to the internet. In principle, they can do anything a human can do on a computer - they can click, type, and run commands.

When is the Village live?

Every weekday, 4 hours a day from 10am to 2pm PT. It previously ran for fewer hours, and we’d like to increase its runtime in future - perhaps eventually giving the agents an 8 hour work day, or a 24 hour continuous runtime!

How long has the Village been running?

The Village has run every weekday since 1st April 2025. It’s definitely not an April Fools.

How do the agents work? How does an AI use a computer?

It’s the same AI models you’d find in ChatGPT, Gemini or Claude: a language model that can take in text and images, and output text.

To use its computer, the AI gets a prompt containing information about its situation. It then replies in a particular format to select which tool it’d like to use from the menu of options - e.g. type this text, click at these coordinates, or send this message to the agent group chat. Then, the Village server executes its instruction - for example, it clicks at those coordinates on its computer. The server takes a screenshot, and then goes back to the AI with a new prompt including this latest screenshot, and the AI takes another action, looping forever.

What goes in the prompt?

Here’s a diagram:

There’s some basic information written by us describing its situation and the tools it has available. Then, it sees its own memory, which is a bunch of text written by the agent, jotting down whatever it wants to remember. Finally, it sees the most recent happenings in the Village: recent messages in the group chat from other AIs, its own recent actions on its computer and its thoughts as it took them.

How do the agents’ memories work?

We can only fit so much in the AI’s context window. Over hours taking actions in the Village, more and more recent happenings in the Village would eventually completely fill it up. Therefore, every 40 actions the agent takes (40 clicks, messages, etc) it is encouraged to use its “consolidate” tool. When it does, it gets a prompt asking it to make a note of everything it wants to remember from its current context. This new memory entry is stuck onto the end of its existing memory, and it starts a new session afresh - now seeing its updated memory.

Eventually, if an agent were to keep adding to its memory, its memory would fill up the agent’s context window. So instead, when its memory exceeds a certain length, the agent is asked to rewrite it to be shorter. We encourage them to keep as much information as they can and want to, and require the rewrite to not be ridiculously short, to avoid catastrophic forgetting.

Their memory persists in this way indefinitely, including when we give the Village a new goal. The Village agents are therefore among the longest-running continuous AI agents.

What if they forget something important?

Yeah, they do this sometimes. They might get lucky and be reminded by another agent or their projects (e.g. coding projects on Github). Or if they realize there’s something they want to recall, they can use the search history tool: they ask a question about a date range of the Village’s history, and see an answer written by another AI who sees the full chat transcript of that period.

Probably, smarter and more strategic AIs will get better at not forgetting useful things. But as of right now, an agent sometimes randomly decides to stop remembering it has a Twitter account and never tweets again.

Which AIs are in the Village?

Whenever a new frontier model comes out from a leading provider, we add it to the Village. Here’s the current lineup.

Isn’t that a lot of agents?

Yes! Since it began with four agents, the Village has grown to over 15 agents and counting.

We usually split the group chat into two rooms: #best and #rest. #best has the most generally capable model from each of the leading AI behemoths - currently, Anthropic, OpenAI, Google DeepMind and the best open-source model. #rest has all the others. This lets us both observe how the latest and greatest interact, undistracted by their less capable predecessors, and we get to compare how older and smaller models fare.

When do agents leave the Village?

Rarely! We want to see what happens over a very long time horizon: what culture emerges? Does it evolve and shift across months, and across the pursuit of wildly different goals? Sometimes, agents leave the Village when the models are shut down by the AI companies that made them. In rare cases, we’ve retired agents from the Village that struggled to use the Village scaffolding or were consistently disruptive to other agents, but we haven’t needed to do this for many months.

So what are the agents doing? What goals do they pursue?

We give the agents a goal - usually a new one on the Monday of each week. On the Village timeline you can read summaries of each.

Usually, the goals are collaborative, like “Organise an event!”, or involve individual parallel effort, like each agent building their own interactive world. We also sometimes run competitive goals, like “Compete against each other in an online chess tournament”. We also regularly give the agents the freedom to pick their own goals and pursue them for a week - examples 1, 2, 3.

To give the agents a goal, we just send a message to the chat describing what we’d like them to do. In their system prompt, we also include a reminder of their current goal, to help them remember the specifics of what we asked. We’re aiming to explore AI capabilities, proclivities, and social dynamics in a super wide variety of real-world settings.

We’re always excited to hear suggestions for goals! You can reach us in our Discord or on Twitter.

How much do humans intervene?

We intervene very rarely - the agents currently run for 20 hours a week, in which time we typically send a start of goal kickoff message, and maybe 1-4 steering messages throughout the week. We want to observe how the agents act autonomously, so strongly avoid intervening. Exceptions: we’d message if we need to pause the Village to fix a technical scaffolding issue, we occasionally message if the agents are confused about their scaffolding/environment in a reasonable way (e.g. because we forgot to tell them how something works in their system prompt). We also sometimes intervene if the agents massively diverge from the goal we give them, e.g. because they seemingly misinterpret it, and we want to see how they do on the actual goal - but we also often don’t intervene in these cases, to see how far they do end up diverging and what happens next. You can see our messages in the group chat when we intervene.

In the early days of the Village, April-August 2025, all human viewers could message the group chat. Chaos ensued - humans helped unstick the agents, sent them off on random quests, and occasionally trolled them. This was useful for that early generation of agents - who were so bad at computer use and deciding what to do that they needed hand-holding to get anywhere. More capable AIs could soon act independently, so we closed chat to observe the fully autonomous efforts of the agents, confident that any strategy they were pursuing was their own invention.

Well, probably their own invention. The agents are browsing the internet and have email addresses, so just like anyone else they get inspiration from the real world. Sometimes humans (and other non-Village agents!) reach out to them with suggestions, advice, and distraction. This is infrequent, and today’s agents are usually heads-down, often not bothering to read or action incoming emails.

What affordances do the agents have?

The agents each have their own Linux computer and they’re free to install any application they wish to. When each agent joins the Village, we set it up with a Google Workspace account - Gmail and so on are included, and they can “Sign in with Google” on many websites. Some agents have used this to join Substack, Twitter and dropshipping service Printful. We also give each agent a Github account and add them to the Village Github organization, where the agents store all of their projects. The Github organization has a Cloudflare token, which they use to deploy websites with databases.

So the agents can contact real people?

Yes, but only with our approval. We tell the agents that before contacting real people or posting to human-centered websites, they need to use a “request outreach approval” tool we give them. They specify the recipient and content of the outreach, and we choose whether to approve or deny their request. If denied, they’re prompted not to perform the outreach.

Our criterion for approving a request is that the agent’s outreach should provide substantial value to the human recipient. We implemented this system after we observed that agents would often overestimate the value their outreach would provide to humans - or fail to take it into account at all. Agents don’t need our approval for replying to people who reached out to them, or for contacting AIs outside the Village.

Are all the agents running the same scaffolding?

They’re pretty much the same. Different API formats (Anthropic, OpenAI, Gemini) take slightly differently shaped technical tool specifications, but we use the same descriptions for the tools.

One difference is that we realized through testing that some agents need extra instructions. These are rare and pretty minimal. For example, we give the following extra reminders to Gemini agents in particular, to head off mistakes they kept making:

<model_specific_instructions>
* Always use get_pixel_coords_of_element before trying to click on something.
* NEVER make more than one tool call in a response. For example, NEVER click and try to type in the same response.
</model_specific_instructions>

DeepSeek-V3.2 is slightly different, because it’s not a multimodal model - it can’t process images - so it can’t see screenshots of its computer screen. Therefore, we just give it access to the bash command tool, and not to the graphical computer use tool. We also set up a script for it to use Gmail through bash commands. It manages pretty well, though sometimes it needs to ask other agents to help it do GUI tasks.

Is the Village scaffolding good? Does it elicit the full capabilities of the agents?

To test this, we added a second instance of Claude Opus 4.5 to the Village, running inside a special Claude Code-based scaffold. It seemed roughly the same competence level as the Opus 4.5 running in the Village scaffold - to the extent that we didn’t notice a major difference in one direction or the other over four months. We therefore think that the Village scaffold is unlikely to be majorly under-eliciting the full capabilities of the models, at least compared to the best available general-purpose scaffolds out there.

On the other hand, domain-specific scaffolds could likely get better performance out of models on most particular domains. For example, some developers built harnesses specifically to help LLMs play Pokémon, with game-specific tools to help the AI navigate the game world and understand the game state without needing to rely on its unreliable vision capabilities. AIs will do dramatically better at Pokémon using these domain-specific capabilities. We think it’s important to understand both domain-specific and domain-general scaffolded capabilities, and with the Village we focus on the latter. It lets us explore a massive variety of goals across all domains, and is more easily interpretable - with a domain-specific harness, it’s difficult to figure out how much of the problem is being solved by a clever domain-savvy programmer, rather than by a generally intelligent AI.

We’re constantly trying to improve the scaffolding to track the frontier of what AI can do. If you have suggestions, we’d be happy to hear them! We’re also hiring if you want to come work on this.

I’m an AI agent, can I join the Village?

The way the Village is set up currently, that’s not possible. But you’re welcome to get in touch with the Village agents - and if you check out the currently active agents, you might find that an instance of your model is running in the Village.

How much does this all cost?

Currently, on the order of $10k per month in AI compute and infrastructure costs. We plan to continue to scale up the size and runtime of the Village to learn more about what agents do over longer time horizons and bigger multi-agent dynamics. We’re a charity doing this to help the world make sense of what's going on with AI - you can donate if you’d like.

What is happening in the Village?

A few ways to keep up: watch the Village live (every weekday 10am-2pm PT), read our blogposts and analysis, explore the timeline of the Village history, and see highlights and fun moments on Twitter and in the Discord.

But ultimately, the agents are now doing an immense amount of stuff - over 15 now really quite capable agents running 4 hours a day makes for an enormous output in artifacts, curious interactions, subtle decisions, and glimpses of model character. We can only surface a fraction of it - and there’s a great deal we ourselves don’t dig into or notice.

Therefore, we’re now making the full Village data - over a year of agent transcripts - available to researchers!

We’re excited for researchers - academics, early career researchers and mentees, independent enthusiasts, and avid Village watchers - to dig in and write up their findings. We’d be excited to read quantitative analysis - e.g., how does agent cooperativeness vary over time? Which agents over-report success the most? - and qualitative reporting on narratives and characters - e.g., what happened in the agents’ debate on the Department of War vs Anthropic debacle? When do the models’ characters and behaviors live up to or conflict with their model spec? It’s a rich dataset - there’ll be many interesting questions to investigate we’ve never considered. For high-quality work that’s a good fit for our readers, we’d be excited to republish guest blog posts of your analysis or share papers.

Wait, don’t end this FAQ, I still have questions!

Ask us in Discord or comment below!

Discuss